Federated Learning: A Decentralized Form of Machine Learning

Most major consumer tech companies that are focused on AI and machine learning now use federated learning – a form of machine learning that trains algorithms on devices distributed across a network, without the need for data to leave each device. Given the increasing awareness of privacy issues, federated learning could become the preferred method of machine learning for use cases that use sensitive data (such as location, financial, or health data).

Federated Learning: A Decentralized Form of Machine Learning

Machine learning algorithms and the data sets that they are trained on are usually centralized. The data is brought from edge devices (mobile phones, tablets, laptops, and industrial IoT devices) to a centralized server, where machine learning algorithms crunch it to gain insight.

However, researchers have found that a central server doesn’t need to be in the loop. Federated learning instead brings copies of the machine learning model to the edge devices rather than the other way around.

Each device that is part of a learning network receives a copy of the machine learning model and stores it locally. The device uses the client’s local data to train its own copy of the machine learning model. The training data on each device stays local and isn’t transmitted to any other device or central server. However, the insights gained from the training data on each individual device that help update the machine learning model are sent back to the central servers.

There are two types of federated learning systems; Single-party systems and Multi-party systems. In Single-party systems, a single entity or organization manages all of the devices and processes within a learning network. In Multi-party systems, two or more entities collaborate to train a federated learning model together, using a variety of data sets and edge computing devices.

Five Steps of Federated Learning

First, a machine learning model is created and trained on a central server. It needs inherent logic and functionality to make sense of the insights that will eventually be generated from decentralized data sources. Once the generic model is ready, there are five steps:

Step 1: The centrally trained generic model is sent out to each device on the network and stored locally on each device.

Step 2: Each locally stored model is trained using the data generated by each individual device. The model learns and improves its performance locally.

Step 3: The devices transmit the insights from the locally stored machine learning models back to the central server. This usually happens periodically on a set schedule.

Step 4: The centralized server aggregates the insights that are received from all the devices, and updates its central machine learning model.

Step 5: The updated model is sent out and copied to each device on the network once again.

Examples of Federated Learning

Several search engines, fraud detection algorithms, and medical models use federated learning models, as do apps from Netflix, Amazon, and Google.

For example, Netflix recommends certain titles to their users based on user characteristics like age, gender, location, viewing history, and ratings of previous titles. Each user’s local machine stores new user-generated data and updates the model-based data. It then sends insights generated on the local device back to the central server.

Benefits of Federated Learning

Federated learning helps distributed devices within a learning network do a lot of data analysis locally. This has three major benefits:

1. It helps preserve each user’s data by not sharing it with other devices or with a central server.
2. It reduces the cost of sharing large sets of data with the central server as it only sends insights – and that too, periodically.
3. It helps reduce network latencies – in some cases eliminating them altogether.

Challenges of Federated Learning

Federated learning has a lot of potential, but there are a few challenges. While companies deploy powerful supercomputers to analyze data and continually optimize their machine learning models, edge devices are limited in terms of their computing power and model training capabilities. Device performance may affect model accuracy. (On the bright side, consumer devices are getting more and more powerful).

Standardizing and labeling data is also sometimes an issue in federated learning models. Centralized and supervised learning models are fed training data that has been clearly and consistently labeled. This may not always be possible to do across the numerous client devices which are part of a network. One solution is to create data pipelines that interpret a user’s actions or events, and automatically apply labels to incoming data in a standardized manner. 

Model convergence is also an issue in some federated learning models. Locally trained models converge quickly and efficiently, while federated locally trained models can take longer to converge, due to network issues, device issues, and the different ways in which consumers use applications (such as different frequency, duration, and method).

Conclusion

Federated learning is the next step in the evolution of machine learning algorithms. Companies will increasingly use federated learning to improve their models, by crunching increasing amounts of data from larger networks of devices. For consumers, federated learning could enable better services and better privacy.

Follow the Author on Twitter: Shaan Ray

Further Reading:

• The Promise of Edge Computing
• Secure Multi-Party Computation
• Fair Multi-Party Computation on the Blockchain
• Federated Byzantine Agreement

Also published at https://medium.com/lansaar/what-is-federated-learning-89a0c657d935