Imagine a car that learned to drive on sunny afternoons, suddenly driving through a rainy night in a city it has never seen. Headlights glare on wet asphalt, raindrops blur the camera, and the traffic looks different. A human driver adapts within minutes. For a car’s perception system, this jump can be enough to miss vehicles and pedestrians it would otherwise spot with ease.
Within the Cynergy4MIE project, TU Delft is working on ways for vehicles in very different environments to learn from each other, without ever sharing their raw sensor data.
Why new conditions are hard
To drive safely, a vehicle must know where every car, cyclist and pedestrian around it is. It does this by combining cameras with LiDAR, a laser sensor that builds a 3D picture of the surroundings, and letting an AI model interpret the result.
That model is only as good as the examples it learned from. When the time of day, the weather or the city changes, performance can drop sharply. Collecting more examples sounds like the obvious fix, but rare conditions are rare and labeling recordings is costly. Combining recordings from many vehicles would help, yet pooling them in one place is rarely an option. The recordings capture people, license plates and routes, which raises serious privacy concerns. They are also enormous: a single test vehicle can produce terabytes of camera and LiDAR data, which is slow and expensive to move.
Federated learning: sharing lessons, not data
Federated learning offers a way out. Think of chefs in different restaurants improving a shared recipe. Nobody mails their ingredients or customer orders around. Each chef cooks in their own kitchen and sends only their adjustments to a coordinator, who combines them into a better recipe and sends it back.
In the same way, each vehicle or fleet trains a shared AI model on its own data and sends back only the updated model. A central server combines these updates and redistributes the improved model. The raw sensor data never leaves its owner.
The catch: one model doesn’t fit all
Federated learning works best when all participants see similar data. Real driving is not like that: one fleet drives in bright daylight, another at night, a third mostly in the rain, and some have far less data than others. This mismatch is called heterogeneity.
Forcing such different participants to agree on a single model produces a compromise that is decent everywhere but excellent nowhere. A common remedy is to let each participant keep part of the model to itself. But which part? Most existing methods fix this in advance, for example “always keep the last step local”, even though the parts that need adapting can differ from one environment to another.
Our approach: let the model decide what to share
An AI model is built from many layers, each turning sensor signals into a richer understanding of the scene. Our idea is to check, layer by layer, whether a participant’s locally trained model still “sees” things the same way as the shared model.
- Layers that still behave like the global model hold knowledge useful to everyone, so they are shared.
- Layers that have drifted far away have likely adapted to something local, such as darkness or rain, so they are kept local.
This check is repeated throughout training, so a layer can rejoin the shared model later, and each participant can make its own choices.
