Modern vision–language models and unified multimodal models can handle tasks such as visual question answering, captioning, and image–text reasoning. When the training data needed to adapt these models are distributed across institutions that cannot share raw records, federated learning coordinates training across sites but raises two practical questions: which parts of the model state should cross the network, and how should large updates be transferred and aggregated efficiently?
This article explains how NVIDIA FLARE addresses those questions—by externalizing large objects, streaming tensors, and offloading aggregation to disk—and presents FedUMM, a concrete example (a William & Mary and NVIDIA collaboration) that federates lightweight LoRA adapters over a frozen BLIP backbone.
Why VLMs are hard to federate
In a centralized experiment, images, captions, VQA examples and generation prompts flow into a single training pipeline. In federated settings these examples are split across clients with different task and modality mixes and different operational constraints. That creates two engineering challenges:
- Define what each client updates and how component-level updates from heterogeneous clients are combined; some approaches share distilled knowledge, others freeze a pretrained backbone and aggregate only small trainable modules.
- Large full-model updates are expensive to serialize, transmit, and hold in memory on the server during aggregation.
Examples: CreamFL illustrates knowledge-distillation approaches, while FedCLIP, FedPIA and FedUMM illustrate adapter-orientated, backbone-frozen strategies.
NVIDIA FLARE: coordination and extensibility
NVIDIA FLARE is an open-source, extensible Python SDK and framework for federated learning and collaborative computing. A typical FLARE job separates global coordination (server scheduling rounds and aggregating updates) from local execution (clients training or evaluating on local data). Site-specific preprocessing, prompt construction, and batching remain on the client.
The FLARE Recipe API offers concise building blocks. A FedAvg recipe pairs a model with a client training script; the same recipe can run in simulation or across real provisioned sites. Early in design, teams should define the client update contract: what stays local, what the client may return, which components it may update, and what metrics are reported. If different clients update different components, the contract should spell out how those component-level updates combine.
Moving and aggregating large model updates
A baseline approach is to fine-tune and aggregate full model parameters, but that creates two memory pressures: serializing/transferring large updates, and holding many client updates in server memory during aggregation. FLARE provides several ways to handle this:
-
Externalize large objects: FLARE can replace large objects in control messages with lightweight references and transfer the data separately, keeping control messages small and enabling payloads larger than typical serialized-message limits. Built-in decomposers handle PyTorch tensors, NumPy arrays and common FLARE structures.
-
Stream tensors: For PyTorch workflows, the FLARE Tensor Downloader streams tensors incrementally with a pull-based protocol so only a requested chunk is serialized at a time, reducing peak memory during model distribution. Chunk size can be tuned. TensorFlow workflows use traditional serialization.
-
Offload aggregation to disk: Streaming lowers transfer memory peaks, but the server may still need to aggregate many client updates. In NVIDIA FLARE 2.8.0, tensor disk offload writes incoming PyTorch FedAvg updates to temporary safetensors files and loads them as needed, preventing the server CPU memory from growing linearly with client count.
These mechanisms complement payload reduction from adapter-based training and allow scenarios such as full-model training, larger adapters, or federated learning with many clients.
FedUMM: federating lightweight adapters over a frozen VLM
FedUMM (William & Mary and NVIDIA collaboration) shows how to minimize what crosses the network using NVIDIA FLARE. In the FedUMM setup each simulated client keeps a frozen BLIP backbone locally and trains LoRA adapters on its data. FLARE coordinates rounds and aggregates only the adapter updates.
FedUMM is designed for generality with modality-specific encoders for vision, audio and text, though current experiments focus on vision–language. Evaluations used VQA v2 and GenEval under Dirichlet-controlled heterogeneity with up to 16 simulated clients.
Key results: in an eight-client comparison, adapter-only federation reduced per-client communication per round from 28.6 GB to 0.094 GB and improved VQA v2 by 0.7 points compared to full-model FedAvg. At eight clients, performance was roughly 97% of the centralized reference on both benchmarks.
The evaluations use simulated sites, synthetic partitions and public general-domain benchmarks. They do not claim clinical performance or formal privacy guarantees; they show that raw training data remain local within the simulated federated workflow.
FedUMM reduces system burden by exchanging only small LoRA adapters. When larger updates are unavoidable, FLARE’s tensor streaming reduces memory pressure during transfer; when aggregation must handle many clients, disk-backed aggregation limits memory use.
Practical checklist for federated multimodal workflows
When designing a federated multimodal workflow consider both what each client should contribute and how updates move through the system:
- Define the update contract: decide what remains local, what each client sends, and how updates are combined.
- Minimize the payload: exchange lightweight adapters when possible; use full-model updates only when required by the task.
- Choose movement and aggregation methods: use externalization and tensor streaming for large in-memory updates, and disk offload on the server when aggregating many updates would exceed memory.
- Evaluate end-to-end: measure model quality together with communication, runtime, memory use, data heterogeneity and failures.
How to get started
Begin by defining the update contract for your workflow and use the NVIDIA FLARE Recipe API to implement and validate the workflow in simulation at the expected client count. Select the payload-handling mechanisms that match your bottleneck—large-object externalization and FLARE Tensor Downloader for large model updates, plus tensor disk offload when server-side aggregation memory is constrained.
For an adapter-based concrete example, explore the paper “FedUMM: A General Framework for Federated Learning with Unified Multimodal Models” and its implementation in the NVIDIA FLARE repository. After establishing a baseline, Auto-FL can help adapt and tune the federated experiment to your datasets and tasks.
To learn more, NVIDIA is hosting NVIDIA Flare Day 2026, a free online event that explores federated learning applications across industries.
