Federated Fine-Tuning of Mixture of Experts Vision Transformers

Mixture-of-Experts (MoE) architectures decouple representational capacity from per-token computational cost: each input activates only a few specialised experts, yielding far greater capacity than dense models at equal activation cost. Federated Learning (FL) trains models on edge devices—aggregating local updates, never centralising raw data—preserving privacy and unlocking edge-resident data. Their integration is hard: cross-device FL faces tight bandwidth and memory budgets and local datasets are heterogeneous (non-IID); MoE sparsity, however, can overcome these barriers. Adapting the partial-replica paradigm of the FoMoE framework—where each client hosts only a subset of the experts—this thesis investigates the federated fine-tuning of a Vision MoE (V-MoE-S/32) on edge devices holding severely non-IID data, combined with Low-Rank Adaptation (LoRA) and Sparsity-aware Model Aggregation. Under a static expert assignment, restricting each client to half or fewer of the expert pool costs only 1–2 percentage points of Top-1 accuracy versus the full-replica baseline, while cutting the per-round communication payload by up to 28% and static on-device memory by up to 36%. A static assignment, however, mismatches the experts a client needs most with those initially allocated. To close this gap, this work designs and evaluates a novel server-orchestrated dynamic expert reassignment protocol that periodically migrates experts towards the clients demanding them most. Rather than raising the accuracy ceiling, it mainly stabilises the federation against sub-optimal initial placements, at a communication overhead of 1–3% of total traffic. Partial expert replication, parameter-efficient adaptation, and sparsity-aware aggregation thus compose into a practical recipe for fitting sparse vision models within edge memory, compute, and bandwidth budgets; by decoupling local hardware limits from model size, this design lets edge networks train far larger architectures.