Federated Neural Radiance Fields on a Swarm of Miniaturized Robots

With a few tens of grams and 10 cm in diameter, autonomous nano-drones are gaining attention in civil and industrial applications, as they can easily navigate in cluttered environments and safely fly close to humans. However, their size limits onboard sensors and computation to low-resolution cameras and sub-100 mW microcontroller units (MCUs), with a few 10s MB of onboard memory. In a swarm configuration, nano-drones can collaborate to efficiently accomplish complex missions, such as 3D object reconstruction, addressed in this thesis. State-of-the-art (SoA) approaches for this task leverage neural radiance fields (NeRF) to reconstruct a scene starting from sparse views and their camera poses. However, training NeRF models requires several GB of memory and over 100k steps to converge, making their execution aboard nano-drones challenging. In this thesis, we demonstrate how to reduce the memory requirements of SoA NeRF, i.e., Instant-NGP, to fit the strict memory budget of a nano-drone's MCU and how to distribute computation on a swarm of nano-drones. First, we perform an in-depth analysis and profiling of memory and operations required by Instant-NGP. Second, we optimize Instant-NGP's key parameters to make its memory requirements compatible with the nano-drone's constraints. Last, we leverage a distributed learning approach, i.e., federated learning (FL), to split the computational workload across multiple nano-drones. We reduce memory and operations by 90.8% and 87.9% respectively, reaching 46.02 MB and 3.5 GOPs/step while obtaining a peak signal to noise ratio (PSNR) of 26.4, only 4.1% lower than the memory-intensive Instant-NGP baseline. Furthermore, we demonstrate that by combining FL with our optimized NeRF model, four drones, each equipped with a fraction of the training set, can achieve reconstruction quality on par with the single agent. Results prove the effectiveness of the proposed strategy, opening the way for in-field deployment of our solution.