Multi-Modal 3D Gaussian Splatting for Holistic Autonomous Driving Scene Understanding

Reusing autonomous driving datasets across sensor configurations requires synthesising observations beyond the camera poses and projection models seen during acquisition. Conventional novel view synthesis benchmarks primarily evaluate views close to the recorded trajectory and provide limited evidence of such generalisation. This thesis presents a dynamic, multi-sensor, multi-modal 3D Gaussian Splatting framework for scene reconstruction and novel view synthesis with perspective and fisheye cameras. Building on SplatAD, the framework combines a shared representation of static environments and tracked actors with native fisheye rendering and supervision from RGB images, optional LiDAR measurements, and geometric pseudo labels. An automatic labelling pipeline adapts pretrained foundation models to heterogeneous cameras, producing monocular depth maps, surface normals, semantic labels, and optical flow without additional manual annotation. Experiments on PandaSet, Argoverse 2, and KITTI-360 combine standard novel view synthesis with extreme NVS, where target views depart substantially from the training distribution in camera pose, intrinsic parameters, or projection model. Multi-modal supervision yields slightly lower standard NVS photometric accuracy than SplatAD, while substantially improving rendered depth, both with and without LiDAR. The framework also enables surface-normal rendering and achieves higher semantic segmentation mIoU than HUGSIM against pseudo labels on KITTI-360. These gains do not translate into uniformly better appearance under view extrapolation or more accurate Gaussian-centre geometry, and optical flow supervision is ineffective in the tested formulation. The results demonstrate the integration of heterogeneous camera models and multiple scene modalities, providing a basis for investigating driving data reuse under changes in sensor configuration.