Relighting Neural Radiance Fields Leveraging Shadow Mapping

NeRF (Neural Radiance Fields) belongs to the category of modern implicit 3D image reconstruction algorithms. NeRF achieves state-of-the-art novel view synthesis of complex scenes by optimizing a continuous volumetric rendering scene function. However, it cannot synthesize under unobserved light conditions. Several relighting methods have been proposed but have shown to be prohibitively expensive to train, failing to gain traction in real-world applications. ReNe (Relighting NeRF) from eyecan.ai presented a novel data set of various objects with complex geometry and a new lightweight architecture that can render real-world objects under one-light-at-time (OLAT) conditions. Any method that aims to generate a realistic scene must have geometrically accurate shadows. ReNe's lightweight architecture solves this by estimating global geometry through its visibility. A common theme of relighting is that approximations about geometry must be made to avoid expensive geometric queries. In this thesis, I propose using a classical computer graphics technique called shadow mapping to create low-cost yet convincing shadows. Shadow mapping utilizes precalculated distance maps from the viewpoint of the light to obtain an understanding of geometry. Through shadow mapping, we can create shadow hints, which are structured and geometrical scalars, to better advise shadow predictions at every point. With no architectural change, the injection of shadow hints produced more accurate shadows. We empirically tested our hybrid approach on the entire ReNe dataset, where we set new state-of-the-art results.