Diffusion Inpainting with Parameter Efficient Learning for the Generation of Architectural Elements

Visualizing an architectural renovation within existing spaces based on user demand poses a formidable challenge. This work presents an approach to subject-driven inpainting tailored for architectural contexts, employing diffusion models to semantically synthesize architectural elements and design styles into real-world images, guided by user-defined prompts. By employing a Denoising Diffusion Probabilistic Model (DDPM) trained on a limited dataset and employing parameter-efficient fine-tuning techniques, notably Low-Rank Adaptations (LoRA), the model achieves superior quality and semantic coherence in inpainting outcomes, preserves its core performance while reducing the number of retrained parameters. A significant contribution of this research is the introduction of a novel loss function that integrates Mean Squared Error (MSE) with the Structural Similarity Index (SSIM), enhancing DDPM training particularly in the preservation of textures, thereby establishing a replicable and adaptable training pipeline. Evaluation of the model's performance encompasses both quantitative metrics and qualitative assessments, including feedback from expert human evaluators and the obtained results demonstrate that the proposed approach outperforms existing methods in terms of both reconstruction quality and semantic coherence of the inpainting result, being computational efficiency, making it well-suited for use in low-resource environments. Overall, this work approach offers a valuable baseline for future research on guided inpainting and subject placement of architectural elements into an existing image.