End-to-End Vision Language Processing of Identity Documents: Detection, Classification and Extraction

The efficient processing of identity documents is essential in sectors such as finance, healthcare and government. Traditional manual methods are often slow and susceptible to errors. This thesis develops an end-to-end pipeline that integrates object detection, image classification and vision-language models to automate the processing of identity documents, including detection, classification and data extraction from images. This work was conducted in two distinct phases: the first phase, conducted during an industry internship, focused on detection and classification using state-of-the-art models, such as RTMDet, YOLOX and EfficientNet, achieving near-perfect accuracy across various document types. The second phase, carried out in an academic setting, leveraged advanced vision language models, such as Donut, Qwen2 and LayoutLMv3, to extract structured information directly from document images. These models demonstrated remarkable accuracy in text extraction tasks, even in challenging real-world scenarios. By integrating object detection, classification and vision-language approaches, this thesis highlights the potential of combining visual and linguistic processing for scalable, efficient and precise document analysis.