Development of an AI-Based System for Automatic Report Generation From MRI Images

Abstract

The increasing demand for brain magnetic resonance imaging (MRI) is placing considerable pressure on radiology departments, where the manual writing of reports remains a time-consuming and cognitively demanding task. This challenge is compounded by the multimodal nature of brain MRI, which typically combines several complementary sequences — T1, T1C, T2, FLAIR, DWI and ADC —, which requires radiologists to integrate information from multiple contrasts before producing a structured report comprising ‘Findings’ and ‘Impression’ sections.

This master’s thesis explores a segmentation-free, multimodal approach to the automatic generation of brain MRI reports. The proposed system processes the MRI sequences available for each scan using a shared SwinUNETR encoder. The extracted visual tokens are enriched using learnable positional encodings and modality embeddings, then integrated by a Fusion Transformer into a unified multimodal representation. This representation is projected into the embedding space of ClinicalT5, a clinically pre-trained linguistic model, which generates the ‘Findings’ section based on the visual features, and then produces the ‘Impression’ section as part of a text-to-text synthesis task.

To overcome the memory constraints of jointly training 3D encoders with transformer-based language models, a staged training protocol was adopted, comprising encoder adaptation through disease classification, offline feature extraction, report generation training, and a light end-to-end fine-tuning step. The system was evaluated using classification metrics (accuracy, macro F1) and text generation metrics (BLEU, ROUGE, METEOR, BERTScore). Experimental results demonstrate that the proposed approach is feasible and consistently outperforms a convolutional baseline built on a ResNet encoder with MedicalNet pretrained weights. However, automatic metrics alone cannot fully capture clinical correctness, underscoring the need for expert validation.

Overall, this work demonstrates the feasibility of a segmentation-free multimodal pipeline for brain MRI report generation, combining a shared 3D visual encoder, modality-aware fusion, and a clinically pretrained language model. The system remains a research prototype, and further validation on larger, more diverse datasets with expert clinical review is required before any practical clinical deployment.