Representation Learning with a Transformer-Based Detection Model for Localized Chest X-Ray Disease and Progression Detection

Medical image interpretation often encompasses diverse tasks, yet prevailing AI approaches predominantly favor end-to-end image-to-text models for automatic chest X-ray reading and analysis, often overlooking critical components of radiology reports. At the same time, employing separate models for related but distinct tasks leads to computational overhead and the inability to harness the benefits of shared data abstractions. In this work, we introduce a framework for chest X-ray interpretation, utilizing a Transformer-based object detection model trained on abundant data for learning localized representations. Our model achieves a mean average precision of ∼$$\sim $$94% in identifying semantically meaningful anatomical regions, facilitating downstream tasks, namely localized disease detection and localized progression monitoring. Our approach also yields competitive results in localized disease detection, with an average ROC 89.1%$$89.1\%$$ over 9 diseases. In addition, to the best of our knowledge, our work is the first to tackle localized disease progression monitoring, with the proposed model being able to track changes in specific regions of interest (RoIs) with an average accuracy ∼$$\sim $$67% and average F1 score of ∼$$\sim $$71%. Code is available at https://github.com/McMasterAIHLab/CheXDetector.