Anatomy-aware disease severity detection in chest X-ray images

Automatic detection of disease in chest radiograph images is a subject of intense activity in the healthcare AI research community. However, these efforts have not adequately addressed the essential task of determining disease severity. Severity descriptors are often part of the findings in radiology reports. In this study, we report the first effort towards using anatomy-aware disease severity labels from the Chest ImaGenome dataset to train a model for accurate severity classification. We propose a multitask architecture that relies on upstream representation learning based on a detection network to extract region-specific features, followed by a learnable region-specific self-attention module that refines the features for simultaneous disease and severity classification. The two subsequent classification layers work in parallel and provide anatomy-aware disease and severity labels. We show that with this architecture we can achieve an area under ROC of 0.79 in severity detection based on anatomy.