Visual Prompting and Adaptation of Vision Language Models for Tumor Classification in Breast Ultrasound

Vision language models (VLMs) for medical image analysis constitute an active area of research. In this work, we systematically investigated the performance of four pre-trained VLMs, consisting of one non-medical (CLIP) and three recent medical VLMs (BiomedCLIP, MedCLIP, and PubMedCLIP) for tumor classification in six breast ultrasound datasets. The VLMs' performance was investigated under (a) zero-shot and few-shot test conditions through adaptation, (b) separate label sets including different text prompts, and (c) different prompting techniques involving text and visual markers. Accuracy, sensitivity, specificity, positive predictive value, and negative predictive value were used as the evaluation measures. All models exhibited susceptibility to the description of the text prompts in zero-shot mode but improved with visual markers and adaptation. Our results established the utility of visual prompting and adaptation in improving the performance and reducing text prompt sensitivity. PubMedCLIP emerged as the most stable VLM for tumor classification in breast ultrasound.