Use of Artificial Intelligence‐Based Platforms to Support Searches in Evidence Synthesis Studies: A Study of the EAACI Methodology Committee

OBJECTIVE: To demonstrate an evaluation of the use of artificial intelligence (AI) in supporting tasks in evidence synthesis (particularly search for primary studies) in the allergy or respirology fields. METHODS: We queried three AI-based platforms specialised on identifying scientific publications using several strategies (i.e., prompting strategies and search approaches) in November 2024 to identify primary studies related to health-related case studies in the allergy or respirology field. We compared how many eligible primary studies we identified using AI-based platforms versus in a systematic search in multiple electronic bibliographic databases. We also compared meta-analytical results obtained with the primary studies identified by querying AI-based platforms versus in the context of a systematic review. Finally, we developed a structured methodological framework and a reporting checklist for using these AI-based platforms. RESULTS: In our main case study, a systematic review of randomised controlled trials (RCTs) informing the Allergic Rhinitis and its Impact on Asthma (ARIA) guidelines, a strategy involving searching multiple AI-based platforms identified 85.7% of all full articles with DOI, but failed to identify unpublished trials registered in trial databases, resulting in an overall identification of 56.3% eligible RCTs. Meta-analytical estimates were similar when considering only the primary studies identified using AI-based strategies versus those identified by the systematic review. We observed a lower performance in the case study of systematic reviews of observational studies. CONCLUSIONS: We provide an example on how it is possible to evaluate AI-based platforms in the support of evidence synthesis, with a focus on the allergy and respirology fields.