Performances of 27 MEDLINE systems tested by searches with clinical questions

OBJECTIVE: To compare the performances of online and compact-disc (CD-ROM) versions of the National Library of Medicine's (NLM) MEDLINE database. DESIGN: Analytic survey. INTERVENTION: Clinical questions were drawn from 18 searches originally conducted spontaneously by clinicians from wards and clinics who had used Grateful Med Version 4.0. Clinicians' search strategies were translated to meet the specific requirements of 13 online and 14 CD-ROM MEDLINE systems. A senior librarian and vendors' representatives constructed independent searches from the clinicians' questions. The librarian and clinician searches were run through each system, in command mode for the librarian and menu mode for clinicians, when available. Vendor searches were run through the vendors' own systems only. MAIN MEASUREMENTS: Numbers of relevant and irrelevant citations retrieved, cost (for online systems only), and time. RESULTS: Systems varied substantially for all searches, and for librarian and clinician searches separately, with respect to the numbers of relevant and irrelevant citations retrieved (p < 0.001 for both) and the cost per relevant citation (p = 0.012), but not with respect to the time per search. Based on combined rankings for the highest number of relevant and the lowest number of irrelevant citations retrieved, the SilverPlatter CD-ROM MEDLINE clinical journal subset performed best for librarian searches, while the PaperChase online system worked best for clinician searches. For cost per relevant citation retrieved, Dialog's Knowledge Index performed best for both librarian and clinician searches. CONCLUSIONS: There were substantial differences in the performances of competing MEDLINE systems, and performance was affected by search strategy, which was conceived by a librarian or by clinicians.