Methods for assessing the competence of physicians' use of MEDLINE with GRATEFUL MED

Methods are described for evaluation of the competence of clinicians in using MEDLINE with GRATEFUL MED, and their application in a trial of self-service MEDLINE access in clinical settings is assessed. The methods are based on analysis of automatic microcomputer monitoring of search transactions followed by postsearch interviews. Measurements include: determination of patterns of use; self-reports of user satisfaction; self-reports of the effect of citations on clinical decisions and patient care; monitoring the citation of literature in patients' charts; independent, expert rating of the relevance of retrieved citations; and evaluation of the recall and precision of novice clinical end-users in comparison with clinicians more experienced in online searching and with reference librarians. The performance of these measures is described, and statistical issues related to the analysis of the nonparametric data produced from search comparisons are explored.