Evaluating measurement variability in clinical investigations: the case of ultrasonic estimation of urinary bladder volume

OBJECTIVE: To evaluate the quality of studies seeking to establish measurement properties (reliability and validity) of ultrasonic estimation of urinary bladder volume. DESIGN: Online searching of the MEDLINE database between 1966 and 1995, and scanning of bibliography of known studies on ultrasonic bladder volume estimation. Study selection and study quality assessment were performed independently by two reviewers. Each article was evaluated for suitability of the reference standard, adequacy of reported blinding of the observers and appropriateness of the statistical index of concordance. The last two of these guidelines were applied to reliability studies (evaluating the relation among observed ultrasonic estimations), and all three guidelines were applied to validity studies (evaluating the relation of ultrasonic estimation with a definitive measurement). POPULATION: One hundred and twenty-five participants enrolled in the five reliability studies and 769 participants in the 27 validity studies selected for appraisal of their quality. MAIN OUTCOME MEASURE: Rate of study compliance with preset criteria for high quality. RESULTS: None of the studies complied with all of the criteria for high methodologic quality. In the five reliability studies, investigators did not report adequate blinding of observers in three (60%) and an appropriate index of reliability was not used in any. Among the 27 validity studies, there was a lack of a suitable reference standard in 6 (22%), an inadequate blinding in 25 (93%), and an inappropriate index of validity in all (100%). CONCLUSION: Based on our guidelines for quality assessment, a large proportion of studies on measurement properties was found to have inadequate methods, raising concern about the credibility of the reliability and validity estimates reported. These deficiencies highlight the lack of rigour employed in the design, conduct and analysis of reliability and validity studies, which has the potential for leading to patient mismanagement due to biases in the assessment of measurement variability in clinical investigations.