Use of Critical Items in Determining Point-of-Care Ultrasound Competence

We previously developed a workplace-based tool for assessing point of care ultrasound (POCUS) skills and used a modified Delphi technique to identify critical items (those that learners must successfully complete to be considered competent). We performed a standard setting procedure to determine cut scores for the full tool and a focused critical item tool. This study compared ratings by 24 experts on the two checklists versus a global entrustability rating. All experts assessed three videos showing an actor performing a POCUS exam on a patient. The performances were designed to show a range of competences and one included potentially critical errors. Interrater reliability for the critical item tool was higher than for the full tool (intraclass correlation coefficient = 0.84 [95% confidence interval [CI] 0.42-0.99] vs. 0.78 [95% CI 0.25-0.99]). Agreement with global ratings of competence was higher for the critical item tool (κ = 0.71 [95% CI 0.55-0.88] vs 0.48 [95% CI 0.30-0.67]). Although sensitivity was higher for the full tool (85.4% [95% CI 72.2-93.9%] vs. 81.3% [95% CI 67.5-91.1%]), specificity was higher for the critical item tool (70.8% [95% CI 48.9-87.4%] vs. 29.2% [95% CI 12.6-51.1%]). We recommend the use of critical item checklists for the assessment of POCUS competence.