How Can I Know What I Don't Know? Poor Self Assessment in a Well-Defined Domain

As the rapidity with which medical knowledge isgenerated and disseminated becomes amplified,an increasing emphasis has been placed on theneed for physicians to develop the skillsnecessary for life-long learning. One suchskill is the ability to evaluate one's owndeficiencies. A ubiquitous finding in thestudy of self-assessment, however, is thatself-ratings are poorly correlated with otherperformance measures. Still, many educatorsview the ability to recognize and communicateone's deficiencies as an important component ofadult learning. As a result, two studies havebeen performed in an attempt to improve uponthis status quo.First, we tried to re-define the limits withinwhich self-assessments should be used, usingRosenblit and Keil's argument that calibrationbetween perceived and actual performance willbe better within taxonomies that are regularlytested (e.g., factual knowledge) compared tothose that are not (e.g., conceptualknowledge). Second, we tried to norm referenceindividuals based on both the performance oftheir colleagues and their own historicalperformance on McMaster's Personal ProgressInventory (a multiple choice question test ofmedical knowledge).While it appears that students are able to (a)make macro-level self-assessments (i.e., torecognize that third year students typicallyoutperform first year students), and (b) judgetheir performance relatively accurately afterthe fact, students were unable to predict thepercentage of questions they would answercorrectly with a testing procedure in whichthey have had a substantial amount of feedback. Previous test score was a much better predictorof current test performance than wereindividuals' expectations.