Efficiency Analysis of Two Written Short-Answer Student Evaluation Formats

The purpose of this study was to compare the efficiency of two written short-answer student evaluation formats. Efficiency was defined in this study as reliability per unit of examination time. Twenty second-year physical therapy students enrolled in the soft tissue injuries portion of the orthopedic course completed a 12-item quiz consisting of two formats (diagnosis and response set) with six parallel content items per format. The outcome measures of interest were 1) students' score, 2) amount of time required to complete each item, and 3) amount of time required by the faculty members to grade each item. The results indicated that a greater reliability per unit of examination time was achieved for the diagnosis format than for the response set format and that the diagnosis format could be graded more rapidly and with a higher level of interrater agreement than the response set format. Students' grades on the diagnosis format items tended to be higher (although not statistically significant) than the grades on the response set format items, suggesting that when a criterion-referenced system is used, the minimal acceptable baseline score should be raised when using the diagnosis format. Further study is required to evaluate the validity of the diagnosis format.