Can students’ reasons for choosing set answers to ethical vignettes be reliably rated? Development and testing of a method

Although ethics is an important part of modern curricula, measures of students' ethical disposition have not been easy to develop. A potential method is to assess students' written justifications for selecting one option from a preset range of answers to vignettes and compare these justifications with predetermined 'expert' consensus. We describe the development of and reliability estimation for such a method -- the Ethics in Health Care Instrument (EHCI). Seven raters classified the responses of ten subjects to nine vignettes, on two occasions. The first stage of analysis involved raters' judging how consistent with consensus were subjects' justifications using generalizability theory, and then rating consensus responses on the action justification and values recognition hierarchies. The inter-rater reliability was 0.39 for the initial rating. Differential performance on questions was identified as the largest source of variance. Hence reliability was investigated also for students' total scores over the nine consensus vignettes. Rater effects were the largest source of variance identified. Examination of rater performance showed lack of rater consistency. D-studies were performed which showed acceptable reliability could nevertheless be obtained using four raters per EHCI. This study suggests that the EHCI has potential as an assessment instrument although further testing is required of all components of the methodology.