Development of a script concordance test using an electronic voting system

Abstract

This paper describes our experience using an electronic voting system (EVS) to develop a Script Concordance Test (SCT) to assess clinical reasoning in medical students. One hundred and fifteen questions were reviewed and voted on in a single two hour session by eleven specialist Obstetricians and Gynaecologists (“experts”). Questions were categorised by the level of agreement. The experts completed an evaluation questionnaire. In eighty-four (73%) questions experts were mostly or completely in agreement. In fifteen of these the modal response was that the information provided did not affect the hypothesis or proposed action. Responses were widely divergent in thirty-one (27%) questions. The experts felt that the SCT tested clinical reasoning, the EVS was an appropriate method for answering SCT questions, and that the SCT is a suitable method of assessment of medical students and resident staff. Experts felt that the SCT questions were difficult to write, that questions needed to be written avoiding regression to the mean, that absolute phrasing in the answer options was often inappropriate, and that more experience with writing questions were required. EVS is an efficient method for completing the development of SCT questions but has the disadvantage of requiring synchronous participation of experts.