Actors performing for the camera, as entertaining as they might be, are not usually associated with solving important problems. But these Hollywood essentials are being put to use in one medical school’s effort to correct some of the shortcomings of text-based multiple-choice exams, performance on which have a big effect on the residency Match process.
Several studies have found that exams consisting of multiple-choice questions—particularly the United States Medical Licensing Examination—are not the best or only evaluation method that should be used in postgraduate residency selection, according to Senthil Kumar Rajasekaran, MD.
He made the point during a presentation at Patient-Centered Care in the 21st Century: Health Systems Science Through the Medical Education Continuum, a student-led thematic meeting organized by members of the AMA Accelerating Change in Medical Education Consortium.
Dr. Rajasekaran is associate dean for academic affairs and professor in the physiological sciences department at Eastern Virginia Medical School (EVMS), a member of the AMA’s consortium.
He cited research showing that, as assessment measures, multiple-choice exams test knowledge. Measuring clinical skills is a different, more complex, task.
“There’s a spectrum,” he said. “Multiple-choice questions on one end. On the other end of the spectrum, if you want a total authentic view [of a student’s skills], observing a student at a bedside is the best way to predict real-world performance. But it is very difficult to [write a test question] like a real-world experience. If you want to know if a student can take care of somebody with heart failure, it’s virtually impossible for every student to be assessed at treating heart failure at the bedside.”
Videos are “valid and authentic”
The buzzwords frequently used in multiple-choice questions are infrequently used during actual patient encounters. Because of the way that such questions are written, they may be too leading and inauthentic, Dr. Rajasekaran said.
Working with the Sentara Center for Simulation and Immersive Learning at EVMS, Dr. Rajasekaran is developing video vignettes of standardized patient encounters. He believes they are a practical, effective option for assessing a student’s clinical acumen.
“The videos can be valid and authentic,” he said. “If you are careful enough in designing these videos, and having the script properly managed with these standardized patients, you can have a reliable way of depicting a clinical scenario that is more like the way our students are going to interact with patients.”
Part of creating an authentic experience is using the right language and representative visuals. To do the latter, Dr. Rajasekaran and his team partnered with Ann Kring, PhD, an expert on facial emotions recognition who works at the University of California, Berkeley. Kring helped to code the facial expressions in each standardized patient video for appropriateness.
Letting clinical cases unfold
The examples of the videos Dr. Rajasekaran presented at the conference were less than a minute long. Patients presented common conditions to an actor playing the part of a physician. After each video audience members were asked to assess the conditions presented. In terms of how learners interact with them, Dr. Rajasekaran believed the videos can be used for a variety of test formats, including short answers, long essays, guided reflection and matching.
Dr. Rajasekaran hopes to secure more funding to create enough vignettes to implement them in EVMS’ curriculum. The initial plan is to use the videos in the pre-clerkship phase of EVMS’ curriculum. He also has plans to create what he calls branching vignettes—videos that allow for multiple paths of treatment—during clerkship education.
For this project, he is teaming with Richard Van Eck, PhD, from University of North Dakota, an expert in gamification in education. Van Eck is helping develop situated gamified video vignettes in assessing clinical reasoning.
“Let’s say, in the above example of heart failure, the physician consults the patient today and then again reviews the patient in a few weeks and makes adjustments,” he said. “They might make a change in care plan by adding or modifying something. All of these developments are not captured [by multiple-choice questions]. But that’s exactly what most doctors do every day: taking care of a patient on a longitudinal basis.”
