Watch the AMA's daily COVID-19 update, with insights from AMA leaders and experts about the pandemic.
Featured topic and speakers
In today’s COVID-19 Update, a discussion on next steps after the elimination of the Step 2 Clinical Skills exam—a portion of the United States Medical Licensure Examination® , or USMLE. Experts say that even though the clinical skills exam has gone away, the USMLE will still be assessing core skills that are necessary for physicians to practice.
Find out more about the exam elimination.
Learn more at the AMA COVID-19 resource center.
Speakers
- Kimberly Lomis, MD, vice president, undergraduate medical education innovations, AMA
- Michael Barone, MD, MPH, vice president, competency based assessment, National Board of Medical Examiners
- Alex Mechaber, MD, associate vice president, physician licensure programs, National Board of Medical Examiners
Transcript
Unger: Hello, this is the American Medical Association's COVID-19 Update. Today, we'll discuss next steps after the elimination of the Step 2 Clinical Skills exam, a portion of the United States Medical Licensure Examination, or USMLE and what it means for medical students. I'm joined today by Dr. Kimberly Lomis, AMA's vice president for undergraduate medical education innovations in Nashville. Dr. Michael Barone, vice president of competency based assessment at the National Board of Medical Examiners, or NBME in Philadelphia. And Dr. Alex Mechaber, associate vice president for physician licensure programs at NBME in Miami. I'm Todd Unger, AMA's chief experience officer in Chicago. Dr. Lomis, we'll ask you to start, can you first provide some context on what the Step 2 Clinical Skills exam was designed to assess and why students were unhappy with it?
Dr. Lomis: Sure Todd, thanks for that. The licensing exams are intended to ensure that our practicing physicians are ready to render safe care. And so, the clinical skills exam was set up as being distinct from the other multiple choice based exams, in that it is a performance setting in which students interact with standardized patients, so that you can better assess their ability to communicate, establish rapport, do all the right steps for good clinical diagnostic reasoning and observe that more directly than through some written examination. The challenge that that presents of course, is that that requires highly trained standardized patients, and can only be conducted at certain centers across the country. That makes it difficult for students because in addition to the price of the examination itself, they would often have to travel to get to one of those validated sites.
And that travel cost was disparate among students. For some, it was a much greater burden than others. So there was some concern and when you couple that with the fact that among U.S. allopathic graduates, the pass rate is quite high, 95% consistently, many students questioned that for the bulk of them, this felt like an unnecessary step, that there was clear evidence that their schools were adequately preparing them. During COVID everything had to be suspended because you couldn't do this in-person examination. And I think that it was an opportunity for all of the previous concerns to really be brought to the fore and step back and think through what is the true value here. And so, I think it's actually quite admirable that the NBME was able to look at the input from multiple stakeholders, recognize that the exam wasn't fulfilling all of the intended purposes and step back and reconsider. And so, it's been an important move on their part and now we have to think through what's next.
Unger: So would you say in essence, there's an enormous burden on several fronts that you pointed out and given that so many people are passing the exam, 95%, it wasn't useful an assessment essentially?
Dr. Lomis: Well, and that is for a specific group. I think we have to be careful. There's lots of different types of students who would take this exam and the pass rates are lower among other subsets of students. And so, it certainly has some value. The other challenge was for a student who didn't pass there wasn't much specifically back about what the deficit was in terms of remediating and being able to develop the skills that were necessary. So again, my colleagues comment further on what they saw to be some of the pitfalls, but those are the types of concerns that were voiced.
Unger: Dr. Barone, I know that the NBME said its commitment to performance-based assessment and clinical skills hasn't in any way been diminished with this decision. Can you talk about your newest area of focus competency based assessment?
Dr. Barone: Yeah, I'm happy to. Thanks. As Dr. Lomis said, there are skills, behaviors that people use in practice that we can't really measure in written examinations. So when we say performance-based assessment, we're looking at those skills and behaviors in different ways. A standardized patient format is just one of many potential ways. In the competency based assessment unit, we're really going to look at two things. How can we develop in collaboration with medical education and with the medical regulatory community competency assessments that are for what I call sampled competencies? Can we get to a communications' assessment that's measuring something that we value?
Can we get to an assessment of things like health system science, which I know is very important to the AMA. Outside of the knowledge of health system science, actually the practice of health system science. The second part of it is assessment is happening all the time in the workplace and in the learning place and it's happening framed on systems of milestones and competencies. Schools spend a lot of time doing assessment and we as NBME would love to participate in how to make that assessment more effective, more rigorous for whatever purpose that is. It need not be for licensure, it may be for a school's curriculum development, curriculum design, and just to have more rigor in the way they assess and advanced students.
Dr. Mechaber: And I'll just add to what Dr. Barone said, and that is the USMLE continues to provide a summit of assessment for the regulatory community for primary purpose of licensure. And as Dr. Lomis mentioned, there is a heterogeneous group of examiners who take the examination where pass rates are different. The discontinuation of the CS exam by no means means that the USMLE similarly, isn't going to still be concerned about assessing core skills that are necessary for physicians to practice these days. As part of the Step 2 CS revitalization process, we looked ... particularly looked at how do we define those clinical skills constructs that we wanted to measure and assess within the exam. And we hope to leverage that work as we continue to plan for the future and look at the USMLE exam going forward.
Unger: So it's interesting to think about how do medical students prepare for this shift and Dr. Lomis, the AMA has long focused on this area through our Accelerating Change in Medical Education Consortium. Can you talk to about competency-based assessment and how that's been playing out?
Dr. Lomis: Sure. Competency-based medical education has been a focus of the consortium for several reasons. And just for the audience who may not be that familiar with the concept. Sometimes when people hear competency-based they assume it's minimum bar, and that's not what we're talking about here. What competency-based education really is, is to focus on the desired outcome of a learning. Whereas, traditionally curricula tend to focus on what courses will be delivered, a course of study that involves certain specific activities and times. If you really flip the equation to look at outcomes, it makes total sense, but that's not how we often have done this. What we love about competency-based approaches is that it acknowledges that there are multiple different pathways to that same end game.
And so, for different students, if you can create greater agility in the programming, so that based on their prior experiences, they come in with such rich experiences coming into medical school and in many different areas. So recognizing their prior skillset and within each individual areas of relative strength and weakness, what competency-based approaches do is allow you to tailor their educational process a bit more to focus on getting each person to that end game. What it involves is much more emphasis on assessment than on delivery. We still have to deliver training, but you have to really increase the rigor of the assessment so that you can see whether each individual is on the right trajectory. And so, that's the work now to do is make sure that each school has really ramped up that competency assessment in an intentional way, targeting each individual student.
Unger: Dr. Mechaber, what other novel assessments is NBME exploring?
Dr. Mechaber: Well the USMLE for a while had been looking at ways to assess these core competencies within the existing exam series. And over the years, we've bolstered knowledge of communication skills, for example, in a number of the Step exams. Our computer based case simulation exercises that occur as part of Step 3 currently, had been looking at both diagnostic and management reasoning and some patient care skills throughout. So really our goal moving forward is to take a deep dive into what we're currently assessing within the USMLE looking at constructs like interpersonal and communication skills and some of the patient care competencies that I mentioned to look at it and see what we're currently delivering or assessing rather, and to look for ways that we can incrementally enhance those assessments moving forward. And we'll do this in a very complementary way with some of the work that you've heard that Dr. Barone is doing within the competency-based assessment unit at the NBME.
Unger: Dr. Barone, anything to add about the NBME's work in developing these novel assessments?
Dr. Barone: I will add one thing, but I just want to emphasize what Dr. Lomis said that focus on outcome orientation versus process orientation, I think is critical. And what we know about assessment is regardless of the assessment, if it looks new and shiny, or if it looks old and fairly traditional, the more we give those assessments to learners, the more they get engaged in self improvement and mastery adaptation, all the things that we're interested in as an education community and I know the AMA is particularly interested in. In terms of novelty, there are tools out there, there are tools like natural language processing that we have actually almost operationalized.
In the last two CS exams, we were ready to actually score patient nodes using natural language processing, that got put on hold as a result of COVID. And there are more immersive simulations that are available through various commercial platforms and some platforms that have been developed in education. These are the types of tools that we plan to deploy, both for those sampled competency assessments that I mentioned, but also thinking about how they could extend an educator's reach, not replace an educator, but extend their reach, to have another tool to be able to provide students with that critical feedback along their learning path.
Unger: Dr. Lomis, when we think about this change how is that going to impact the role of the medical school in ensuring the standards of clinical care for students? Are they going to have to change? What will happen differently?
Dr. Lomis: It's really important that students recognize that this in no way indicates a devaluing of these types of assessments and I think what you're hearing is actually an opportunity to increase them even more, but just do it in a different way. It now becomes the duty of the school. If we made this argument against the exam, that the schools are in a position to do this, we got to now step up. And so, is each school, do they have the appropriate resources, the faculty bandwidth to really shift this to a formative process. As Dr. Barone mentioned, we'd love this to be an opportunity to allow students to focus more on their development and their growth and their readiness for the next of moving into GME, as opposed to the pressure of passing one event in one moment of time.
And so, the schools now have the obligation to provide that feedback intermittently over time. And importantly, when they see someone who isn't progressing to the degree that they would hope, that there's an intervention to get them there. It's not to say you can't move forward, it's you need extra help in this particular area, let's get you to where you need to be. And so, hopefully the entire culture can shift from looking good on paper or performing for these single isolated high stakes events, towards really thinking about every single day we all have to improve throughout our entire career. How do we put that into place at the beginning as students, and then model it throughout our career as faculty to show that that growth continues and doesn't stop once you graduate.
Unger: Dr. Barone, what are some of the issues with such assessment being based solely in individual schools without a more centralized, standardized approach?
Dr. Barone: Yeah, Todd, sometimes I ask myself the question and I think it's a question for the broader community of education and regulation. And when I say regulation, I mean, the way physicians are licensed in the United States. Of course, we have states and territories that do this independently. And that question is what conveys with graduation from the U.S. school? What conveys with graduation from an international school? In the United States, both for osteopathic physicians and for MD physicians, we have an independent audit, which are the licensing examinations.
So that's how USMLE is set up and Dr. Mechaber talked about some of the work within USMLE. I think other assessments can be used in support of education and educators making good inferences to say, "This is somebody who's competent and ready, or maybe even proficient" and how that information is transmitted across the UME to GME continuum or potentially to licensure. Although at this point, licensure does remain an independent audit. That's really that opportunity as we get more confident than these assessments, as we work through some of the perceived conflicts of interest. I think we have a bright future.
Unger: Dr. Mechaber, when we obviously think about the last year and the need for our physicians to be prepared, to respond to health emergencies like we've seen the pandemic. How does that translate back into these competencies in the training for medical students?
Dr. Mechaber: Sure. The pandemic, I think forced all medical schools in particular and the health care system in particular to really adapt to health care needs of the population. By that, I mean really embracing the virtual environment, embracing technology, embracing telehealth and telemedicine. And I think medical schools really had to begin to prepare trainees for those skills. And so, I think as part of the assessment community, our responsibility will be to provide those tools to assess those new skills that are needed for the future and provide tools for the medical schools to be able to perhaps even teach it and provide the formative assessment more effectively as well.
Unger: Last question, love to hear from all of you. Dr. Barone, why don't you start first and talk about just immediate and longer term steps from what we've discussed today and provide a timeframe for them.
Dr. Barone: I think in terms of immediate steps, I'll talk about what we plan in our competency-based assessment unit at NBME. We're seeing this as an opportunity as has been mentioned, we're going to spend the next couple of months really exploring where our priorities are, who our collaborators can be. And then embark on a series of projects. Some of these projects will be focused on bringing an assessment into the community in a pilot way. Some of them may be focused on research. For example, there are schools that are working on assessment frameworks, like entrustable professional activities. Is that an area we want to get involved in? All that prioritization is to come. I also think this is coinciding with the Coalition for Physician Accountability's work on that UME to GME continuum and what competencies and assessments are necessary to make that handoff more of a smooth and warm handoff rather than the cliff that some of us describe it now.
Dr. Mechaber: And I'll just add from the USMLE perspective as I mentioned, we're going to look at ways to enhance clinical skills assessment within the series, think about that moving forward, think about developmental priorities in terms of what that exam looks like and perhaps even some right sizing of the exam as we ...
Dr. Lomis: And I'll chime in. I think many students remain concerned about the disruptions related to COVID-19 and particularly how it impacts their transition into GME and the upcoming selection cycle in 2021 for residency positions. And there is great variability now of some students have certain types of tests and others don't. And so, really working with the Coalition for Physician Accountability, which has been mentioned, the AMA has strong representation throughout all of the committees of that coalition, and making sure that residency programs are mindful of this diversity of what information is available and how do we make the process as equitable as possible.
And so, the coalition has a committee that's specifically addressing the upcoming cycle, as well as the longer term work that was mentioned by Dr. Barone on the bigger issues surrounding the UME to GME transition. And so, we are fortunate at the AMA to have a voice in all of these conversations and bring the voice of our section from our students, as well as residents as input into these discussions. And we do have compiled resources available to our educators to follow all this complexity as we continue to struggle to respond to the disruptions related to COVID-19.
Unger: If there are medical students or resident physicians looking for more resources, where might they find them Dr. Lomis?
Dr. Lomis: Yeah, we have a great resource page on the AMA website. We have links there to external bodies reports as well as resources that we've created ourselves to help people.
Unger: Excellent. That's at ama-assn.org/covid-19. I want to thank you, Doctors Lomis, Barone and Mechaber for being here today and sharing your perspective and this important information. That's it for today's COVID-19 Update. We'll be back with another segment tomorrow. Thanks for joining us. Please take care.
Disclaimer: The viewpoints expressed in this video are those of the participants and/or do not necessarily reflect the views and policies of the AMA.
