Residency program administrators have long sought more holistic ways of screening applicants, and augmented intelligence (AI) has the potential to provide groundbreaking insights into who’s a good fit by finding patterns in very complex data. But AI—often called artificial intelligence—is only as good as the data that’s put into it, and it can easily be overrun by the same biases that plague the conventional human review process.
A workshop at the inaugural AMA GME Innovations Summit featured presentations by three experts in residency admissions who summarized the state of the art in machine-learning approaches to applicant screening, as well as what needs to be done to ensure holistic approaches are equitable and result in better care for underserved populations.
The need for better methods
One of the challenges underlying residency applicant screening is simply the volume of applications.
“There is a huge pile of residency applicants for every available position,” said Jesse Burk-Rafel, MD, an internal medicine hospitalist at New York University Langone Health, leaving “a whole team of beleaguered program directors all exhausted and frustrated.”
Program directors have therefore taken to using simplistic screening measures—such as USMLE Step 1 scores, clinical honors or Alpha Omega Alpha membership—just to get down to a manageable number of applications. But many of these are not predictive of success in residency or in practice.
“The main thing that we've seen is that these screening tools are innately biased,” said Ben Drum, MD, a third-year resident in the internal medicine and pediatrics program at the University of Utah School of Medicine. For example, “we know that Step 1 scores are associated with socioeconomic privilege. … I had friends who had spent thousands of dollars in order to try and get the best Step 1 score possible, and some medical students are unwilling to go into that much more debt or just don't have that financial capability.”
Learn more about the AMA Reimagining Residency initiative.
Making the best use of AI
While machine-learning screening algorithms can look at hundreds of characteristics to determine what makes a successful resident, they too can suffer from bias, the presenter noted.
“Artificial intelligence learns from what it's trained on,” Dr. Burk-Rafel said. “So if it's a biased process, if there are measures in there … that are known to be biased, such as the USMLE, it will perpetuate [them].”
In addition, as applicants learn more about the algorithms, some may try to game the system—such as by loading up their applications with desirable keywords.
Residency programs can put a check on these shortcomings by incorporating more diversity in their human reviewers, said Charlene Green, director of the office of admissions and the office of student and resident diversity at UC Davis School of Medicine. Nurses, medical assistants, social workers and even campus police chiefs might all contribute valuable insights on applicants.
“Everyone has bias, right?” Green said. “When you have non-diverse people selecting—trying to select—diverse people, you miss some of the things that maybe you don't understand about identity and background.”
The demographics of the U.S. patient population demand a more inclusive approach to screening, she added, noting that the U.S. population in 2019 was 18.3% Hispanic and 13% African American, yet only 5.3% of active residents identified as Hispanic and just 4.4% identified as African American.
Finding new ways of valuing applicants’ backgrounds and the experiences they bring to the table “will help us achieve ultimately what we're hoping to focus on, which is patient outcomes and those patients who currently aren't being served as well as others,” Green said.
Check out an episode of the “AMA COVID-19 Update” on how the pandemic has changed the residency application process.
Drs. Burk-Rafel and Drum also talked specifically about their programs’ machine learning screening projects, including details of development and next steps, and Ms. Green detailed UC Davis’ clinical tailored pathway programs to recruit learners to work in underserved communities.
The AMA GME Innovations Summit also featured workshops on boosting resident well-being, eliminating workforce shortages and improving residents’ clinical teaching skills.
