AMA STEPS Forward® podcast
What EHR Metadata Tell Us About Team-Based Interventions for Inbox Reduction
Dec 2, 2023
Featured topic and speakers
Host Christine Sinsky, MD, AMA vice president of professional satisfaction, and guest A Jay Holmgren, PhD, assistant professor of medicine at UC San Francisco and the Center for Clinical Informatics and Improvement Research, discuss how assessing Epic Signal data and other EHR metadata can be a useful tool for evaluating team-based interventions to reduce inbox volume.
- Get the EHR Inbox Reduction Checklist (PDF)
- Explore the EHR Inbox Reduction toolkit
Explore the research mentioned in today’s episode:
- Assessment of Electronic Health Record Use Between U.S. and Non-U.S. Health Systems
- Assessing the Impact of the COVID-19 Pandemic on Clinician Ambulatory Electronic Health Record Use
- Physician Note Composition Patterns and Time on the EHR Across Specialty Types: A National, Cross-Sectional Study
- Primary Care Physician Gender and Electronic Health Record Workload
Speaker
- A Jay Holmgren, PhD, assistant professor of medicine, University of California—San Francisco
Host
- Christine Sinsky, MD, vice president of professional satisfaction, American Medical Association
Listen to the episode on the go on Apple Podcasts, Spotify or anywhere podcasts are available.
Transcript
Speaker: Hello, and welcome to the AMA STEPS Forward® podcast series. We'll hear from health care leaders nationwide about real-world solutions to the challenges that practices are confronting today. Solutions that help put the joy back into medicine. AMA STEPS Forward® program is open access and free to all at stepsforward.org.
Dr. Sinsky: Welcome everyone to this week’s STEPS Forward® podcast. I’m Dr. Christine Sinsky, the vice president of professional satisfaction at the AMA, and I’ll be your host today. Today we’ll be discussing how EHR event log data can be used to understand and therefore improve the clinical work environment for physicians and their teams. And we’ll be focusing our discussion on the inbox.
And for that reason, it’s my great pleasure to have as our guest one of the nation’s foremost researchers in using EHR event log data. His research has been instrumental in helping health system leaders and others understand the nature of the work physicians face with the EHR broadly and with the inbox in particular.
Dr. A Jay Holmgren is an assistant professor of medicine in the Department of Medicine and Center for Clinical Informatics and Improvement Research at the University of California—San Francisco. His research has been published in top-tier peer-reviewed journals and in the popular press, including The New York Times, The Atlantic, Politico and NPR. We’ll be talking about some of his research here today. Dr. Holmgren received his PhD in health policy in 2021 from Harvard University and the Harvard Business School. Welcome, Dr. Holmgren.
Dr. Holmgren: Thank you so much for having me. Thrilled to be here.
Dr. Sinsky: I’m so glad you could join. So let’s set the stage, A Jay, by bringing in the voice of the physician. It’s been said that doctors don’t leave their jobs, they leave their inboxes. As a researcher who’s led some of the most important research on the EHR in the inbox, can you help us understand what’s going on?
And I thought we’d start with the pre-pandemic era. In 2020, you published a study comparing U.S. physicians to physicians in other countries, all of whom used the same vendor’s EHR. Could you describe the data source that you used and why it was useful?
Dr. Holmgren: Absolutely. So this study really came about because we have long known that the technology alone is not the reason that there’s so much work that physicians are required to do in the electronic health record. There are all sorts of things that they do that are required, not because of some feature of a software, but because of some feature of the policy environment or the payment model they practice in or what their organization asks them to do.
So to study this, we took what was at the time of very newly available type of data—which you mentioned earlier—EHR event log data, which is the metadata that is created when physicians or other users of the electronic health record click on something, document something, move between different tabs within the EHR. Every one of those creates a unique digital footprint that says, “Here is the exact time that this user did this exact task in this patient’s record,” which is an incredibly rich data source for researchers.
And as this became more broadly available, we sort of partnered with Epic Systems, who is the largest EHR vendor in the United States that has clients all over the world, and pitched them a study idea where we said, “We’re really curious about how physicians in the United States compare in their EHR use to physicians around the world, those who are practicing in Europe, in Australia and New Zealand, and in the Middle East,” because they have clients in those locations.
And so we came to them with a study proposal and asked, “Can we get data from both your U.S. clients as well as your international clients and compare?” Because there are so many things that are unique to practicing medicine in the United States. And if you’re familiar at all in the health policy world, as I’m sure many of our listeners are, the U.S. is an outlier both in terms of how much money we spend and our particular vagaries of our payment system, as well as an outlier for how much health we get for that money, which is unfortunately an outlier in the wrong direction.
So when we use this data source, which is Epic’s Signal dataset, which is a metadata aggregation tool that they have built, we’re able to see all sorts of really interesting differences between U.S. physicians and physicians in their international peer countries. And I think that really showed up most specifically in this overall EHR time. And I think this particular figure really sums everything up, which is that the median U.S. physician, someone who is right in the middle of how much time they spend in the EHR, would be equal to the non-U.S. physician at the 99th percentile. So the average doc in the U.S. is spending as much time in the EHR as the absolute most EHR-burdened physicians around the world. And I think that’s a really shocking statistic.
Dr. Sinsky: I think so too, and I think this was just a brilliant use of a natural experiment, if you will, to compare U.S. versus non-U.S. physicians because it’s not all the design of the EHR, right? It’s decisions at implementation and regulation, and what the EHR is used for. I wonder if you could tell us what you found in this study specifically around the EHR inbox.
Dr. Holmgren: Absolutely. One of our most salient findings here was that U.S. physicians received three times more inbox messages compared to their international peers. And I think that’s a big difference and it’s driven by a couple of things. The first is that the U.S. was a much earlier adopter of patient portals relative to many international countries. So U.S. patients have had the ability to message their physician via the portal for much longer, and so that’s sort of disseminated amongst patients a little bit more.
The second is that the U.S. has a lot more system-generated messages, prescription messages, et cetera, all of these things, results messages, that just show up because the organization has decided that the primary care doctor needs to be CC’d on any correspondence with a specialist or all results need to go to this many physicians on the care team. And the end result is that U.S. physicians spend significantly more time dealing with the in-basket compared to their international peers, almost three times as much in terms of in-basket time per day.
Dr. Sinsky: And at the AMA, we’re working to help people find ways to go upstream and reduce some of that inbox volume, reduce some of those system messages. So we suggest instituting policies, you order it, you own it, you don’t CC every other doctor, you don’t CC the physician. And we recommend “DC the CC.” Stop CCing your notes just because you can, only when you have a specific request of the receiving physician.
So I wanted to go further in that study because I looked at it before our conversation and I was surprised that the inbox time in the U.S. was only 13 minutes a day. That just doesn’t have face validity. How do we make sense of that?
Dr. Holmgren: Absolutely, and I think this is a really great story of how a dataset has evolved over time and has gotten more accurate and allows us to better understand the dynamics, because I think if you look at more recent papers that have used the same Epic Signal data, you’ll get very different numbers.
And so the reason that we have this 13 minutes a day is, first of all, this was pre-pandemic and I think we’ll get into this soon, but there was a significant rise in inbox work that came with the COVID-19 pandemic onset. The second is that this is a number that is 13 minutes per active day, which means any day that you log into the EHR counts as an active day within a month. So it may be that physicians were doing an hour of inbox work every single day that they were seeing patients, but then sometimes they’d log into the EHR on a Saturday and just look up a few things, or maybe they were logging into the EHR because they also had jobs doing research or other administrative tasks and they wouldn’t do inbox work.
So this was one of the earliest sort of issues we ran into as Signal data, which is, what is the correct denominator? How should we be normalizing this sort of time-based measure? Is it by the number of days they were scheduled to see patients? Is it by the number of patients they saw? And I think that there’s still value in doing multiple versions of this, but this was one of the earliest studies where we had this data in a cross-institutional way, and at the time, active day was the only denominator we had.
And then the final thing is that these are all physicians at these organizations that provided any ambulatory care during the entire year of 2019. And while primary care physicians spend a much larger amount of time on the inbox, as do some certain medical specialties, such as medical oncologists spend a lot of time in there as well, your surgical colleagues are spending a significant less amount of time in the inbox, especially prior to the pandemic. So this is sort of grouping everyone together in a way that masks the true burden on the most burdened physicians, those who are really spending a lot of time in the inbox.
Dr. Sinsky: So the bottom line still is that using the same methodology, flat as it might be for estimating inbox time, U.S. physicians were spending three times as much time per day as our international colleagues.
I might also say, and I know you know this well, that the clock stops after five seconds of no mouse movement or keyboard movement. So if you’re reading a note from a patient and you might be spending 60 to 90 seconds on it, most of that time doesn’t count because you haven’t moved the mouse. And if you’re on the inbox screen and then you go off to look at a lab or you go back to read the last note, all that time off the inbox screen doesn’t count as well.
Well, you’re right. That first study, that looked at pre-pandemic times, but you studied trends in the inbox burden that crosses over the time of the pandemic, and you particularly looked at one kind of inbox message, the patient medical advice request. Can you tell us what you found?
Dr. Holmgren: Absolutely. I think one of the most shocking findings I’ve ever seen come out of my research, and I’ve been doing this for almost a decade now, is that we saw an immediate and dramatic increase in the number of patient medical advice requests, which are messages specifically asking for clinical advice or clinical care and not including messages like, “Oh, I filled out the survey,” or “I want to make an appointment,” or “Can I get a refill request?”
With a 57% increase right at the onset of the COVID-19 pandemic, right in March of 2020. And I think one of the most concerning factors going forward is that this never went down. So a lot of things disrupted the delivery of ambulatory care, especially right in that March/April of 2020 period. But many of them returned to normal. By the end of summer, beginning of fall 2020, we started to see a new normal appear as both telemedicine and in-person care were broadly available to patients or health care providers and physicians sort of figured out what they were going to do, how they were going to do masking protocols.
But that high level of patient medical advice requests of patients saying, “Hey, I can contact my physician through the portal and I want to be able to do this,” has stayed. And we’ve been doing some additional research that this has continued to be true through the end of 2022.
So it looks like this is also part of our new normal, where we have an extremely elevated level of patient demand for this type of asynchronous messaging care.
Dr. Sinsky: Right. We’ve met a demand, we’ve met a need for patients to have this between-visit communication, but we haven’t built the systems around that to make that workload manageable for physicians. And I’ve seen data that maybe 10 to 30% of inbox messages are patient medical advice requests, but the time per message is much greater for the medical advice requests than it is for others. So the proportional burden is higher because of how important these messages are and how much time they take to resolve safely.
Dr. Holmgren: That’s absolutely the case. And in the same study, we looked at what each medical advice request message costs in terms of physician time, and it’s somewhere between three to five minutes just in writing the reply in the inbox, not including any time thinking about that message, going back into the patient’s chart or their previous note to look up history. So that’s likely also an underestimate.
So when we see a big increase and they’re the most costly message in terms of physician time, we know that’s a pretty serious additional workload on the physician workforce, which is already doing a lot of EHR-based work.
Dr. Sinsky: And it may be an underestimate because as you know from some of the research summit meetings that we’ve held, other research suggests that we need to multiply that number from signal for minutes by a factor of two or three to get the actual real-time that might be being spent. So if it’s five minutes per message on the inbox screen, maybe it’s more like 15.
Dr. Holmgren: Absolutely.
Dr. Sinsky: You’ve also studied these medical advice request trends and how they vary by gender and specialty, and how the course of those trends has varied by gender and specialty. Can you tell us what you’ve learned there?
Dr. Holmgren: Absolutely. So when we look at things by specialty, we consistently find that primary care physicians are by far the most burdened with the inbox, which I think makes a lot of sense. They are oftentimes quarterbacking the care of a patient, and they’re the ones that a patient goes to first and says, “Hey, I have this problem.” Their patients might be in some sort of managed care insurer platform where they need a referral from their primary care physician to see a specialist. And just in general, a PCP is the first person you go [to] when you’re having a health event. And a lot of these messages are messages asking for diagnostics, saying, “Hey, do I need to come in for this? Should I go to urgent care or can I wait until you have a next appointment slot?”
There are several medical specialties which also get a lot of patient medical advice request messages. I mentioned earlier the role of medical oncologists. These are physicians who are often taking over the primary care role for their patients. And several other medical specialists, especially those dealing with patients who have multiple chronic conditions, get a lot of patient medical advice request records, especially compared to their procedural or surgical colleagues. Dermatologists get relatively few messages compared to those primary care physicians.
The other and enduring effect that we see is that female physicians receive significantly more messages from their patients than male physicians do. And this was true pre-pandemic. And we recently published a study in JAMIA with my colleague Lisa Rotenstein, who is a primary care physician who will be joining UCSF from Harvard in the fall, that found that COVID actually exacerbated this. So when we saw that big increase in March of 2020, that fell the hardest on female physicians who also do more EHR time in general.
So the burden of inbox messages is not evenly distributed. It falls the hardest on primary care physicians, several medical specialists, and female physicians by far the hardest.
Dr. Sinsky: There’s a wrinkle in here that’s come from other colleagues’ research that I think is interesting, and that is physicians who are full-time clinical who then go down to 0.8 clinical actually end up increasing their inbox time. And that makes sense to me because when physicians decrease their clinical hours, they don’t often or always reduce their clinical panel at the same rate.
So they may keep the same number of patients, but now only have 80% of the appointment times available. And so more of the care for that full panel gets pushed to the virtual realm, gets pushed to the patient portal, and so their time on the patient portal goes up. And it seems to me that’s a fruitful area for future research.
Dr. Holmgren: Absolutely. And I think it’s one of the things that worries me the most. People ask me, what is the future of the clinical workforce and EHR burden and burnout look like? Is, I really worry about, this cycle where what happens is a physician feels burnt out because of their workload or other factors that drive them into a situation where they want to reduce their clinical workload. They reduce that, but their inbox work continues to increase, and this sort of intersects with another important policy issue, which is the undersupply of primary care physicians in the United States. Their patient panel doesn’t have anywhere else to go and they can’t get an appointment. So they send more messages and that causes them to feel even more burned out.
And we see this regularly here at UCSF Health where our very part-time physicians receive more messages per hour compared to their more full-time colleagues because their patients want to talk to them, they want to talk to their primary care doctor. They can’t find another one to switch to and so they go with what is available when there are no appointment slots, which is messaging.
And so this sort of almost “doom loop” of more messages to reduced hours to more messages is really concerning because I think it’s a really big predictor of burnout for our primary care physician workforce, which is very critical to maintaining the health of our patient populations.
Dr. Sinsky: And that story that you just told really illustrates to me the power of using EHR event log data, because without that, you wouldn’t be able to identify that these physicians are actually spending more time rather than less time on the most burdensome aspect of work. And so it allows health system leaders to think more creatively about how we might maintain our physician workforce.
Well, I’d like to switch our attention to another bit of research that you’ve done. You were awarded a competitive grant from the AMA for a research project that investigates practice strategies for improving inbox management. So we know we’ve got a problem, what can we do about it? I believe this was a mixed-method study where you did both quantitative and qualitative work.
Can you tell our listeners first how this research idea got started?
Dr. Holmgren: So once again, this goes back to a natural experiment that we wanted to take advantage of, which was back in 2019, UCSF Health went through a big inbox transformation where we wanted to have more non-physician staff take ownership of inbox work, especially patient messages.
So departments and clinics at UCSF started to staff up and add more staff. This was in the fall of 2019. Understandably in the spring of 2020, something happened that dramatically derailed our ability to continue our existing operational projects and to hire more nonclinical staff to handle messages, which was obviously COVID. So what happened here is we had a lot of variation here at UCSF where some departments had very few messages from patients actually needing to be touched by the physician, something around the 15 to 20% of messages were handled by the physician.
Whereas other clinics, even within the same specialty that just happened to be across town at a different physical location, had 90% of their messages being touched directly by the physician because they hadn’t managed to staff up in time.
And so this gave us a great natural experiment to look, OK, what happens when you’re able to use more team-based support to triage these messages, to handle the ones that can be handled by a non-physician staff member and to make sure that the ones that flow to the physician are ones that are sort of the top of the license that truly need physician-level diagnostics or clinical skills in order to be addressed properly.
And what we found in the quantitative data is this has a big effect. There is a significantly lower inbox burden for physicians practicing in those clinics with a lot of staff support. Using the power of EHR metadata to say, we can prove the physician time savings that we realized from this operational transformation and show health system leaders the quantifiable effect of this type of team-based EHR work, which I think is really powerful because otherwise you’re sort of left reading the tea leaves from either surveys or even just trying to talk to physicians ad hoc.
But I think using this sort of quantification, you can say, “Here is our return on investment from this team-based transformation,” which I think is fantastic.
Dr. Sinsky: Well, you’re singing my song here, A Jay. In fact, I think I spoke at UCSF a few years ago and shared the story, or at least the experience that I’ve had that I knew I had an inbox. I probably could find my way to it, but I never ever went there. Why? Because I have two nurses and they are well-trained and work exclusively with me. So we’re a stable team, and they are able to answer the majority of the inbox messages. And those that do require higher level input, they do the research on it, and then they come to me and we talk about it, and then we make a decision and then they operationalize it.
And what I’m hearing is this natural experiment for those clinics that did staff up, their inbox burden for their physician was remarkably less—15% of the messages required their engagement versus 90% in the clinics that hadn’t yet staffed up. So, wonderful work. I look forward to seeing that in print at some point. Thanks for sharing that.
I want to talk about one other study that I think you’ve been involved in, and that was looking at the association of telemedicine visits and pajama time, because I think many innovators, health system leaders, policymakers have assumed that if we convert a lot of visits to telemedicine, it will be less burdensome for physicians. But I’m not sure that turned out to be the case.
Dr. Holmgren: That’s exactly right. And so we kicked off this study with a few questions. The first is, what happens with telemedicine and time spent outside of the clinic on the EHR? And also, what happens when you’re doing a significant amount of your work via telemedicine, these synchronous video visits and patient portal messages. Because an open question is still, what caused that big increase at the onset of COVID? Was it telemedicine? Do patients have more follow-up questions when they’re getting care virtually rather than in person?
And so we found two important things, I think from that study, which is under review right now. The first of which is that telemedicine was associated with significantly more time outside of clinic hours on the EHR, which is especially concerning because study after study has found this sort of work outside of work, this pajama time is associated with physician burnout, with leaving practice, with reducing your clinical hours. And the second is that we didn’t find any association between patients receiving care via telemedicine and messaging their physician.
So on the bright side, it doesn’t seem like telemedicine has worse communications or that patients have more follow-ups. It’s just an increased awareness of the patient portal that came along with COVID or a desire to avoid in-person interactions that stayed throughout the post-COVID era as they realized how convenient and nice it was to be able to message your physician at any time.
But on the downside, this broad expansion of telemedicine may actually increase the burden on our physician workforce, which I think has really important implications for health system leaders thinking, how much telemedicine should I embrace and how should we schedule our physicians who are providing a lot of virtual care? As well as for policymakers, because the future of telemedicine is certainly greater than it was in the pre-COVID era.
But I don’t think it’s totally set into how we’ll pay for this or how much it should be used going forward as we think about what are our priorities balancing patient access to convenient virtual care and physician well-being and physician time spent in the electronic health record doing documentation work.
Dr. Sinsky: So, A Jay, my last question has to do with AI, and there’s a lot of enthusiasm about ChatGPT and generative AI as a solution to visit note documentation. I understand you’re doing a multi-center trial. Can you give us a sneak preview of what you’re doing?
Dr. Holmgren: Absolutely. So, many of you are probably familiar with all of these studies around ChatGPT that have been published over the past year or so. And I think a lot of them have shown a great power of generative AI, but I think some of them are focusing on perhaps the wrong questions. For example, it doesn’t really matter to me as a patient if ChatGPT can pass USMLE Step 2. That’s simply not going to be a reasonable replacement for a real physician.
So instead, what we’re evaluating is Epic has recently integrated ChatGPT in a few pilot sites, including academic medical centers around the country, to draft responses to these patient medical advice request emails. I think this is where artificial intelligence has more potential, which is as a physician efficiency extender, which is to say what we want to do is take the physician, and if we can, remove administrative burden and make them faster at doing these types of tasks. And so it automatically drafts these messages and then the physician edits them or deletes it and rewrites a response themselves.
And our evaluation is based around, well, how much is this working? Are we actually reducing physician time spent in the inbox? Are we reducing physician time spent after hours doing these responses over lunch, over dinner, when they would be otherwise at home in their off time? So we currently don’t have any data yet. We’re in a first phase of data collection, but I’m really excited about the possibility that this is going to reduce time.
That said, there are really important caveats here that we’re in the land of the unknown unknowns. There are questions around, well, is ChatGPT more aggressive at prescribing or ordering than a physician would’ve otherwise been? And does it sort of push physicians to be more aggressive if you subtly nudge them towards more aggressive prescribing, ordering, imaging, testing, than they otherwise would’ve been? That’s an important question.
Right now, the answer is we don’t know. So it’s still such early days that we have to balance this hope that these can be tools to reduce burden and improve EHR efficiency for our physician workforce with concerns and caution around this need to make sure that we aren’t dramatically changing how physicians would practice or exacerbating existing health disparities or increasing overall costs by more aggressively treating patients than they otherwise would.
So while I’m excited about the potential for AI in the future at various tasks, especially automating away some administrative work, I’m concerned and I think we need to approach it with caution appropriately due to any time that we are introducing a new technology into direct patient care.
Dr. Sinsky: Well, I agree with you on that, and I hope you also look at the cognitive workload because I think that there’s a possibility that with some of the new technology, while it seems to save time, it may increase the cognitive workload of reviewing the output that the AI generated, rather than the output that you yourself would’ve generated. And looking for the subtle errors or the hallucinations or other insertions of untruthfulness, that may take a substantial amount of one’s cognitive bandwidth to sort through.
Well, thank you so much for joining us today, Dr. Holmgren. I am so pleased to have had the chance to work with you outside of this podcast and during the podcast. For our listeners, I’d like to direct you to the episode description where you will find some of the studies that we referred to here. You’ll also find some resources that we have put together through the STEPS Forward® program, including a toolkit on reducing the inbox, as well as an inbox reduction checklist. And with that, thank you so much for joining us.
Speaker: Thank you for listening to this episode from the AMA STEPS Forward® podcast series. AMA's STEPS Forward® program is open access and free to all at stepsforward.org. STEPS Forward® can help put the joy back into medicine by offering real-world solutions to the challenges that your practice is confronting today. We look forward to you joining us next time on the AMA STEPS Forward® podcast series, stepsforward.org.
Disclaimer: The viewpoints expressed in this podcast are those of the participants and/or do not necessarily reflect the views and policies of the AMA.
