With augmented intelligence (AI) systems becoming increasingly capable of acting semiautonomously or even autonomously—commonly referred to as agentic AI—the AMA is leading the way on a number of fronts to ensure that the technology develops in a responsible way.
In October, the AMA unveiled “A Governance Framework for Agentic AI Assuming Clinical Functions” (PDF), which will guide the Association’s work with physicians and other stakeholders to ensure that increasingly capable AI improves care while clinical authority and accountability remain aligned.
The overall governing principle of the framework is that clinical authority should be earned through evidence and maintained through demonstrated performance. In addition, authority and oversight should evolve as evidence, performance, risk or the technology itself changes.
The framework outlines four connected questions that AMA’s work will focus on:
- Function: What clinical function is the AI performing? Define the task, patient population, setting and intended use.
- Evidence and risk: What demonstrates that it can perform that function and what happens if it fails? Evaluate performance, safety, clinical benefit, population fit and uncertainty in proportion to potential consequence.
- Authority and oversight: What should the AI be permitted to do and under what conditions? Set the permissible scope of action and the human or institutional oversight required for safe use.
- Accountability: Who is responsible for the risks and outcomes? Allocate accountability according to authority, control, knowledge of risk and ability to prevent or mitigate harm.
The AMA is specifically tackling the new health AI questions raised in the areas of liability, licensure and regulation.
From AI implementation to digital health adoption and EHR usability, the AMA is fighting to make technology work for physicians, ensuring that it is an asset to doctors and the patients they care for. That includes the AMA Center for Digital Health and AI, which works to ensure physicians help shape how AI is developed, implemented and regulated across the healthcare system, with patient safety and physician-led care remaining at the center of those efforts.
Aligning clinical AI liability
As AI takes on an expanding role in clinical care, questions about who is responsible when these technologies cause patient harm become increasingly urgent. The AMA Center for Digital Health and AI is advancing a new framework for aligning liability with each party’s authority and ability to prevent or mitigate harm.
“Patients must have meaningful avenues for recourse when clinical AI causes harm, and physicians must be able to use these technologies with confidence that responsibility reflects what they can reasonably know and control,” said the AMA President Willie Underwood III, MD, MSc, MPH. “As AI assumes more clinical functions, developers, healthcare organizations and physicians each have important responsibilities. We need clear expectations that protect patients, support sound clinical judgment, and ensure accountability remains with those best positioned to prevent harm.”
The AMA is calling for an approach that establishes well-defined, complementary responsibilities across the clinical AI lifecycle:
- Developers and manufacturers should demonstrate and maintain the safety and effectiveness of their products and address risks within their control.
- Healthcare organizations and other deployers should be accountable for how AI systems are selected, implemented, governed and monitored in real-world settings.
- Physicians should be responsible for the clinical judgment they can meaningfully exercise, based on the system’s intended use, known limitations and the information available at the point of care.
- Patients must retain meaningful avenues for seeking accountability and appropriate remedies when harm results from defective AI systems or other actionable failures.
The framework recognizes that multiple parties may bear liability for the use of AI when something goes wrong, with accountability aligned with each party’s ability to prevent harm. It also emphasizes that physician oversight cannot substitute for adequate evidence, appropriate system design or the conditions necessary to make clinical review meaningful.
AMA shaping policy, structures
Physicians are likely familiar with AI’s ability to take ambient notes during visits, or using it to comb through patient records to discover who may be overdue for a screening or an office visit. But its capabilities are beginning to venture into areas that even more directly affect interactions and decisions about patients, a trend that isn’t expected to slow down. That makes it increasingly important for physicians to take the lead on shaping the emerging technology and the policies that govern it.
Take for instance that in October, Utah became the first state to let AI prescribe adults acne medication after they have submitted facial scans and medical information to an algorithm in an app called Nolla Health. Initially, every prescription will require a doctor’s approval, but later phases of the trial will let AI write prescriptions for the drugs independently, according to a report in The BMJ. The trial is being allowed under the Utah AI Regulatory Sandbox, a state-sponsored program lawmakers enacted that is designed to test and deploy artificial intelligence products under customized, temporary regulatory relief.
Debate about accountability and responsibility that swirl around advances like the one Utah is making is why the AMA is bringing physicians together with people who build, deploy and regulate AI. The goal is for the evolving technology to be governed in a responsible way and in a way that aims for better patient care. As part of that work, in September, the AMA and the Federation of State Medical Boards (FSMB) brought together leading experts—from physicians, state medical board representatives, patients and health systems to technology developers, regulators, policymakers, legal experts and academics—to talk about the accountability and responsibility for the practice of medicine in the AI era.
Participants agreed that AI is already changing how medicine is practiced, but they had different opinions on how quickly and significantly it will transform clinical practice. Experts also discussed how existing regulatory, licensure and liability frameworks were not designed for AI and need to be modernized, raising questions such as who is responsible for ensuring that AI is used safely and who bears legal responsibility when AI contributed to harm, among other questions.
Jennifer Goldsack, MChem, MA, MBA, OLY, is the AMA’s new senior vice president of digital health and AI, and she is set to moderate a Nov. 17 panel discussion on state sandboxes and healthcare AI licensing at the HLTH conference being held in Las Vegas. The panel will examine how state regulatory sandboxes are being used to test AI-enabled tools and how emerging licensing and governance frameworks may shape their use in health care.
Speakers are scheduled to discuss the opportunities and risks of these approaches, with a focus on safety, liability, transparency and the future of state-level AI regulation. Panelists in the discussion will include:
- Shannon Curtis, the AMA’s senior director of policy development.
- Zach Boyd, PhD, director of the Utah Office of Artificial Intelligence.
- Utah immunologist Jonathan Olsen, DO.
AMA recommendations to FDA
In mid-October, the AMA sent a 14-page letter (PDF) to the Food and Drug Administration with recommendations on how to modernize the ways in which GenAI-enabled technologies are evaluated, regulated and monitored.
The letter came in response to the FDA’s request for feedback about its August report, “Considerations for the Regulation of Generative Artificial Intelligence (GenAI) Enabled Medical Devices.”
“The AMA recognizes that AI technologies present novel challenges for FDA oversight that may not be fully addressed by regulatory approaches developed for static medical devices,” AMA CEO John Whyte, MD, MPH, wrote in the letter to the FDA’s director of the Digital Health Center of Excellence, Rick Abramson, MD, MHCDS.
“We therefore support FDA’s efforts to evaluate and develop regulatory approaches that are appropriately tailored to the unique and evolving characteristics of AI-enabled technologies while maintaining robust standards for patient safety and product performance,” the AMA letter says. “Consistent with the AMA’s longstanding position, such oversight should incorporate a least-burdensome, risk-based framework that supports continued innovation in healthcare.”
The letter emphasized two overarching principles to the FDA:
- Regulatory agility cannot come at the expense of safety and effectiveness.
- Physician expertise should inform the standards, not serve as the safety net.
Dr. Underwood added that “physicians have a critical role in defining what good performance looks like, and the AMA is ready to help build the evidence and oversight frameworks that will allow trustworthy AI that improves care. Manufacturers must remain accountable for the performance of their products, and patients and physicians deserve the confidence that these technologies are safe and effective.”
