New Rules

The New Rules of Healthcare AI: Takeaways

The new frontiers of policy and regulation, with Rick Abramson, MD, Director, Digital Health Center of Excellence, FDA, Carmel Shachar, JD, MPH, Assistant Clinical Professor of Law, Harvard Law School, and Rafid Fadul, MD, MBA, Chief Medical Officer, ARPA-H.

1. Trust is the product. Everything else is a feature.

The chain of logic from the session was simple: without trust there's no adoption, and without adoption there are no better outcomes. That turns the usual startup playbook upside down. Most health AI companies compete on model performance, benchmark scores, and speed. But a tool that outperforms clinicians and sits unused hasn't improved a single patient outcome.

Right now the industry has a trust deficit, and it's partly self-inflicted. A wave of hype-driven AI products has taught clinicians and patients to be skeptical, and every overpromising pitch deck makes the next honest product harder to adopt. Regulation isn't the enemy of innovation here. It's the infrastructure that makes innovation usable. Trust is the bedrock of regulation, and regulation, at its core, exists to create trust.

If your go-to-market strategy doesn't include a plan for earning trust, you don't have a go-to-market strategy. You have a demo.

2. Stop waiting for the government to make people trust you.

One of the sharpest points was that trust isn't exclusively the government's job. The private sector has to earn it too. That rebukes builders who treat FDA clearance as the finish line, as if a regulatory stamp transfers credibility automatically.

It doesn't. Health systems, professional societies, clinicians, academics, and companies all share responsibility. A company that does the legal minimum and waits for regulators to vouch for it is outsourcing its credibility, and it will be treated accordingly.

Clearance is the floor for trust, not the ceiling. The companies that win will be the ones that publish their data, show their failures, and build trust faster than regulators can require it.

3. "Human in the loop" is often a liability shield dressed up as patient safety.

This was the most uncomfortable truth in the room. For many organizations, the human in the loop isn't there to improve outcomes. They're there so someone can be blamed when things go wrong. The session put it bluntly: human oversight often comes down to knowing whose throat to choke.

That matters because it means we may be designing clinical workflows around blame instead of performance. If a human reviewer adds friction, cost, and delay without improving accuracy, and mainly exists to absorb legal risk, then the patient is paying for the institution's insurance policy.

If AI will be better than most clinicians at certain tasks, the question has to be asked: what is the human actually for?

Before you add a human to the loop, ask whether they're there to protect the patient or to protect the institution. The answers are not always the same.

4. But accountability isn't nothing, and pretending it is would be naive.

The counterargument deserves real weight. Accountability is a legitimate part of care. When things go sideways, patients and families need someone who can explain, take responsibility, and make it right. There's also a behavioral reason for human involvement: patients are more likely to follow through on care when a real person is part of it, because of the social contract between patient and provider.

So the honest position isn't "remove the human." It's "be clear about why the human is there." Accountability, adherence, and relationship are valid reasons. Covering the institution legally is not a clinical rationale.

The future may separate the two jobs clinicians do today. AI handles the judgment where it's measurably better, and humans handle accountability, relationships, and adherence. That's not a downgrade for clinicians. It may be a return to the parts of medicine only humans can do.

5. The weakest link in human-AI collaboration may be the human.

The study discussed in the session is one of the most provocative data points in healthcare AI. On clinical diagnostic cases, AI alone substantially outperformed doctors working alone. Doctors working with AI barely improved on doctors alone. The reason: physicians overruled the AI when it was right, pulling combined performance down toward their own baseline.

That undercuts the comforting story that "AI plus doctor" is always the best of both worlds. In this study, the combination captured almost none of the AI's advantage. The bottleneck wasn't the machine hallucinating. It was the human refusing to listen.

We've spent years worrying about whether clinicians can trust AI. We should spend equal time asking whether clinicians know when to defer to it. Overconfidence is a patient safety issue too.

6. The "AI alone vs. human in the loop" debate is the wrong debate.

The FDA's position is more nuanced than either side of the argument. The agency plans to evaluate software function by function. Some functions will perform best with AI alone. Others will require human oversight. There's no universal answer, and no blanket rule is coming.

For builders, the implication is clear. Stop arguing about human-in-the-loop in the abstract and start generating evidence for your specific use case. Which task, which population, compared against what, with what outcome? That's the conversation regulators want.

Anyone making sweeping claims that AI should always or never operate autonomously isn't paying attention. The future is granular, and the evidence belongs to whoever generates it first.

7. We're holding AI to a standard we've never held doctors to.

One of the strongest arguments in the session was that critics keep poking holes in AI tools that could dramatically expand access, while demanding a level of perfection no clinician has ever been held to.

Pharma offers a better model. In clinical trials, new drugs aren't compared to perfection. They're compared to the standard of care, and the question is whether they improve on it. AI should be judged the same way. And for millions of people in rural and underserved communities, the realistic standard of care isn't a specialist. It's a long drive, a long wait, or nothing at all.

That makes the comparator question central. Do we compare AI to a specialist, a primary care physician, a nurse practitioner, or no care? The answer changes everything about what "good enough" means.

Every time an AI tool that beats the current standard of care is blocked for falling short of perfection, somebody who would have been helped goes without care. Excessive caution has costs too. We just don't count them.

8. Society is behind the evidence, and that gap is a cultural problem, not a technical one.

Even if AI alone outperforms humans on specific tasks, the session acknowledged that society will take time to accept it. The trust gap isn't about capability. It's about comfort, culture, and the emotional weight of letting a machine make decisions about our bodies.

The self-driving car comparison applies. Autonomous vehicles can post strong safety records and still face public resistance, because people judge machine errors more harshly than human ones. Healthcare will face the same asymmetry. One high-profile AI mistake will outweigh thousands of quiet successes in public perception.

The biggest barrier to AI in healthcare may not be regulation, technology, or cost. It may be our discomfort with forgiving machines the way we forgive people.

9. What gets measured gets funded, and we're not measuring enough.

The session pointed to a chicken-and-egg problem between regulation and reimbursement: products need a regulatory pathway to get reimbursed, and they need reimbursement to justify the investment that gets them through regulation.

The Alzheimer's example makes the stakes concrete. There are no strong preventive therapies partly because there are no good biomarkers, and without a measurable endpoint, no venture investor will fund the work. Capital follows measurability. Where outcomes can't be measured, innovation starves.

The same logic applies to AI. Without shared evidentiary standards, validated endpoints, and post-market monitoring, investors can't underwrite outcomes-driven products, payers can't reimburse them, and regulators can't clear them with confidence.

The most valuable investment in healthcare AI right now may not be another model. It may be the boring infrastructure that proves which models work.

10. The business model will decide whether AI helps patients.

The session was blunt about economics. One view argued that AI will be inflationary, as new technologies in healthcare historically have been, and that AI can't simultaneously lower costs and raise quality.

Then comes the sales reality. If an AI tool isn't reimbursed, the only way to sell it to a provider is on return on investment: efficiency, throughput, revenue capture. That means the tools that get bought are the ones that make health systems money, not necessarily the ones that make patients healthier. Only when AI is reimbursed does it become possible to sell it on quality.

Right now the market rewards AI that bills better, not AI that heals better. Until reimbursement changes, we should expect the industry to build what it's paid to build. If you want outcomes-driven AI, fix the payment model first.

11. The federal agencies are finally moving together, and that window won't stay open forever.

The session highlighted an unusual level of collaboration among federal agencies. Regulators, payers, and research funders are coordinating in ways that have historically been rare, which means regulatory and reimbursement pathways could be designed together instead of in sequence.

For builders, this is a rare opening. The rules are being written now, and stakeholders who engage by submitting comments, participating in pilot programs, and sharing data will shape them.

If you're building in healthcare AI and not engaging with regulators right now, you're letting someone else write the rules you'll live under for the next decade.

12. Meeting patients where they are isn't a UX principle. It's a trust strategy.

Trust isn't built in press releases or regulatory filings. It's built in the moment a patient decides whether to engage. Tools that fit into people's lives, speak their language, and respect their circumstances earn trust. Tools that feel imposed don't.

This matters most in communities with good historical reasons to distrust the medical establishment. For them, a well-designed AI tool could be a way around the institutions that have failed them. A poorly designed one, or one that feels like a cheap substitute handed to "the have-nots," will deepen that distrust.

The communities most skeptical of healthcare may be the ones most open to a better alternative. The opportunity isn't to replicate the old system with AI. It's to build what the old system never offered.

13. Not deploying AI is also a choice, and it has casualties.

The session framed AI as having transformative potential to reduce health disparities and improve outcomes, and called using it a societal responsibility. AI is a tool for good in the right hands and a tool for harm in the wrong ones. But the ethical burden doesn't only fall on those who deploy it.

When a tool that could expand access or catch a missed diagnosis sits unused because of institutional inertia, liability fears, or impossible standards, real people bear the cost. They're just harder to see than the victims of an AI error.

The question isn't only "What if AI gets it wrong?" It's also "What happens to the patients we leave behind while we wait?"