Most health systems don’t have an AI problem; they have an operationalization problem.
Underneath that operationalization problem are two journeys running at once. The technology journey (the platform, the data pipelines, the evaluation infrastructure) is, relatively speaking, the tractable one. The organizational journey: changing how teams govern, adopt, and hold themselves accountable to new workflows, is the harder, slower one, and it’s the one most health systems underinvest in. ChatEHR is as much a case study in the second journey as the first.
The pilots are everywhere. The proofs-of-concept are compelling. And yet the vast majority of clinical AI tools never leave the lab, dying quietly as demos, grant deliverables, or “cool things sitting on someone’s computer that we can’t put anywhere,” as Lead Data Scientist Juan Banda said.
ChatEHR is a chat interface embedded directly in the EHR that lets clinicians query structured and unstructured patient data in plain language: “Has this patient ever had a positive biopsy?” “Summarize the last 12 months of cardiology notes.” Behind that interface sit governed data pipelines built to cut through “250 novels’ worth of data” per patient, as Aditya Sharma, senior manager of data engineering at Stanford Medicine, put it.
But the real lesson for health IT organizations isn’t the software — it’s the operating model. Here’s how Stanford built it, spanning both the technology and organizational journeys, and what your organization can borrow.
1. Treat cross-functional ownership as the default, not the exception
At Stanford, no single department owns ChatEHR. The core team spans data scientists, data engineers, integration experts, nursing informatics professionals, product managers, UX designers, and clinical champions — reporting up through IT, operations, nursing, and compliance.
That structure is intentional. Clinical AI fails when it’s handed off between silos: data science builds it, IT integrates it, operations is expected to adopt it. Stanford instead makes clinical and operational stakeholders co-owners from the start.
For health IT leaders: Before scoping your next AI initiative, map who needs to co-own it, not just consult on it. If your clinical champions and operational leaders aren’t accountable for outcomes alongside your technical team, you’re building a faster horse that no one will ride.
2. Invest in the boring infrastructure first
If ChatEHR is the visible tip of the iceberg, the mass beneath it is data engineering, integration, and secure infrastructure. Building the tool required:
- Establishing secure, governed access to clinical data at scale
- Designing pipelines to unify data across multiple EHR instances and external systems
- Embedding the interface inside the EHR — no new portal, login, or workflow toggle
None of this is glamorous, and none of it is optional. “That investment from leadership makes my job possible,” Mr. Banda said.
For health IT leaders: No amount of AI experimentation reaches the bedside without modern, HIPAA-compliant infrastructure underneath it. Fund the chassis before you fund the features.
3. Build evaluation and monitoring into the product lifecycle
In consumer tech, success is measured in launch dates and feature counts. In healthcare AI, those metrics barely scratch the surface.
Stanford’s team measures accuracy, hallucination rates, and response times, but also the questions clinicians actually care about: Does this save time? Does it catch important details? Does it fit the workflow, or add friction? They study real-world usage — which questions get asked, at what point in the workflow, and how often outputs are accepted or overridden — and feed that back into model tuning and product design.
Critically, they align with emerging clinical AI oversight frameworks (such as FURM and MedHELM) and treat post-deployment monitoring for performance, safety, and equity as a core part of the lifecycle — not an afterthought.
For health IT leaders: Define “how will we know this is working, and for whom?” before go-live. Monitoring isn’t a compliance checkbox; it’s how you catch drift, safety issues, and inequities after the tool is live.
4. Reframe go-live as the beginning, not the end
For many health IT projects, go-live is the finish line. For AI, it’s the start of the “long, sometimes exhausting middle.”
Stanford’s team names two traps worth guarding against:
- The “faster horse” trap. Incremental automation builds credibility quickly, but if that’s all you build for five years, the team has no future. Reserve capacity for genuinely transformative work.
- The “only ROI” trap. “Some of the most important things we’re learning don’t show up as clean ROI right away,” said Product Manager Duncan McElfresh. “They show up as new workflows, new capabilities, new ways of thinking about the enterprise.”
For health IT leaders: Resource the post-launch phase — training, feedback channels, iterative tuning, workflow redesign — as deliberately as the build phase.
5. Keep the human in the loop — by design
At Stanford, ChatEHR augments clinical judgment; it does not replace it. “You’ve got to be the human in the loop,” said Nerissa Ambers, director of health informatics transformation. “We don’t just let AI run the show here.”
The tool has already assisted in multiple cancer diagnoses — a reminder that the goal is to surface what a clinician might otherwise miss, not to make the decision for them.
For health IT leaders: Design workflows that make it easy for clinicians to accept, override, or verify AI outputs. Track those override patterns as a signal.
6. Give teams leadership air cover and permission to be bold
Ask the Stanford team what made their work possible, and they don’t start with algorithms. They start with leadership: funding cross-functional teams for real, granting latitude to experiment within security guardrails, and accepting that some projects will fail.
In hindsight, Mr. McElfresh wishes they’d been bolder: “Ask for forgiveness not permission more often. Be less risk-averse.”
The point isn’t recklessness with patient care. It’s recognizing that absolute caution carries its own risk — the risk of falling behind, and of letting promising ideas wither under rules that were never written with modern AI in mind.
The replicable part isn’t the model; it’s the machinery
The most important insight from Stanford’s experience is that the durable, transferable asset isn’t the AI model. It’s the organizational model.
“The technology itself is only half the equation,” said Nikesh Kotecha, head of data science. “The true breakthrough is the organization we are delivering to enable it. … Without that time investment, these innovations simply never reach the patient.”
Other Stanford teams are now building on ChatEHR’s foundation — the secure data access, integration patterns, and governance processes — rather than reinventing the wheel. The flagship product has become a shared chassis.
For the rest of the industry, the choice is coming into focus: treat AI as a rolling series of pilots, or treat it as the beginning of a new kind of infrastructure. Most organizations can eventually solve the technology journey; fewer invest in the organizational one. The systems that make that investment are the ones that turn AI from a rolling series of pilots into permanent infrastructure.
Key takeaways
- Operationalization, not innovation, is the bottleneck. Most clinical AI tools fail not because the technology is weak, but because the organization can’t move them from pilot to practice. AI adoption is two journeys, not one — the technology journey is necessary but not sufficient; the organizational journey (governance, adoption, accountability) determines whether AI reaches the bedside.
- Make it cross-functional by default. Clinical, operational, informatics, and technical stakeholders should co-own AI initiatives — not simply consult on them.
- Fund the infrastructure first. Secure, governed, modern data pipelines are the precondition for AI reaching the bedside.
- Build evaluation and monitoring into the lifecycle. Measure accuracy and workflow fit, safety, and equity — before and after go-live.
- Treat go-live as the start of the hard part. Resource training, iteration, and workflow redesign, and avoid both the “faster horse” and “only ROI” traps.
- Keep humans in the loop by design. AI should augment clinical judgment, with easy paths to verify and override.
- Leadership air cover is decisive. Funding, latitude, tolerance for failure, and permission to be bold determine whether AI ideas survive.
- The replicable asset is the operating model, not the algorithm. Build a chassis other teams can share.
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.
