The pilot we killed: Why Houston Methodist pulled an ER tool that nagged physicians

Advertisement

A few years ago, Houston Methodist brought in an AI tool meant to help its emergency department get ahead of patient surges. About six months later, the health system turned it off. The tool hadn’t sped anything up, and the physicians it was supposed to help had come to dread it.

Roberta Schwartz, PhD, executive vice president and chief innovation officer of Houston Methodist, said the system goes into technology deals expecting some of them to fail. Contracts are written to scale across all of its campuses, but a product starts in one place, and the system tries to keep the right to exit at a set point in the trial.

“Our contracts were set up so that at the end of a year, if we couldn’t make a product work, we could exit as friends and recognize that it wouldn’t necessarily work,” she said.

The ED tool, which Houston Methodist used around the start of the COVID-19 pandemic, was built to predict surges from patterns such as the time of day and day of the week, then alert the ED team so staff could change how they worked. It also pulled data from the system’s Epic EHR to nudge emergency physicians toward a disposition, the decision on whether to admit a patient or send them home.

“The whole goal was to reduce the amount of time in the emergency room for the patients. It did not do that, and it totally annoyed the physicians,” Dr. Schwartz said.

Part of the problem was how the tool read the EHR. In Epic, radiology orders opened only when a patient reached the imaging suite. The tool couldn’t see orders that hadn’t been opened yet, so it began asking physicians for a disposition before their patients’ tests had been done.

“The physician was like, ‘The tests aren’t run. Like you’re kidding, right? Like that makes no sense.’ They just got more and more annoyed every time this product ran,” Dr. Schwartz said.

The surge alerts missed, too.

“It would start beaming out the answers and the pushing at very odd times. Like it’s a surge, and you’d be like, ‘There are three patients in the emergency room,’ and there’d be 100 patients in the emergency room, and it wouldn’t push out the surge,” she said.

Dr. Schwartz also faulted the vendor’s staffing. The company had sold the product to several hospitals and couldn’t put its best people at each one.

“They really failed to bring in the A team. They had now sold this at multiple different hospitals, and they could not deploy an A team into every organization. And I don’t think we got an A team,” she said.

The call to stop belonged to the people closest to the work: the head of the emergency department, its head of quality, its administrators and the IT team that had brought the product in. When anyone in a group like that wants to keep going, Dr. Schwartz said, the others try to understand why before moving on.

“After pushing this product for about six months, we finally said that’s enough,” she said.

Houston Methodist didn’t disclose what it paid. Dr. Schwartz said deals like this one typically carry an implementation fee, but the larger expense is the organization’s own staff time.

“I think no matter what you’re paying for the product, sometimes that’s the cheapest of the costs compared to the amount of time and distraction your people put into these things,” she said.

The system still believed in the idea. It brought in a second product to do the same job, trialed it before going fully live and exited that one, too.

“I think what I learned is the space wasn’t ready. It just wasn’t sophisticated enough. The products did not do what they said they could do, and we spent a lot of time pushing something that just annoyed our physicians,” Dr. Schwartz said.

She said she asks the same readiness question of tools Houston Methodist is weighing now, including documentation tools from Epic.

“The question is, are they really ready for prime time? And sometimes yes, sometimes no,” she said.

What can close the gap, she said, is a vendor that commits its best people to fixing what’s broken.

“When they really do bring the A team to the table and make every effort in the universe to fix what’s not working or say this is not something we can do well, it’s very different than someone saying, ‘Well, it’s working at other places. Why isn’t it working at yours?'” Dr. Schwartz said.

Killing the ED tool didn’t change Houston Methodist’s approach to new technology, she said.

“It reinforced the fact that we will continue to do things in the way that we’re doing them, which is not being scared to try things and not being scared to say, okay, it’s not working,” she said.

Editor’s note: This is the latest installment of a recurring feature. Email gbruce@beckershealthcare.com with other examples.

Advertisement

Next Up in Innovation

Advertisement