Analysis: Watson for Oncology was over-marketed, now underwhelms hospitals and oncologists

  • Small
  • Medium
  • Large

The artificial intelligence platform Watson for Oncology hasn't met the high expectations its developers at IBM laid out, according to an analysis by STAT writers Casey Ross and Ike Swetlitz.

Watson for Oncology is a cloud-based platform that mines information from more than 300 medical journals and more than 200 textbooks to provide clinical decision support. The platform advertises its ability to identify relevant treatment options and administration instructions to personalize care for individual breast, lung, colorectal, gastric, cervical and ovarian cancer patients.

"The interviews suggest that IBM, in its rush to bolster flagging revenue, unleashed a product without fully assessing the challenges of deploying it in hospitals globally," Mr. Ross and Mr. Swetlitz wrote. "Its flaws are getting exposed on the front lines of care by doctors and researchers who say that the system, while promising in some respects, remains undeveloped."

Here are five insights into Watson for Oncology.

1. IBM hasn't published scientific, peer-reviewed papers on Watson for Oncology's performance. Although the system is deployed at healthcare facilities across the globe, no independent, third-party investigations into the technology exist. The only studies on Watson for Oncology are conference abstracts, most of which were conducted by customers or IBM staff.

2. The majority of the conference abstracts detail positive results, focusing on Watson for Oncology's high rate of agreement with experts when it comes to treatment recommendations. However, the STAT writers noted high concordance only demonstrates the system is "competent in applying existing methods of care, not that it can improve them."

3. Jean Thompson, a nurse at Jupiter (Fla.) Medical Center, is tasked with spending 90 minutes a week feeding patient data into Watson for Oncology. She recalled one instance, in which it recommended a chemotherapy regimen to a lung cancer patient, as underwhelming.

In instances where the system provides unexpected recommendations, Watson for Oncology proves similarly frustrating. Dr. Taewoo Kang, a surgical oncologist in South Korea, noted Watson for Oncology sometimes leaves him confused, since the system doesn't explain reasoning behind its suggestions.

4. Watson for Oncology may also suffer from human bias in its recommendations. The system uses cloud-based supercomputing to sift through large-scale datasets and published literature. However, its recommendations are not based on analysis of patient outcomes. Instead, they are based on input from physicians at New York City-based Memorial Sloan Kettering Cancer Center.

"The system is essentially Memorial Sloan Kettering in a portable box," the STAT writers noted, adding the hospital's physicians tend to treat affluent patients, and may lack a diverse knowledge base. "Its treatment recommendations are based entirely on the training provided by doctors, who determine what information Watson needs to devise its guidance as well as what those recommendations should be."

5. In recent years, "hundreds" of companies have debuted artificial intelligence products in the healthcare industry, according to the STAT writers, including tech giants like Amazon, Google and Microsoft. In the end, IBM might be done in by its own extreme promises, which set unrealistic expectations for its clients.

"IBM ought to quit trying to cure cancer," Peter Greulich, a former IBM brand manager, told the STAT writers. "They turned the marketing engine loose without controlling how to build and construct a product."

Click here to view the full article.

Copyright © 2021 Becker's Healthcare. All Rights Reserved. Privacy Policy. Cookie Policy. Linking and Reprinting Policy.


Featured Whitepapers

Featured Webinars