IBM's Watson supercomputer, once hailed as a revolutionary cancer treatment tool, reportedly gave physicians inaccurate cancer treatment advice, and company medical specialists and customers reported "multiple examples of unsafe and incorrect treatment recommendations," according to internal IBM documents reviewed by STAT.
Here are 10 things to know:
1. The documents blame IBM engineers and New York City-based Memorial Sloan Kettering Cancer Center — one of the early adopters of Watson for Oncology — for poorly training Watson software by using just a few hypothetical cancer cases instead of real patient data as well as treatment recommendations from a few specialists as opposed to "guidelines or evidence." This calls into question the validity of the tool as physician's personal preferences trumped IBM's touted machine learning analyses. IBM also promised Watson used historical patient data, but according to the documents, that was not the case.
2. According to STAT, the documents — slide decks from June and July 2017 by then-IBM Watson Health Deputy Chief Health Officer Andrew Norden, MD — include customers' assessments of Watson for Oncology that say it produced "often inaccurate" recommendations that pose "serious questions about the process for building content and the underlying technology." No patient deaths have resulted from hospitals' use of Watson for Oncology, STAT reports.
3. IBM executives have repeatedly told customers that Watson for Oncology's advice is based on real patient data and physicians around the world love the tool.
4. A previous STAT investigation showed Watson for Oncology wasn't living up to the company's expectations, but these new documents show IBM officials knew its product was making recommendations that conflicted with national treatment guidelines and that physicians did not find useful for treating patients. All the while, IBM executives were speaking favorably about the supercomputer.
5. "This product is a piece of s—," one physician at Jupiter Hospital in Florida told IBM executives, according to the documents. "We bought it for marketing and with hopes that you would achieve the vision. We can't use it for most cases."
6. The documents also showed that several IBM employees told Dr. Norden the product was "very limited."
7. In a statement to STAT, IBM defended its software, stating: "We have learned and improved Watson Health based on continuous feedback from clients, new scientific evidence and new cancers and treatment alternatives. This includes 11 software releases for even better functionality during the past year, including national guidelines for cancers ranging from colon to liver cancer."
8. A Memorial Sloan Kettering spokesperson told STAT the documents reveal the "robust nature of the process" of building and deploying Watson for Oncology in clinical care, and that the cancer center is working with IBM to improve the accuracy of the system's recommendations.
9. "Patient safety is paramount," the cancer center's spokesperson said. "While Watson for Oncology provides safe treatment options, treatment decisions ultimately require the involvement and clinical judgement of the treating physician… No technology can replace a doctor and his or her knowledge about their individual patient. To that point, the tool is also not equivalent to the cancer care delivered at MSK."
10. IBM recently laid off a number of Watson employees, including several from its three recently acquired companies Truven, Phytel and Explorys. Those employees revealed several issues with the company's faulty internal organization and inability to make Watson profitable.
To access the complete STAT report, click here.