In November 2022, six prominent law schools — including Harvard, Yale and Stanford — withdrew from U.S. News and World Report's esteemed university rankings. A domino effect ensued, leading medical schools, hospitals and health systems to question the various systems that grade complex organizations — especially the one that, until this year, would crown a single institution the "best."
The first domino falls — and it's Harvard
Seventeen days into 2023, Boston-based Harvard Medical School spurred an exodus of medical schools from U.S. News rankings. Within two weeks, thirteen renowned institutions had joined the movement, their deans publishing statements denouncing the system they once submitted data to. The universities alleged the rankings' reliance on peer assessment scores from deans, admissions directors and academics along with standardized testing scores created an elitist mentality and reduced complex institutions to mere figures.
"Medical education cannot be reduced to a set of numbers that purport to reflect its quality," said David Muller, MD, dean for medical education at New York City-based Mount Sinai's Icahn School of Medicine. "Candidates to medical school want to know about culture and climate, mentorship, opportunities for research and community service, wellness initiatives, curricular outcomes, and the depth and breadth of student support."
Although each school could still be ranked using publicly available data, their boycott caught the attention of U.S. News officials. The publication pushed back its medical school ranking release date, citing an "unprecedented number of inquiries from schools." The list was published more than three weeks after its target date using updated methodology that sought to prioritize student outcomes — for example, its research ranking added NIH Grant Awards as a measure of research quality; increased the weight of faculty-to-student ratios; and reduced the weight of reputation surveys, MCAT and GPA scores.
But one question lingered, as many of the schools participating in the boycott were attached to academic health systems: what would become of the hospital rankings?
Hospital rankings spark debate
Some health system leaders spoke their minds on the unfolding controversy, including Fritz François, MD — executive vice president and vice dean, chief of hospital operations for New York City-based NYU Langone Health — and Gbenga Ogedegbe, MD — director of the Institute for Excellence in Health Equity at NYU Langone Health. In an opinion piece for The Wall Street Journal, the executives argued that medical schools were wrong to denounce a well-respected ranking system on the basis of inclusivity.
"What these schools are really saying is that meritocracy can't coexist with diversity," the pair wrote. "This is a presumptuous — and dangerous — perpetuation of the negative stereotype that students from backgrounds that are underrepresented in medicine are of lesser quality or unable to compete."
Two resident physicians at Seattle Children's had a different opinion. Madeline Wozniak, MD, and Chinenyenwa Mpamaugo, MD, argued that hospitals should follow law and medical schools' example in an article published to Health Affairs. The two noted that although sickle cell disease affects nearly three times as many people and causes 40 times more hospitalizations than cystic fibrosis, U.S. News' children's hospital rankings award 19 points for "success in managing cystic fibrosis patients," but only one point for having a formal sickle cell disease program. Sickle cell patients are usually Black, which indicates an inequitable evaluation on the publication's part, per the physicians.
Shortly after the Health Affairs article was published, U.S. News announced it would update the methodology of its children's hospital rankings. The publication increased the weight of its "commitment to best practices" metric by 3% and its "commitment to equity, diversity and inclusion" metric by 0.33%. It also decreased the weight of expert opinion scored by 3% across all specialties, and updated DEI measures to "better represent key activities involved in this area of work," according to a news release.
But the publication was not off the hook yet; health systems themselves began criticizing the hospital rankings, beginning with Bethlehem, Pa.-based St. Luke's University Health Network. On May 24, the system's CEO and chief quality officer announced they would no longer submit data to the publication, which requests information from children's hospitals and hospitals offering maternity care to compile its rankings. In a letter to Ben Harder, U.S. News' managing editor and chief of health analysis, the executives alleged the rankings run on "misguided methodology." They critiqued the inclusion of expert opinion scores and compared the ranking system to a "popularity contest."
Philadelphia-based Penn Medicine's University of Pennsylvania Health System became the second health system to withdraw from the rankings. Kevin Mahoney, the health system's CEO, spoke against them in a June 26 news release, alleging the regularly-changing methodology makes it difficult to draw meaningful conclusions over time. He also claimed that the rankings incentivize health systems to "expend resources both to compete for placement in the rankings and promote their position on the list."
Even the city attorney of San Francisco weighed in on the debate, asking U.S. News to send data substantiating its claims that the hospital rankings are "authoritative," based on "world-class data and technology," and help patients and families "find the best healthcare," "make data-informed decisions," and "find sources of skilled inpatient care."
"I think there is, in some quarters, an overemphasis on numerical rankings," Mr. Harder told Becker's in early June. "If you are one of the top 50, you're doing really well. Whether you're 48 this year or 42, we'd be among the first to acknowledge there is often no statistical difference — numerical difference is within a margin of error. We do continue to publish numerical rankings because we believe there is a difference between No. 1 and No. 50. We don't see patients using the rankings driven by numerical placement."
His comment foreshadowed a July announcement: one of the most influential rankings would "rank" no more.
The end of the ordinal honor roll
U.S. News shared seven changes to the hospital ranking methodology as the debate over its future festered, including refinements to health equity measures, greater weight on objective quality measures and less weight on expert opinion scores.
Then, the publication announced it would no longer attribute ordinal rankings to its honor roll hospital. Instead, all honor roll hospitals would be grouped in the same category — although ordinal rankings would still be listed for the 15 specialties U.S. News gauges, as well as the regions where it publishes.
"While we continue to value the usefulness of the Honor Roll in identifying broad clinical excellence, we also believe that the current format obscures the fact that all of the Honor Roll hospitals have attained the highest standard of care in the nation," U.S. News wrote in a July letter to hospitals.
Critics leap on Leapfrog
U.S. News wasn't the only system to face backlash this year. The Leapfrog Group — known for assigning letter grades based upon hospital safety — also received public pushback from healthcare leaders. In August, James Berry, CEO of Tahlequah, Okla.-based Northeastern Health System, alleged Leapfrog ratings have created a "medical social media bully pulpit."In November, Salisbury, Md.-based TidalHealth and the Delaware Healthcare Association spoke out after zero Delaware hospitals received an 'A' from Leapfrog.
The organizations alleged that if they do not respond to the voluntary survey, their scores are greatly reduced; that Leapfrog uses past data which excludes recent performance improvement efforts and thus scores hospitals inaccurately; and that the grading system is biased towards large tertiary hospitals in cities, as it does not account for physician scarcities in rural areas.
Leah Binder, president and CEO of The Leapfrog Group, told Becker's Dec. 14 that the organization often receives complaints from hospitals that are unhappy with their grades.
"At the same time, we see hospitals that are able to take a constructive approach to a poor grade, and over time, turn it into an excellent grade," Ms. Binder said."I think it's very important for hospitals to focus on their quality and safety, even when they're distressed by their grades."
"I sympathize with them, but I want them to focus on quality and safety," Ms. Binder continued. "And boards and leaders of hospitals should never lose sight of the importance of their grade and other ratings. Don't just assume that because you got a poor grade, the grade must be wrong."
Ms. Binder added that Leapfrog grades are not perfect, and the public can propose methodology changes each year. However, the grading system utilizes "30 measures and a bevy of world experts on patient safety" to refine that methodology, and she believes in its validity.
"You owe it to your patients to do the very best you can no matter how offended you are by one of these rating systems," Ms. Binder said. "Do your best because lives are at stake."
A shift, but not an end
On Dec. 5, U.S. News published its list — not an ordinal ranking — of the best hospitals for maternity care, and issued an inaugural list of hospitals that provide access to maternity care for underserved communities.
"Nationally, 680 hospitals — an all-time high — submitted data to be considered for this objectively determined recognition," Mr. Harder told Becker's. U.S. News has continued to expand its offerings as health systems continue to display interest, and will rank ambulatory surgery centers in 2024.
There's no doubt that recognition is important to hospitals, no matter what package it comes in: a safety grade, a number-one ranking spot, a place on a collective honor roll. It's a source of pride and affirmation, a verification badge that brings international patients to the doorstep, Evaline Alessandrini, MD, chief operating officer of Cincinnati Children's Hospital Medical Center, told Becker's in September. The hospital was named America's best children's hospital by U.S. News for the first time this year.
"Why we're so proud of the U.S. News No. 1 ranking is because that, in and of itself, is not the end game, right?" Dr. Alessandrini said. "It is an affirmation or a verification of the way that we operate. And the way that we deliver care, it's working."