MD Anderson Cancer Center’s uninterrupted seven-year stretch as the top cancer hospital in the U.S. News & World Report rankings has come to an end.
Memorial Sloan Kettering Cancer Center has broken the spell of being the perpetual runner-up and moved to the lead.
The result is as close as cancer care can come to a photo finish:
- MSKCC: 100 percent.
- MD Anderson: 99.9 percent.
The contest between MD Anderson and MSKCC provides the consumers and researchers alike with the opportunity to examine what the U.S. News hospital rankings measure—and, just as importantly, what they don’t measure.
While many health services researchers dismiss the magazine’s ranking system as non-scientific, their own institutions aren’t reticent to use these rankings aggressively in marketing. MD Anderson has been heralding its primacy on billboards at airports, in ads that pop up on computer screens and news publications, and in outgoing messages on the cancer center’s phone system.
The top spot in U.S. News rating of cancer hospitals represents the ultimate bragging rights in oncology. High grades from U.S. News can help compete for patients and enhance ability to raise funds.
The change at the pinnacle of U.S. News rankings appears to have occurred because the magazine changed the methodology it uses to calculate scores. The scoring system this year changed in two ways, both of which likely influence rankings, especially in the case of MD Anderson vs. MSKCC.
- The safety score was expanded to include two new metrics and given a greater weight in the overall composite score. MSKCC gets significantly higher marks in safety than MD Anderson. In 2013, safety score accounted for 5 percent of the grade. In 2014, it accounts for 10 percent.
- The “reputational” score, in which MD Anderson consistently does better than MSKCC, became a less important component in calculation of the overall grade. Last year, the reputational score accounted for 32.5 percent of the grade. Now, it accounts for 27.5 percent.
The manner in which institutions use the U.S. News rankings is symptomatic of the intense thirst for metrics that could enable consumers to compare institutions—and even categories of institutions, such as academic centers vs. community care.
Full ranking of cancer centers, released today, is posted on the U.S. News website.
What does the 0.1 percent gap really mean?
“Not much,” said Rena Conti, an economist at the University of Chicago and co-author, with this reporter, of a recent paper on metrics of quality of care published in the Journal of Oncology Practice. “The differences in the rankings are to health services research what astrology is to astrophysics. In this contest over top bragging rights, the joy of victory is as irrelevant as the agony of defeat.”
Avery Comarow, health rankings editor at U.S. News, said the overall 0.1 difference in rankings is probably meaningless, but the difference in safety scores likely conveys meaning.
“I would hate for any family or patient to make a decision based on five points or ten points, let alone one-tenth of a point,” Comarow said to The Cancer Letter. “Cancer is one of those broad specialties where a patient’s own cancer and a particular cancer center’s experience and expertise in dealing with that cancer is far more important than a few points difference in score between a couple of centers.”
Patient safety—where MSKCC scored in the excellent range and MD Anderson in the mediocre range—is another matter.
“[The safety metric] matters enough to make it an important part of the score,” Comarow said. “Whether patients should be making that the basis of a decision is an individual choice. I would look at something like that fairly carefully myself.
“I would look at something like pressure ulcers. And it would probably alert me to the necessity of having someone with me. If I am an inpatient following surgery, I can’t advocate for myself. That person would make sure that I am turned regularly and I am clean, and I am on one of those special mattresses that alternate pressure on different parts of the body. I would want someone watching after me, paying attention to some of these potential deficiencies or problems that the patient safety score might reflect. It would probably not determine my final choice.”
In safety overall, MSKCC got the score of five—the highest. MD Anderson scored two. The same difference in scores was observed in the metric of preventing pressure ulcers, or skin breakdown from prolonged bed rest, weight, dressing and other factors.
“In cancer, more than anything else, I would want to know how patients have done at that center,” Comarow said.
The U.S. News index measures 30-day survival, and both centers received the same top score of 10. “I wish like crazy that five-year survival were available, but it’s not,” Comarow said. “Since five-year survival isn’t available, then a short-term proxy is what we use. And here you can see that most of the centers at the top of the rankings do very, very well.”
The Problem of Reliable Metrics
Survival is indeed recognized the gold standard in drug trials, but it’s not viewed as a reliable measurement of quality of care, especially in comparison of outcomes at different institutions.
Different hospitals get different mixes of patients. For example, a patient at a top-level academic cancer center may be there because of severity of her disease. On the other hand, that same patient could be better educated, more motivated, better insured, healthier and able to travel.
There is no known way to adjust for such differences in patient populations (The Cancer Letter, March 18, 2011).
The reputational score—still a dominant component of the U.S. News index—seems not to have reflected what insiders, particularly faculty, describe as morale problems at MD Anderson (The Cancer Letter, May 23).
Many insiders attribute these problems to top leadership, particularly the institution’s president, Ronald DePinho. Some of these insiders say morale problems have affected the quality of care (The Cancer Letter, Sept. 20, 2013).
The U.S. News index isn’t calibrated to measure such changes inside institutions, and in fact seeks to avoid gauging sudden changes by averaging out some metrics over three years.
According to a document describing the methodology in the index:
“The process component of the IHQ score is represented by a hospital’s reputation. For these rankings, the concept of reputation speaks to an institutional ability to develop and sustain a system that delivers high-quality care to especially challenging patients. It can be seen as a form of peer review. A hospital’s reputational score is based on the average of responses from the three most recent annual surveys of board-certified physicians conducted for the Best Hospitals rankings, which for the 2014-15 rankings were conducted in 2012, 2013 and 2014.
“A random sample of 200 board-certified physicians is selected in each specialty from the American Medical Association (AMA) Physician Masterfile, a database of more than 850,000 physicians. The physician sample is stratified by census region—West, Northeast, South and Midwest—and by specialty to ensure appropriate representation. The final aggregated sample includes both federal and nonfederal medical and osteopathic physicians in all 50 states and the District of Columbia.
“The surveyed physicians were asked to nominate the hospitals in their specific field of care, irrespective of expense or location, they consider best for patients with serious or difficult conditions. Up to five hospitals could be listed. For the 2014-15 rankings, a new initiative was added to address declining response rates by the survey sample drawn from the AMA database and to evaluate a broader set of physician responses. An additional survey was conducted with the Doximity online panel of physicians. The results were analyzed separately and incorporated as a small percentage of the reputation score for 2014.”
The methodology paper shows that 31.3 percent of the 200 cancer specialists randomly selected to set the score responded to the survey. After that, the score is combined with another survey and averaged over three years. Last year, MD Anderson’s reputational score was 67.7 percent. This year, it’s 67.5. MSKCC scored 62 percent last year and 64.7 percent this year.
Of course, the weight given to the score has been reduced, a change that favors MSKCC.
Past Irregularities With Data Submitted by MD Anderson
Last year, this publication reported that systematic misclassification of emergency patients at MD Anderson Cancer Center has enhanced that institution’s rating by U.S. News over the past seven years.
The miscounting led to exclusion of nearly 40 percent of admissions, was discovered and corrected in mid-2009, but no reliable way could be found to adjust the results to reflect the missing data, officials at U.S. News and MD Anderson confirmed.
Since U.S. News averages data over three years, the results of the MD Anderson top rating by the magazine released last year were still partially based on tainted data.
Insiders say that MD Anderson had been submitting incorrect data submitted to Centers for Medicare and Medicaid Services. U.S. News doesn’t ask hospitals to provide data directly, relying instead on government databases, which are less prone to tampering.
The problem was caused by an error, MD Anderson officials said. The error was discovered by MSKCC officials and acknowledged by their counterparts at MD Anderson, but U.S. News editors said a recount would be impossible, because of the volume of missing data. Just as importantly, the methodological pillar of the index—not accepting data from institutions directly—was at stake.
Insiders, including Comarow, say that had data been submitted, the top spot would have been likely to have been traded by MD Anderson and MSKCC (The Cancer Letter, July 19, 2013).
The data are averaged over three years, and the last of the tainted data were used in the U.S. News 2013 rankings.
“The statistics based on volume now reflect reality at MD Anderson,” Comarow said of the current year’s results. “These three years should be clean.”
MSKCC is “proud and honored” to be named the number one hospital for cancer care in the nation, officials said.
“We are delighted and honored to once again be a top hospital for cancer,” said MSKCC President and CEO Craig Thompson in a statement. “But this recognition is really a reflection of an entire community of exceptional people—our dedicated staff of almost 13,000 and our patients, who have placed their trust in our care. We may be number one in the national rankings for this year, but our patients are number one to us every day.
“While competition among medical institutions in the New York metro area is high (MSK ranked number seven of all hospitals in New York City), MSK’s singular focus on cancer means the hospital far outranks its neighbors for cancer care.”
Ranking second this year is a sign of success, MD Anderson officials said in a statement.
“MD Anderson has been ranked either first or second for the 25 years that the U.S. News & World Report ‘Best Hospitals’ survey has been conducted. We’re honored by the designation and thank U.S. News & World Report for providing this important service to readers.
“Ranking No. 2 this year among all of the other superb cancer hospitals reflects our success in recruiting and retaining world recognized faculty and staff, extending our reach globally for the benefit of cancer patients, increasing our philanthropic support and securing our financial future.
“We continue to be recognized as one of the best cancer hospitals and take great pride in this year’s ranking as in other years. We also congratulate Memorial Sloan Kettering Cancer Center on their No. 1 ranking this year, as well as all of the superb cancer hospitals that received recognition.”
The U.S. News index was never intended to provide bragging rights—and marketing advantage—based on minute differences in scores, Comarow said.
“It’s never been something that we’ve especially cared for,” he said. “The differences between No. 1 and No. 2 and No. 3 and No. 4 is so small that there are very few specialties where these differences between the very top hospitals amount to a hill of beans.
“It’s just human to think of horse races, and if you are a marketing person, it’s part of your DNA.”
Matthew Bin Han Ong contributed to this story.
