U.S. News & World Report’s annual ranking of the 50 best U.S. hospitals is flawed, according to a study in the Annals of Internal Medicine, which found that the list more often reflects facilities’ reputations rather than the quality of care delivered.

To determine the role of facilities’ reputation in U.S. News hospital rankings, Dr. Ashwini Sehgal, a physician at Case Western Reserve University and MetroHealth Medical Center in Cleveland, calculated the reputation scores of the top 50 hospitals across 12 specialties in the magazine’s 2009 list and compared them with their overall score. Sehgal determined that there was little relationship between a hospital’s reputation and its scores on objective quality measures, such as mortality.

Specifically, Sehgal found that reputation scores coincided with the overall ranking 100% of the time for the top hospital in each specialty, 97% of the time for the top five hospitals in each specialty and 89% of the time for the top 20 hospitals in each specialty.

To tabulate reputation scores, U.S. News has about 250 board-certified physicians vote on hospitals in their respective specialty. These scores account for 33% of a hospital’s overall rating. Facilities that lack national recognition despite attaining high quality often receive zero or a few points for their reputation score and therefore are unlikely to be among the top-ranking facilities, Seghal notes.

Furthermore, Sehgal contends that the reputation component is vulnerable to significant variation. Although the editor of U.S. News says the reputational score is a “valid and legitimate form of peer review,” he acknowledges that the survey traditionally identifies hospitals that are “national in scope,” according to Bloomberg.

The reputation scores are detrimental to all hospitals, Sehgal says, stating that they may impede quality improvement efforts because even a successful improvement program is unlikely to raise a hospital’s ranking. Hospitals with high reputation scores also should be wary of the measure, because that reputation can “deteriorate quickly” and facilities may lose their status following a single negative event, according to Sehgal.

Noting that consumers use U.S. News hospital rankings to make decisions about health care, Sehgal contends that the magazine needs to alter or eliminate the reputation score and address additional care factors, like patient satisfaction (Sehgal, Annals of Internal Medicine, 4/20 )