Calculating Individual Complication Rates Is Complicated

Rating hospitals on the basis of complications is one thing, but when you publish complication-rate scorecards for individual surgeons, as ProPublica did recently with nearly 17,000 surgeons nationwide, things can get personal.

ProPublica, an independent investigative-journalism group, examined five years of Medicare records for eight common elective procedures, three of which—knee and hip replacements and spinal fusions—orthopaedists perform. For each of the eight procedures, a panel of at least five physicians, including relevant specialists, reviewed 30-day readmission data to determine whether the readmission represented a complication; if a majority agreement was not achieved, the case was excluded from analysis. The analysis also excluded trauma and other high-risk cases, along with cases that originated in emergency departments.

Overall, complication rates were 2% to 4%. About 11% of the doctors accounted for about 25% of the complications.

In a ProPublica article about the scorecard, Dr. Charles Mick, former president of the North American Spine Society, is quoted as saying, “Hopefully, [the scorecard] will be a step toward a culture where transparency and open discussion of mistakes, complications, and errors will be the norm and not something that’s hidden.”  For its part, the AAOS responded with a press release that welcomed transparency but cautioned that “the surgical complication issue is much more complex, and cannot be effectively addressed without considering all of the variables that impact surgery, care, and outcomes.”

Pre-emptively, ProPublica clarified its methods in a separate article. Any 30-day readmission that the panel determined to be a complication was assigned to the surgeon who performed the original procedure. After compiling a raw complication rate for each doctor, researchers screened each patient’s health record and assigned a “health score.” That health score was used as part of a mixed-effects statistical model to determine an individual’s adjusted complication rate. No rate is reported if a surgeon performed a procedure fewer than 20 times.

Over the years, physician groups have complained that conclusions derived from Medicare data are inherently flawed, an argument that one orthopaedist made in the ProPublica article, citing the “multitude of inaccurate and confusing information that is provided to state and federal organizations.” Interestingly, two renowned patient-outcome experts cited in the ProPublica article came to separate conclusions. Dr. Thomas Lee, chief medical officer at healthcare-metrics consultancy Press Ganey, was quoted as saying that “the methodology was rigorous and conservative,” while Dr. Peter Pronovost, director of the Armstrong Institute for Patient Safety and Quality at Johns Hopkins, told ProPublica in an email just prior to the scorecard release that “it would be highly irresponsible to present this to the public in its current form, or to make an example of any surgeon based on faulty data analysis.”

In another take on ProPublica’s ratings, radiologist Saurabh Jha spins a yarn on KevinMD  of two fictional orthopaedists, Dr. Cherry Picker and Dr. Morbidity Hunter. The moral of this tale, Dr. Jha says, is that ProPublica’s scorecard is “a reservoir of Sampson’s paradox…when the data says ‘bad surgeon,’ the surgeon might in fact be a Top Gun—a technically gifted Morbidity Hunter—the last hope of the poor and sick.”

Obviously the ProPublica scorecard has touched many a nerve among hip/knee-reconstruction and spine surgeons. Have you looked at your numbers? What do you think? Please join the discussion by clicking on the “Leave a comment” button in the box above, next to the article title.