“Inflation” and Bias in Letters of Recommendation
OrthoBuzz occasionally receives posts from guest bloggers. This guest post comes Christopher Dy, MD, MPH in response to 2 recent studies in JBJS Open Access.
It’s that time of year when many of us write and review letters of recommendation (LOR) for orthopaedic residency applicants. LOR have always played a large part in the ranking and selection of applicants, and they may be weighed even more heavily during the upcoming “virtual-interview” season. Many applicants present remarkable objective measures of accomplishment, accompanied by 3 to 4 glowing LOR from colleagues. But can all these people really be that good? I am not the first to wonder whether “grade inflation” has crept into the writing of recommendation letters.
As letter writers, we fulfill two important, but potentially conflicting, roles:
- Mentors: We want to support the applicants who have worked with us.
- Colleagues: We want to be honest with our peers who are reviewing the applications.
In addition, this dynamic is now playing out in the context of our profession’s efforts to increase the racial and gender diversity of the orthopedic workforce. This begs the question as to whether there are differences in how we describe applicants based on race and gender.
To help answer that question, our research team analyzed LOR from 730 residency applications made during the 2018 match. Using text-analysis software, we examined race- and gender-based differences in the frequency of words from 5 categories:
- Agency (e.g., “assertive,” “confident,” “outspoken”)
- Communal (e.g., “careful,” “warm,” “considerate”)
- Grindstone (e.g., “dedicated,” “hardworking,” “persistent”)
- Ability (e.g., “adept,” “intelligent,” “proficient”)
- Standout (e.g., “amazing,” “exceptional,” “outstanding”)
We hypothesized that men and women would be described differently, expecting, for example, that agency terms would be used more often for describing men and communal terms more often for describing women.
Our hypothesis was almost entirely wrong. The agency, communal, grindstone, and ability words were used similarly for both male and female applicants. Standout words were used slightly (but significantly) more often in letters describing women. When comparing word usage in LOR for white candidates to those of applicants underrepresented in orthopedics, standout words were more commonly used in the former, and grindstone words were more commonly used in the latter. Interestingly, neither gender nor race word-usage differences were observed when LOR using the American Orthopaedic Association (AOA) standardized letter format were analyzed.
In a separate but related study, we looked at the scores given in each of the 9 domains of the AOA standardized letter of recommendation. These scores clustered far “to the right,” with 75% of applicants receiving a score of ≥85 in all domains. While I am certain that orthopaedic residency applicants are universally very talented all-around, this lopsided scoring distribution makes it very hard to differentiate among candidates. Furthermore, 48% of applicants were indicated as “ranked to guarantee match.” I suspect that the “ranked to guarantee match” recommendation is made more often than it should be. Again, this “inflation” makes it challenging for applicants to stand out – and may have especially important implications in this year’s virtual-interview environment.
What I take away from these two studies is that our natural tendency as orthopedic surgeons is to write effusively about our student mentees. Perhaps the differences in how we describe applicants based on their race and gender can be mitigated by using the AOA standardized letter format, but that format has a profound ceiling effect that makes it hard to discern the “cream of the crop.”
As a specialty, we are truly fortunate to have such excellent students vying to be orthopedic surgeons, and it is quite possible that nearly all of the applicants applying for our residency programs would make great orthopedic surgeons. However, it would help us to have a baseline measure of how we rate our students. Having some kind of benchmark against which to measure our past rankings and how they compare to those of our peers would help immensely.
Christopher Dy, MD, MPH is a hand and wrist surgeon, an assistant professor of orthopaedic surgery at Washington University School of Medicine in St. Louis, and a member of the JBJS Social Media Advisory Board.
Nov. 14 Webinar – Diversity in Orthopaedics: Taking Action to Drive Change
In many areas of the US, the orthopaedic workforce does not mirror the patient population being treated. The need for workforce diversity is more than a social concern or a “good-business” practice. Diversity, or the lack of it, directly affects the quality of patient care as well as access to care.
On Wednesday, November 14, 2018 at 8:00 PM EST,the American Orthopaedic Association (AOA) and The Journal of Bone & Joint Surgery (JBJS) will co-host a one-hour complimentary webinar that offers practical advice on how to achieve greater diversity in your orthopaedic workforce. The guidance comes from four orthopaedists with an impressive track record of success in meeting this challenge head-on:
- Regis O’Keefe, MD, PhD, FAOA
- Mary O’Connor, MD, FAOA
- Julie Samora, MD, PhD, MPH
- Kristy Weber, MD, FAOA
Moderated by Lisa Lattanza, MD, Professor and Vice Chair of Diversity and Professionalism and Chief of Hand, Elbow, and Upper Extremity Surgery at UCSF, this webinar will conclude with a 15-minute Q&A session during which attendees can ask questions of the panelists.
Seats are limited, so REGISTER NOW.
Sept. 11 Webinar – Assessment of Bone Health for the Orthopaedic Surgeon
Orthopaedic care teams can play an active role in evaluating and optimizing their patients’ bone health to help prevent primary and secondary fragility fractures and to improve postsurgical outcomes. In just about any orthopaedic scenario, helping patients optimize their bone health is an imperative for the delivery of quality care.
On Tuesday, September 11, 2018 at 8 pm EDT, the American Orthopaedic Association (AOA) and The Journal of Bone & Joint Surgery (JBJS) will host a complimentary one-hour webinar that will cover the basics of a bone-health assessment by orthopaedists.
- Christopher Shuhart, MD will discuss the fundamentals of bone-related laboratory workups and bone densitometry studies.
- Joe Lane, MD, FAOA will identify bone-health “red flags” in orthopaedic patients, including common nutritional deficiencies.
- Paul Anderson, MD, FAOA will cover recent advances in bone-density measurements.
Moderated by Douglas Lundy, MD, MBA, FAOA, orthopaedic trauma surgeon at Resurgens Orthopaedics, this webinar will include a 15-minute live Q&A session during which attendees can ask questions of the panelists.
Seats are limited so REGISTER NOW.
Fragility Fracture Workshop & Symposium—June 28 & 29, 2018
On Thursday evening, June 28 and all day Friday, June 29 in Boston, The American Orthopaedic Association (AOA) and the National Association of Orthopaedic Nurses (NAON) will present two educational/networking events concentrating on secondary fragility fracture prevention.
The Thursday evening Workshop, available only to those attending the Friday Symposium, will convene clinicians with expertise in counseling and treating fragility fracture patients. “This new two-hour workshop provides an additional opportunity to learn more about identifying, assessing, counseling, and treating fragility fracture patients,” said program co-chair Debra Sietsema, PhD, RN. “The Workshop also includes special breakout stations on calcium, FRAX, and the AOA’s ‘Own the Bone’ initiative.”
The all-day Symposium on Friday focuses on how to establish a multidisciplinary secondary fragility fracture program. In addition, the Symposium will include relevant case studies demonstrating how to translate the principles into hospital, private-practice, or clinic settings. “This Symposium is a great opportunity for orthopaedic surgeons and allied health professionals to get the full picture in one day,” said Dr. Sietsema. “Attendees will gain both basic and expanded knowledge to put their programs in place.”
Register by May 15 to receive early-bird pricing for these important events. NAON members and clinicians from enrolled Own the Bone institutions save an additional $50.
More Progress in Preventing Secondary Fragility Fractures
How well do fracture liaison services (FLSs) work in terms of patients who’ve had a fragility fracture receiving a recommendation for anti-osteoporosis treatment? Very well, according to findings from an analysis of more than 32,000 patients by Dirschl and Rustom in the April 18, 2018 edition of The Journal of Bone & Joint Surgery.
A fracture liaison service is a coordinated, multidisciplinary model of care designed to reduce the risk of future fractures among patients who’ve sustained a primary fragility fracture. (Click here for another recent JBJS article about the FLS model.) The American Orthopaedic Association (AOA) has been a major proponent of the FLS model, and it is a cornerstone of the AOA’s “Own the Bone” national quality-improvement program.
Dirschl and Rustom found that between 2009 and 2016, at 147 sites participating in an FLS through Own the Bone, 72.8% of 32,671 patients initially evaluated for a fragility fracture received a recommendation for anti-osteoporosis treatment. That’s a vast improvement compared with previous reports that indicate only 20% of patients with a fragility fracture received either an osteoporosis evaluation or treatment. In this current study, a sedentary lifestyle and having a parent who had sustained a hip fracture were the patient factors associated with those most likely to receive a recommendation for treatment.
OrthoBuzz editors were surprised to read that anti-osteoporosis treatment was initiated in only 12.1% of the patients in this study. When we asked JBJS Editor-in-Chief Marc Swiontkowski, MD for a further explanation, he noted that the study captured data only from the initial post-fracture encounter between patients and FLS clinicians. The percentage of patients initiating treatment would have been much higher, he said, if the data had included those who followed up their initial FLS evaluation with a primary care physician. He also remarked that some people are dissuaded from taking an FDA-approved prescription anti-osteoporosis medication by the disproportionate focus on side effects that patients read in social media and the lay press. And there are some patients for whom prescription anti-osteoporosis drugs are truly contraindicated.
But with an estimated 2 million people in the US sustaining a fragility fracture each year, these results indicate substantial progress in practices that will prevent secondary fractures.
Click here for a listing of upcoming Own the Bone events.
April 12 Webinar – Managing Osteoarthritis: Moving from Volume to Value
According to the CDC, in 2013, the total national arthritis-related medical care costs and earnings losses among adults were $303.5 billion, or 1% of the 2013 US Gross Domestic Product.
One response to statistics like that is the notion of “value-based health care.” How far has the orthopaedic community moved from a volume/fee-for-service-based model to one in which patients achieve the best possible musculoskeletal outcomes, payers expend the fewest possible dollars, and providers throughout the episodes of care are fairly compensated for their skill and compassion?
On Thursday, April 12, 2018 at 8:00 pm EDT, the American Orthopaedic Association (AOA) and The Journal of Bone & Joint Surgery (JBJS) will host a complimentary one-hour webinar that will answer these thorny questions by discussing the cost drivers behind the problem, where arthritis management stands currently, and where the value-based care bandwagon is heading.
Kevin Shea, MD, an expert in developing clinical practice guidelines, will discuss the crucial differences between “irrational variation” and “rational, patient-centered variation.”
Antonia Chen, MD, director of arthroplasty research at Harvard Medical School, will demystify the many attempts to measure and improve the quality of joint replacement and will address quality and value in the nonoperative management of osteoarthritis.
Gregory Brown, MD, a Tacoma, Washington-based surgeon specializing in knee reconstruction, will peer into the future of health insurance, patient empowerment, and robust orthopaedic registries.
Moderated by Douglas Lundy, MD, orthopaedic trauma surgeon at Resurgens Orthopaedics, this webinar will include a 15-minute live Q&A session during which attendees can ask questions of the panelists.
Webinar—Helping Clinically Excellent Colleagues Who Have Challenging Behaviors
Sometimes the most talented, skilled physicians with whom you work are also prone to displaying challenging behaviors. Often, these physicians are not cognizant of how their colleagues perceive them, so how can you—as the supervisor, friend, and/or peer of such clinicians—help ensure that patients continue to benefit from their clinical and surgical gifts without behavioral difficulties diminishing their contributions?
On Thursday, October 26, 2017 at 8:00 pm EDT, the American Orthopaedic Association (AOA) and The Journal of Bone & Joint Surgery (JBJS) will host a complimentary webinar that will deliver practical and effective methods you can use to help physicians who are clinically outstanding, but behaviorally difficult, start to make remedial changes.
The presentations about how to be helpful to such colleagues will be led by:
- Gerald Hickson, senior VP for Quality, Safety, and Risk Prevention at Vanderbilt University Medical Center
- William Hopkinson, professor of orthopaedic surgery at Loyola Medicine
- George Russell, professor and chair of orthopaedic traumatology at the University of Mississippi Medical Center
Moderated by Dr. Douglas Lundy, orthopaedic trauma surgeon at Resurgens Orthopaedics, this webinar will include a 15-minute live Q&A session during which attendees can ask questions of the panelists.
Seats are limited, so register now!
JBJS/AOA Webinar–Implementing Medicare’s Quality-Incentive Programs
Are you confused and frustrated by Medicare’s Quality-Incentive Programs, such as the Merit Based Incentive Payment System (MIPS), Comprehensive Care for Joint Replacement (CJR) program, and the Surgical Hip and Femur Fracture Treatment (SHFFT) model? If so, this webinar is for you.
On Tuesday, August 15, 2017 at 8:00 PM EDT, The Journal of Bone & Joint Surgery (JBJS) and the American Orthopaedic Association (AOA) will host a complimentary LIVE webinar featuring the following speakers and topics:
- Brian McCardel, MD will discuss choosing MIPS-related quality measures, improving performance on those measures, and qualifying for bonuses.
- Thomas Barber, MD, FAOA will focus on managing clinical care including how to deliver low-cost high-quality care for high-risk orthopaedic patients.
- Alexandra Page, MD will discuss partnering with hospitals and post-acute organizations to improve patient care and reap financial rewards.
Moderated by Douglas Lundy, MD, FAOA, the webinar will include a live Q&A session between the audience and panelists.
Upcoming Own the Bone Events
Spring and Summer 2017 are busy seasons for Own the Bone, the American Orthopaedic Association’s national post-fracture, systems-based, multidisciplinary fragility fracture prevention initiative:
Webinar: Components of a Secondary Fracture Prevention Program – May 4
On Thursday, May 4, at 5:00pm CDT (6:00pm EDT) Paul A. Anderson, MD, FAOA, from the University of Wisconsin, and Karen Cummings, PA-C, from the University of Michigan, will discuss the components of a successful secondary fracture prevention program.
Join the National Association of Orthopaedic Nurses (NAON) and The American Orthopaedic Association for this full-day event on Friday, June 23. Attendees will receive a Fragility Fracture Symposium Certificate of Completion and continuing education credit.
Fragility Fracture Workshop at PAOS Annual Conference – August 21, Baltimore, MD
Join Physician Assistants in Orthopaedic Surgery (PAOS) and Own the Bone for a full-day fragility fracture and bone health workshop on the first day of the PAOS Annual Conference, Monday, August 21, in Baltimore.
Own the Bone Webinar: Atypical Fractures and Osteo Medications
On Thursday, February 23, 2017, at 6:00 pm EST, the Own the Bone initiative will offer a webinar titled “Atypical Fractures and Osteoporosis Medication Considerations”
James Goulet, MD, from the University of Michigan, will discuss atypical fractures and other rare outcomes of the use of osteoporosis medication, including what to look for and how to treat these occurrences. He will also address drug holidays, and how and when to continue treatment on these complex cases.
The American Orthopaedic Association (AOA) developed Own the Bone as a quality improvement program to address the osteoporosis treatment gap and prevent subsequent fragility fractures.
0.75 hour of CME credit is available.
Recent Comments