“Inflation” and Bias in Letters of Recommendation
OrthoBuzz occasionally receives posts from guest bloggers. This guest post comes Christopher Dy, MD, MPH in response to 2 recent studies in JBJS Open Access.
It’s that time of year when many of us write and review letters of recommendation (LOR) for orthopaedic residency applicants. LOR have always played a large part in the ranking and selection of applicants, and they may be weighed even more heavily during the upcoming “virtual-interview” season. Many applicants present remarkable objective measures of accomplishment, accompanied by 3 to 4 glowing LOR from colleagues. But can all these people really be that good? I am not the first to wonder whether “grade inflation” has crept into the writing of recommendation letters.
As letter writers, we fulfill two important, but potentially conflicting, roles:
- Mentors: We want to support the applicants who have worked with us.
- Colleagues: We want to be honest with our peers who are reviewing the applications.
In addition, this dynamic is now playing out in the context of our profession’s efforts to increase the racial and gender diversity of the orthopedic workforce. This begs the question as to whether there are differences in how we describe applicants based on race and gender.
To help answer that question, our research team analyzed LOR from 730 residency applications made during the 2018 match. Using text-analysis software, we examined race- and gender-based differences in the frequency of words from 5 categories:
- Agency (e.g., “assertive,” “confident,” “outspoken”)
- Communal (e.g., “careful,” “warm,” “considerate”)
- Grindstone (e.g., “dedicated,” “hardworking,” “persistent”)
- Ability (e.g., “adept,” “intelligent,” “proficient”)
- Standout (e.g., “amazing,” “exceptional,” “outstanding”)
We hypothesized that men and women would be described differently, expecting, for example, that agency terms would be used more often for describing men and communal terms more often for describing women.
Our hypothesis was almost entirely wrong. The agency, communal, grindstone, and ability words were used similarly for both male and female applicants. Standout words were used slightly (but significantly) more often in letters describing women. When comparing word usage in LOR for white candidates to those of applicants underrepresented in orthopedics, standout words were more commonly used in the former, and grindstone words were more commonly used in the latter. Interestingly, neither gender nor race word-usage differences were observed when LOR using the American Orthopaedic Association (AOA) standardized letter format were analyzed.
In a separate but related study, we looked at the scores given in each of the 9 domains of the AOA standardized letter of recommendation. These scores clustered far “to the right,” with 75% of applicants receiving a score of ≥85 in all domains. While I am certain that orthopaedic residency applicants are universally very talented all-around, this lopsided scoring distribution makes it very hard to differentiate among candidates. Furthermore, 48% of applicants were indicated as “ranked to guarantee match.” I suspect that the “ranked to guarantee match” recommendation is made more often than it should be. Again, this “inflation” makes it challenging for applicants to stand out – and may have especially important implications in this year’s virtual-interview environment.
What I take away from these two studies is that our natural tendency as orthopedic surgeons is to write effusively about our student mentees. Perhaps the differences in how we describe applicants based on their race and gender can be mitigated by using the AOA standardized letter format, but that format has a profound ceiling effect that makes it hard to discern the “cream of the crop.”
As a specialty, we are truly fortunate to have such excellent students vying to be orthopedic surgeons, and it is quite possible that nearly all of the applicants applying for our residency programs would make great orthopedic surgeons. However, it would help us to have a baseline measure of how we rate our students. Having some kind of benchmark against which to measure our past rankings and how they compare to those of our peers would help immensely.
Christopher Dy, MD, MPH is a hand and wrist surgeon, an assistant professor of orthopaedic surgery at Washington University School of Medicine in St. Louis, and a member of the JBJS Social Media Advisory Board.
“True Grit” Among Millennial Orthopaedists in Training
In a survey-based study published in the July 17, 2019 issue of The Journal of Bone & Joint Surgery, Samuelsen et al. made a hypothesis arising from a popularly held assumption about millennials: that orthopaedic residency applicants (predominantly millennials, with a mean age of 27.3) would have lower grit and self-control scores than attending orthopaedic surgeons (mean age of 51.3). The findings contradicted that hypothesis.
Surveys were completed by 655 (28%) of 2,342 attendings who received the questionnaire and by 455 (50.8%) of 895 orthopaedic residency applicants from the 2016-2017 resident match. The authors found that the residency applicants demonstrated higher mean grit scores (4.12 of 5.0) than the attending orthopaedic surgeons (4.03) (p <0.01). When compared to the general population, residency applicants and attendings scored in the 70th and 65th percentiles of grit, respectively.
The American Heritage Dictionary defines “grit” as “indomitable spirit” or “pluck.” In the medical literature, where “grit” has received a lot of attention lately, the concept is defined as “steadfast passion and perseverance for long-term goals, especially in the setting of hardship and setbacks.” However grit is defined, Samuelson et al. say it “has consistently been proven to be associated with success in…medical environments.”
Three other interesting findings:
- There were no significant differences in self-control or conscientiousness scores between the 2 groups.
- Both age and number of years in practice were positively correlated with self-control scores in the practicing-surgeon group.
- Among attending surgeons, the number of publications correlated with higher grit, self-control, and conscientiousness scores.
Samuelson et al. offer a possible explanation for the impressive grit scores among residency applicants: matching into orthopaedic residency has become increasingly competitive over the past several decades and “applicants to orthopaedic surgery…tend to represent the individuals at the top of their medical school classes.” Conversely, the authors suggest that grit, self-control, and conscientiousness scores could be used to identify applicants, residents, or junior staff “who are at risk for attrition during training or burnout in their careers.”
Having postulated that, however, the authors are quick to add that “it is unclear if [these findings] will be predictive of career success in the next generation of orthopaedists.”
Click here to see a 1-minute video commentary about these findings by Chad A. Krueger, MD, JBJS Deputy Editor for Social Media.
Residents in the OR—What’s the Risk?
Orthopaedic educators have long confronted the subtle implication that resident participation in surgical care can contribute to patient harm or even death. While there have been numerous changes in residency education to improve the supervision and training of residents, the reality is that surgical trainees have to learn how to operate. This fact can leave surgical patients understandably nervous, and many of them heave heard rumors of a “July effect”—a hypothetical increase in surgery-related complications attributed to resident education at the beginning of an academic year. To provide further clarity on this quandary, in the November 21, 2018 issue of The Journal, Casp et al. examine the relationship between complication rates after lower-extremity trauma surgery (for hip fractures, predominantly), the participation and seniority of residents, and when during the academic year the surgery occurred.
The authors used the NSQIP surgical database to examine >1,800 patient outcomes after lower-extremity surgery according to academic-year quarter and the postgraduate year of the most senior resident involved in the case. The analysis revealed two major findings:
- Overall, there was no “July effect” at the beginning of the academic year in terms of composite complication rates.
- Cases involving more senior residents were associated with an increased risk of superficial surgical site infection during the first academic quarter.
While the authors were unable to provide a precise reason for the second finding, they hypothesized that it could have been related to more stringent data collection early in the academic year, senior-resident inexperience with newly increased responsibilities, or the warm-temperature time of year in which the infections occurred. Casp et al. emphasize that the database used in the study was not robust in terms of documenting case details such as complexity and the degree of resident autonomy, which makes cause-and-effect conclusions impossible to pinpoint.
Although this large database study does not answer granular questions regarding the appropriate role of residents in orthopaedic surgery, it should stimulate further research in this area. Gradually increasing responsibility is necessary within residency programs so that residents develop the skills and decision-making prowess necessary for them to succeed as attending surgeons. Studies like this help guide future research into the important topic of graduate medical education, and they provide patients with some reassurance that the surgical care they receive is not affected by the time during the academic-calendar year in which they receive it.
Marc Swiontkowski, MD
JBJS Editor-in-Chief
More Efficient Orthopaedic Education Needed
The October 4, 2017 issue of JBJS contains another in a series of “What’s Important” personal essays from orthopaedic clinicians. This “What’s Important” article comes from Drs. Peter Scoles and Shepard Hurwitz.
The authors suggest that integration of medical school curricula with the first year of postgraduate training is a practical approach to improving efficiency and reducing costs to both doctors in training and the academic medical centers that help train them. In explaining specific ways to change the paradigm for training orthopaedic surgeons, the authors conclude that an integrative approach would accelerate the process for qualified candidates, while lowering costs and ensuring adequate training opportunities for all.
If you would like JBJS to consider your “What’s Important” story for publication, please submit a manuscript via Editorial Manager. When asked to select an article type, please choose Orthopaedic Forum and include “What’s Important:” at the beginning of the title.
Because they are personal in nature, “What’s Important” submissions will not be subject to the usual stringent JBJS peer-review process. Instead, they will be reviewed by the Editor-in-Chief, who will correspond with the author if revisions are necessary and make the final decision regarding acceptance.
Try Clinical Classroom
Recent Comments