OrthoBuzz occasionally receives posts from guest bloggers. This guest post comes Christopher Dy, MD, MPH in response to 2 recent studies in JBJS Open Access.
It’s that time of year when many of us write and review letters of recommendation (LOR) for orthopaedic residency applicants. LOR have always played a large part in the ranking and selection of applicants, and they may be weighed even more heavily during the upcoming “virtual-interview” season. Many applicants present remarkable objective measures of accomplishment, accompanied by 3 to 4 glowing LOR from colleagues. But can all these people really be that good? I am not the first to wonder whether “grade inflation” has crept into the writing of recommendation letters.
As letter writers, we fulfill two important, but potentially conflicting, roles:
- Mentors: We want to support the applicants who have worked with us.
- Colleagues: We want to be honest with our peers who are reviewing the applications.
In addition, this dynamic is now playing out in the context of our profession’s efforts to increase the racial and gender diversity of the orthopedic workforce. This begs the question as to whether there are differences in how we describe applicants based on race and gender.
To help answer that question, our research team analyzed LOR from 730 residency applications made during the 2018 match. Using text-analysis software, we examined race- and gender-based differences in the frequency of words from 5 categories:
- Agency (e.g., “assertive,” “confident,” “outspoken”)
- Communal (e.g., “careful,” “warm,” “considerate”)
- Grindstone (e.g., “dedicated,” “hardworking,” “persistent”)
- Ability (e.g., “adept,” “intelligent,” “proficient”)
- Standout (e.g., “amazing,” “exceptional,” “outstanding”)
We hypothesized that men and women would be described differently, expecting, for example, that agency terms would be used more often for describing men and communal terms more often for describing women.
Our hypothesis was almost entirely wrong. The agency, communal, grindstone, and ability words were used similarly for both male and female applicants. Standout words were used slightly (but significantly) more often in letters describing women. When comparing word usage in LOR for white candidates to those of applicants underrepresented in orthopedics, standout words were more commonly used in the former, and grindstone words were more commonly used in the latter. Interestingly, neither gender nor race word-usage differences were observed when LOR using the American Orthopaedic Association (AOA) standardized letter format were analyzed.
In a separate but related study, we looked at the scores given in each of the 9 domains of the AOA standardized letter of recommendation. These scores clustered far “to the right,” with 75% of applicants receiving a score of ≥85 in all domains. While I am certain that orthopaedic residency applicants are universally very talented all-around, this lopsided scoring distribution makes it very hard to differentiate among candidates. Furthermore, 48% of applicants were indicated as “ranked to guarantee match.” I suspect that the “ranked to guarantee match” recommendation is made more often than it should be. Again, this “inflation” makes it challenging for applicants to stand out – and may have especially important implications in this year’s virtual-interview environment.
What I take away from these two studies is that our natural tendency as orthopedic surgeons is to write effusively about our student mentees. Perhaps the differences in how we describe applicants based on their race and gender can be mitigated by using the AOA standardized letter format, but that format has a profound ceiling effect that makes it hard to discern the “cream of the crop.”
As a specialty, we are truly fortunate to have such excellent students vying to be orthopedic surgeons, and it is quite possible that nearly all of the applicants applying for our residency programs would make great orthopedic surgeons. However, it would help us to have a baseline measure of how we rate our students. Having some kind of benchmark against which to measure our past rankings and how they compare to those of our peers would help immensely.
Christopher Dy, MD, MPH is a hand and wrist surgeon, an assistant professor of orthopaedic surgery at Washington University School of Medicine in St. Louis, and a member of the JBJS Social Media Advisory Board.
