“Inflation” and Bias in Letters of Recommendation
OrthoBuzz occasionally receives posts from guest bloggers. This guest post comes Christopher Dy, MD, MPH in response to 2 recent studies in JBJS Open Access.
It’s that time of year when many of us write and review letters of recommendation (LOR) for orthopaedic residency applicants. LOR have always played a large part in the ranking and selection of applicants, and they may be weighed even more heavily during the upcoming “virtual-interview” season. Many applicants present remarkable objective measures of accomplishment, accompanied by 3 to 4 glowing LOR from colleagues. But can all these people really be that good? I am not the first to wonder whether “grade inflation” has crept into the writing of recommendation letters.
As letter writers, we fulfill two important, but potentially conflicting, roles:
- Mentors: We want to support the applicants who have worked with us.
- Colleagues: We want to be honest with our peers who are reviewing the applications.
In addition, this dynamic is now playing out in the context of our profession’s efforts to increase the racial and gender diversity of the orthopedic workforce. This begs the question as to whether there are differences in how we describe applicants based on race and gender.
To help answer that question, our research team analyzed LOR from 730 residency applications made during the 2018 match. Using text-analysis software, we examined race- and gender-based differences in the frequency of words from 5 categories:
- Agency (e.g., “assertive,” “confident,” “outspoken”)
- Communal (e.g., “careful,” “warm,” “considerate”)
- Grindstone (e.g., “dedicated,” “hardworking,” “persistent”)
- Ability (e.g., “adept,” “intelligent,” “proficient”)
- Standout (e.g., “amazing,” “exceptional,” “outstanding”)
We hypothesized that men and women would be described differently, expecting, for example, that agency terms would be used more often for describing men and communal terms more often for describing women.
Our hypothesis was almost entirely wrong. The agency, communal, grindstone, and ability words were used similarly for both male and female applicants. Standout words were used slightly (but significantly) more often in letters describing women. When comparing word usage in LOR for white candidates to those of applicants underrepresented in orthopedics, standout words were more commonly used in the former, and grindstone words were more commonly used in the latter. Interestingly, neither gender nor race word-usage differences were observed when LOR using the American Orthopaedic Association (AOA) standardized letter format were analyzed.
In a separate but related study, we looked at the scores given in each of the 9 domains of the AOA standardized letter of recommendation. These scores clustered far “to the right,” with 75% of applicants receiving a score of ≥85 in all domains. While I am certain that orthopaedic residency applicants are universally very talented all-around, this lopsided scoring distribution makes it very hard to differentiate among candidates. Furthermore, 48% of applicants were indicated as “ranked to guarantee match.” I suspect that the “ranked to guarantee match” recommendation is made more often than it should be. Again, this “inflation” makes it challenging for applicants to stand out – and may have especially important implications in this year’s virtual-interview environment.
What I take away from these two studies is that our natural tendency as orthopedic surgeons is to write effusively about our student mentees. Perhaps the differences in how we describe applicants based on their race and gender can be mitigated by using the AOA standardized letter format, but that format has a profound ceiling effect that makes it hard to discern the “cream of the crop.”
As a specialty, we are truly fortunate to have such excellent students vying to be orthopedic surgeons, and it is quite possible that nearly all of the applicants applying for our residency programs would make great orthopedic surgeons. However, it would help us to have a baseline measure of how we rate our students. Having some kind of benchmark against which to measure our past rankings and how they compare to those of our peers would help immensely.
Christopher Dy, MD, MPH is a hand and wrist surgeon, an assistant professor of orthopaedic surgery at Washington University School of Medicine in St. Louis, and a member of the JBJS Social Media Advisory Board.
Owning the Bone in Spine Surgery
This post comes from Fred Nelson, MD, an orthopaedic surgeon in the Department of Orthopedics at Henry Ford Hospital and a clinical associate professor at Wayne State Medical School. Some of Dr. Nelson’s tips go out weekly to more than 3,000 members of the Orthopaedic Research Society (ORS), and all are distributed to more than 30 orthopaedic residency programs. Those not sent to the ORS are periodically reposted in OrthoBuzz with the permission of Dr. Nelson.
Approximately 20% of patients who undergo spine surgery have osteoporosis, which has a significant impact on spine-surgery complications such as failure of fixation devices and collapse fractures following fusion procedures. In a recent critical analysis review, authors focus on improving outcomes by identifying and optimizing patients with osteoporosis prior to spine surgery. The multidisciplinary team involved in that process should include primary care providers, endocrinologists, physical therapists, and orthopaedic surgeons.
The predominant tool for assessing bone mineral density (BMD) is dual x-ray absorptiometry. The diagnosis is based on a T score, which represents the number of standard deviations between the patient’s BMD and that of a healthy 30-year-old woman. Standard deviations ≤─2.5 define osteoporosis. The Z score is similar to the T score but compares the patient to an age- and sex-matched individual.
A history of low-energy fracture, such as a wrist fracture following a fall from a standing height, is considered a sentinel event for suspicion of fragility fractures. The combination of a fragility fracture and low BMD is considered to be severe osteoporosis. The most common form of osteoporosis is associated with a postmenopausal decrease in mineralization, but there are other causes. These include advanced kidney disease, hypogonadism, Cushing disease, vitamin D deficiency, anorexia and/or bulimia, rheumatoid arthritis, hyperthyroidism, primary hyperparathyroidism, and some medications (e.g., anticonvulsants, corticosteroids, heparin, and proton pump inhibitors).
Forty-seven percent of patients undergoing spine deformity surgery and 64% of cervical spine surgery patients have low vitamin D levels. Postoperative bone health can be enhanced in women ≥51 years old with daily intake of 800 to 1,000 units of vitamin D and 1,200 mg of daily calcium. There is no solid evidence that pre- or postoperative bisphosphonates have a positive impact on bone healing. Conversely, some series have shown that teriparatide, an anabolic parathyroid hormone, may improve time-to-fusion and help reduce screw pull-out after lumbar fusion in postmenopausal women.
Calcitonin has been shown to reduce the incidence of vertebral compression fracture, but there is no concrete evidence that it supports spine-fusion healing. Similarly, there is no strong evidence for the use of estrogen or selective estrogen receptor modulators in this surgical scenario. There is evidence that when the human monoclonal antibody denosumab is combined with teriparatide, spine-fusion healing may be improved relative to the use of teriparatide alone. Finally, the review article identifies screw size, screw position, and other surgical considerations that can improve fixation strength.
Using the “Own the Bone” practices promulgated by the American Orthopaedic Association and the technical considerations described in this review, we should be able to mitigate osteoporosis-related postoperative complications in spine-surgery patients.
Webinar–Diversity in Orthopaedics: Taking Action to Drive Change
On Wednesday, November 14, 2018 at 8:00 PM EST, the American Orthopaedic Association (AOA) and The Journal of Bone & Joint Surgery (JBJS) will co-host a one-hour complimentary webinar that offers practical advice on how to achieve greater diversity in your orthopaedic workforce. The guidance comes from five orthopaedists with an impressive track record of success in meeting this challenge head-on:
- Regis O’Keefe, MD, PhD, FAOA
- Mary O’Connor, MD, FAOA
- Julie Samora, MD, PhD, MPH
- Kristy Weber, MD, FAOA
- Lisa Lattanza, MD, FAOA
Recognizing the lack of diversity in the profession of orthopaedics as a critical issue, this webinar is one of many AOA initiatives supporting increased diversity within the profession.
Seats are limited, so REGISTER NOW.
JBJS: Aiding and Abetting Orthopaedic Residents
No matter how you look at it, orthopaedic residency is a relentlessly challenging five or six years. The Journal of Bone & Joint Surgery offers the following special services to make life and learning a little easier for orthopaedists in training:
- Complimentary access to all JBJS journals via the AOA’s Council of Orthopaedic Residency Directors (CORD)
- Guidance for getting the most out of your Journal Club
- Annual grants ($2,500) to support Journal Club activities
- Free access to JBJS Podcasts, Videos, and Webinars
- Opportunities to participate in the JBJS blog, OrthoBuzz
Residents who connect now with JBJS establish a solid foundation for a career of lifetime orthopaedic learning. Click on the “Residents” button under “Editorial Resources” at www.jbjs.org to find out more.
Own the Bone Webinar: Atypical Fractures and Osteo Medications
On Thursday, February 23, 2017, at 6:00 pm EST, the Own the Bone initiative will offer a webinar titled “Atypical Fractures and Osteoporosis Medication Considerations”
James Goulet, MD, from the University of Michigan, will discuss atypical fractures and other rare outcomes of the use of osteoporosis medication, including what to look for and how to treat these occurrences. He will also address drug holidays, and how and when to continue treatment on these complex cases.
The American Orthopaedic Association (AOA) developed Own the Bone as a quality improvement program to address the osteoporosis treatment gap and prevent subsequent fragility fractures.
0.75 hour of CME credit is available.
Guest Post: Own the Bone Improves Osteoporosis Care
OrthoBuzz occasionally receives posts from guest bloggers. This guest post comes from Brett A. Freedman, MD.
In the December 21, 2016 edition of the Journal of Bone & Joint Surgery, Bunta, et al. published an analysis of data from the Own the Bone quality improvement program collected between January 1, 2010 and March 31, 2015. Over this period of time, 125 sites prospectively collected detailed osteoporosis and bone health-related data points on men and women over the age of 50 who presented with a fragility fracture.
The Own the Bone initiative is more than a data registry; it’s a quality improvement program intended to provide a paradigm for increasing the diagnostic and therapeutic recognition (i.e. “response rate”) of the osteoporosis underlying fragility fractures among orthopaedic practices that treat these injuries. With more than 23,000 individual patients enrolled, and almost 10,000 follow-up records, this is the most robust dataset in existence on the topic.
This initiative has more than doubled the response rate among orthopaedic practices treating fragility fractures. The number of institutions implementing Own the Bone grew from 14 sites in 2005-6 to 177 in 2015. According to Bunta et al., 53% of patients enrolled in the Own the Bone quality Improvement program received bone mineral density testing and/or osteoporosis therapy following their fracture.
Own the Bone was a natural progression of rudimentary efforts that came about during the Bone and Joint Decade, and it marks a strategic effort on the part of the American Orthopedic Association to identify and treat the osteoporosis underlying fragility fractures. Multiple studies have demonstrated that only 1 out of every 4 to 5 patients who present with a fragility fracture will receive a clinical diagnosis of osteoporosis and/or active treatment to prevent secondary fractures related to osteoporosis. Ample Level-1 evidence demonstrates that the initiation of first-line agents like bisphosphonates, or second-line agents when indicated, can reduce the chance of a subsequent fragility fracture by at least 50%. We know these medicines work.
We also know that osteoporosis is a progressive phenomenon. Therefore, failing to respond to the osteoporosis underlying fragility fractures means we as a medical system fail to treat the root cause in these patients. The fracture is a symptom of an underlying disease that needs to be addressed or it will continue to produce recurrent fractures and progressive decline in overall health.
The members of the Own the Bone initiative must be commended for their admirable work. We as an orthopedic community need to attempt to incorporate lessons learned through the Own the Bone experience into our practice to ensure that we provide complete care to those with a fragility fracture. The report from Bunta et al. represents a large—but single—step forward on the journey toward universal recognition and treatment of the diminished bone quality underlying fragility fractures. I look forward to additional reports from this group detailing their continued success in raising the bar of understanding and intervention.
Brett A. Freedman, MD is an orthopaedic surgeon specializing in spine trauma and degenerative spinal diseases at the Mayo Clinic in Rochester, MN.
Dispatch from South Africa: COMOC 2016
I recently returned from the 13th meeting of the Combined Orthopaedic Associations, affectionately known as COMOC 2016. This meeting is unique in that it brings together seven different national orthopaedic organizations from six countries (America, Australia, Britain, Canada, New Zealand, and South Africa).
The concept for this combined meeting originated with R.I. Harris, a Canadian orthopaedic surgeon who had been the president of both the Canadian Orthopaedic Association and the American Orthopaedic Association (AOA). Dr. Harris felt that improved communication between American, British, and Canadian orthopaedic surgeons would be of benefit to all. He was also responsible for the institution of the American-British-Canadian (ABC) Traveling Fellowship.
The first combined meeting involved only US, Canadian, and British orthopaedic surgeons. At that time travel would have been by ship or train. The original idea was to hold this meeting every six years and to move the venue from country to country on a predetermined schedule. This year, COMOC was held in Cape Town, South Africa, and in six years the US will be the host country.
The structure of the meeting is unique in that countries are given a forum to present orthopaedic issues most relevant to their national organizations. On Monday, April 11, both the American Academy of Orthopaedic Surgeons and the AOA presented plenary sessions. On Tuesday Australia took its turn in the morning, and New Zealand presented in the afternoon. Wednesday saw a presentation from the United Kingdom, with Canada taking the podium on Thursday. The plenaries wrapped up on Friday with the host South African Orthopaedic Association.
This meeting is an enduring link with the past and the future, continuing the orthopaedic tradition of fellowship and friendship that is the hallmark of our specialty. The Cape Town meeting was exceptional in venue, content, and organization. The Local Organizing Committee and Programme Committee are to be congratulated for an exceptional job in developing a program that maintained significant audience interest despite the competing attractions of Cape Town and the South African countryside.
When COMOC comes to America in 2022, I hope North American orthopaedists—especially younger ones—will take the once-in-a-career opportunity to meet and talk with musculoskeletal colleagues from all over the world.
James P. Waddell, MD, FRCSC
JBJS Deputy Editor
Vitamin D Controversy Is Focus of Free Webinar
On Thursday, December 10, 2015, from 6:00 to 6:30pm EDT, the Own the Bone initiative will offer a free webinar titled “Vitamin D in Chaos: A Common Sense Approach for Orthopaedics.”
Neil C. Binkley, MD, from the University of Wisconsin will review the physiology of vitamin D, current approaches to 25(OH)D testing, and recommendations for treatment of those whose levels are low. Defining “low” vitamin D status remains extremely controversial, but many fracture patients have vitamin D inadequacy that may contribute to low bone mass and fragility fracture risk.
The American Orthopaedic Association (AOA) developed Own the Bone as a quality improvement program to address the osteoporosis treatment gap and prevent subsequent fragility fractures.
JBJS Reviews Editor Echoes Call for Improved Osteoporosis Prevention and Treatment
In last month’s Editor’s Choice, JBJS Editor in Chief Vern Tolo. MD, called for more concerted efforts among orthopaedists to link care of fragility fractures to evaluation and treatment of osteoporosis. Now, JBJS Reviews Editor in Chief Thomas Einhorn, MD, echoes Dr. Tolo’s message in reference to the May 2 JBJS Reviews article on managing patients with osteoporotic distal radial fractures:
According to Dr. Einhorn, “This must-read article provides a concise summary of how to advance the diagnosis and treatment of osteoporosis and fragility fractures. The authors explain the latest evidence about the ‘three main pillars’ of treatment of distal radial fractures in people with osteoporosis: primary prevention, acute management, and reduction of risk of future fractures. The strides made among US orthopaedists to recognize and manage osteoporosis with programs such as the American Orthopaedic Association’s ‘Own the Bone’ initiative have been commendable. However, on a global scale, our specialty is woefully behind in taking an aggressive approach toward prevention and treatment of osteoporosis.”
Editor’s Choice – May 5, 2014
The article “Declining Rates of Osteoporosis Management Following Fragility Fractures in the U.S., 2000 through 2009” by Balasubramanian, et al. in the April 2, 2014 JBJS is a bit discouraging, but it will hopefully serve as a wake-up call for orthopaedic surgeons to re-engage with our patients to diagnose and treat previously undetected osteoporosis.
Fragility fractures–which primarily affect the vertebrae, hip, distal radius, or proximal humerus–are often the initial indication of osteoporosis in older individuals. For more than a decade, orthopaedic surgeons treating these fractures have been strongly encouraged to evaluate patients in this age group for the osteoporosis generally associated with these fractures. The American Orthopaedic Association (AOA) in 2005 began developing the Own the Bone program, specifically addressing the need to evaluate and treat osteoporosis, as well as the fracture, in these patients. The AOA has formed liaisons with several other national organizations to advance this program, and by late 2013, 44 states had hospitals implementing Own the Bone at their local institutions.
This article is sobering. Despite concerted efforts to link care of fragility fractures to evaluation and treatment of co-existing osteoporosis, these authors report an actual decrease in the rate of osteoporosis management for these patients. Only one-third of the women and one-sixth of the men in this retrospective cohort study were evaluated and treated according to current clinical guidelines.
This is an important public health issue. Despite the fact osteoporosis management involves non-operative treatment, it is essential that orthopaedic surgeons become more cognizant of the association between fragility fractures and osteoporosis treatment, and put in place a protocol to ensure that these patients are evaluated and treated for osteoporosis, as well as for the fracture. Osteoporosis may not be under the direct guidance of the orthopaedic surgeon, but the recognition of this potential problem is squarely within the practice scope of orthopaedists, who are well positioned to initiate secondary prevention measures for these older individuals.
Try Clinical Classroom
Recent Comments