Guest Post—Peer Reviewers: Who Are They and Do They Agree?
OrthoBuzz occasionally receives posts from guest bloggers. This guest post comes from Christopher Dy, MD, MPH, in response to a recent study in PNAS.
I am a young surgeon, but I have been submitting papers and grants for peer review for 11 years, since I was a third-year med student. I have tasted the bitterness of rejection more times than I would like to admit, several times at the hands of JBJS. But I will say, without a doubt, that the peer-review process has made my work better.
Acknowledging that our work is far from perfect at the point of submission, most of us have turned the question around: How good and reliable is the peer-review process? Several related questions arise quickly: Who are the “peers” doing the reviewing? We put weeks and months into writing a paper or submitting a grant, which then vanishes into the ether of a review process. How do we know that we are getting a “fair shake” from reviewers, who, being human, carry their own biases and have their own limitations and knowledge gaps—in addition to their expertise? And do the reviewers even agree with each other?
Many authors can answer “no” to that last question, as they have likely encountered harmony from Reviewers 1 & 3 but scathing dissent from Reviewer 2. Agreement among reviews was the question examined by Pier et al. in their recent PNAS study. Replicating what many of us consider the “highest stakes” process in scientific research, NIH peer review, the authors convened four mock study sections, each with 8 to12 expert reviewers. These groups conducted reviews for 25 R01 grant proposals in oncology that had already received National Cancer Institute funding. The R01 is the most coveted of all NIH grants; only a handful of orthopaedic surgeons have active R01 grants.
Pier et al. then evaluated the critiques provided by the reviewers assigned to each proposal, finding no agreement among reviewer assessments of the overall rating, strengths, and weaknesses of each application. The authors also analyzed how well these mock reviews paired to the original NIH reviews. The mock reviewers (all of whom are R01-funded oncology researchers) “rated unfunded applications just as positively as funded applications.” In their abstract, Pier et al. conclude that “it appeared that the outcome of the [mock] grant review depended more on the reviewer to whom the grant was assigned than the research proposal in the grant.”
From my perspective as a taxpayer, this is head-scratching. But I will leave it to the lay media to explore that point of view, as the New York Times did recently. As a young clinician-scientist, these results are a bit intimidating. But these findings also provide empirical data corroborating what I have heard at every grant-funding workshop I’ve attended—your job as a grant applicant is to communicate clearly and concisely so that intelligent people can understand the impact and validity of your proposed work, regardless of their exact area of expertise. With each rejection I get, either from a journal or a funding agency, I now think about how I could have communicated my message more crisply.
Sure, luck is part of the process. Who you get as a reviewer clearly has some influence on your success. But to paraphrase an axiom I’ve heard many times: The harder I work, the more luck I seem to have.
Christopher Dy, MD, MPH is a hand and peripheral nerve surgeon, an assistant professor at Washington University Orthopaedics, and a member of the JBJS Social Media Advisory Board.
Thank You, JBJS Reviewers
During the 2017 AAOS Annual Meeting in San Diego next week, JBJS will formally recognize its 30 Elite Reviewers.
In the days leading up to the Annual Meeting, we would like to profile and thank some of our additional outstanding reviewers. Today, let’s meet Peter Stern and Kevin Garvin:
Peter Stern, MD
University of Cincinnati
What do you like best about reviewing for JBJS?
Expands my knowledge base and permits me to participate in a activity which I believe is paramount to broadening the orthopedic peer reviewed knowledge base.
How do you find time to review for JBJS?
I make the time on the weekend. Get up early.
What do you see as JBJS’ role in shaping the future of orthopaedics?
It’s the #1 peer-reviewed journal and provides accurate msk information in
many domains such as JBJS, JBJS Reviews, the virtual recertification course, Case Connector, etc.
Kevin L. Garvin, MD
Nebraska Medical Center
What do you like best about reviewing for JBJS?
The opportunity to review cutting edge research and hopefully
help improve the clarity of the research.
How do you find time to review for JBJS?
I just make time for doing the reviews.
What do you see as JBJS’ role in shaping the future of orthopaedics?
One of the leading media for communicating new research.
Thank You, JBJS Reviewers
During the 2017 AAOS Annual Meeting in San Diego next week, JBJS will formally recognize its 30 Elite Reviewers.
In the days leading up to the Annual Meeting, we would like to profile and thank some of our additional outstanding reviewers. Today, let’s meet Wudbhav Sankar and Leon Benson:
Wudbhav Sankar, MD
Children’s Hospital of
Philadelphia
What do you like best about reviewing for JBJS?
Contributing to our-peer reviewed process for improving
understanding and treatment of msk disorders.
How do you find time to review for JBJS?
Nights and weekends! It’s a sacrifice like any other.
What do you see as JBJS’ role in shaping the future of orthopaedics?
Pivotal as the highest impact factor and most widely read ortho
journal. It’s the standard.
Leon Benson, MD
University of Chicago
What do you like best about reviewing for JBJS?
The intellectual challenge.
How do you find time to review for JBJS?
I take my laptop with me everywhere.
What do you see as JBJS’ role in shaping the future of orthopaedics?
JBJS will continue to be both the leading edge as well as the
foundation of contemporary orthopaedic knowledge for our
patients.
Thank You, JBJS Reviewers
During the 2017 AAOS Annual Meeting in San Diego next week, JBJS will formally recognize its 30 Elite Reviewers.
In the days leading up to the Annual Meeting, we would like to profile and thank some of our additional outstanding reviewers. Today, let’s meet John L. Eady and William M. Ricci:
John L Eady, MD
Dorn VA Hospital,
Columbia, SC
What do you like best about reviewing for JBJS?
The trust placed in my reviewing abilities by the editors of JBJS as
well as the authors who submit their works for consideration for
publication.
How do you find time to review for JBJS?
One doesn’t “find” the time to do the reviews. The time has to be set
aside and dedicated to doing the review as soon as an invitation to
review is received.
What do you see as JBJS’ role in shaping the future of orthopaedics?
It will become even more labor-intensive to achieve the core goals of
selecting well-written, evidence-based articles for publication as the
age of instant, unsubstantiated messaging proliferates. The
leadership of JBJS will need skills unheard of presently to create
new methods which insure the continued selection of core
knowledge that shapes the direction of value-based orthopaedics.
William M. Ricci, MD
Washington University,
St. Louis
What do you like best about reviewing for JBJS?
Serving the orthopaedic community to ensure good research is
published in its best form.
How do you find time to review for JBJS?
Just like anything else, if something is a priority, you make time
for it.
What do you see as JBJS’ role in shaping the future of orthopaedics?
JBJS provides the orthopaedic community an opportunity to see
what peers believe is important in the field.
Thank You, JBJS Reviewers
During the 2017 AAOS Annual Meeting in San Diego next week, JBJS will formally recognize its 30 Elite Reviewers.
In the days leading up to the Annual Meeting, we would like to profile and thank some of our additional outstanding reviewers. Today, let’s meet Winston J. Warme and A. Seth Greenwald:
Winston J. Warme, MD
University of Washington
What do you like best about reviewing for JBJS?
I enjoy being a part of the peer-review process at the flagship
orthopaedic journal.
How do you find time to review for JBJS?
It is a priority, so I make time to do it.
What do you see as JBJS’ role in shaping the future of orthopaedics?
With forward thinking Editors-in-Chief like Dr. Swiontkowski, and top
shelf contributors, JBJS will continue to be The Journal to read, with
cutting-edge articles and thoughtful editorials.
A. Seth Greenwald,
D. Phil. (Oxon)
Orthopaedic Research
Laboratories
What do you like best about reviewing for JBJS?
The opportunity to use a lifetime of one’s orthopaedic learning to address contemporary clinical submissions.
How do you find time to review for JBJS?
I think it important enough to allocate both personal and professional
time to stay abreast of the state-of-the-art as well as contribute my
knowledge and background in its evaluation.
What do you see as JBJS’ role in shaping the future of orthopaedics?
I believe the journal’s role in shaping the future of orthopaedics is the
reporting of contemporary clinical outcomes associated with
musculoskeletal problems, as well as maintaining its presence as a
focus of education for surgical practitioners. The journal should strive
to establish its presence as the reporter of evolving practices and the
financial structures, both in the private and public sectors, that
facilitate healthcare for the American people.
Thank You, JBJS Reviewers
During the 2017 AAOS Annual Meeting in San Diego next week, JBJS will formally recognize its 30 Elite Reviewers.
In the days leading up to the Annual Meeting, we would like to profile and thank some of our additional outstanding reviewers. Today, let’s meet Kodali Silva R.K. Prasad and Alastair Younger:
Kodali Silva R. K. Prasad, MD
Price Charles Hospital
What do you like best about reviewing for JBJS?
JBJS has evolved a uniquely rigorous dual peer review system with graded
content review and methodological review coupled with editorial supervision, a formidable combination to pre-empt system failure and ensure sustained excellence. While conceding that a perfect study is probably a virtually impossible entity, I believe that this stringent dual review system with editorial control constitutes an optimal approach toward perfection. JBJS’ stable of journals offer a comprehensive range of publications, unlikely to be replicated in current orthopaedic literature. Consistently high impact factor of JBJS, as yet unrivalled among general orthopaedic journals, is a testament to continuous publication of significant scientific research and original contributions to orthopaedic literature.
How do you find time to review for JBJS?
I view the review process as a medium for wider contribution to the
progress of trauma and orthopaedic surgery and momentum of creative
impetus for future achievement. I consider it a duty for present and future
generations and make time outside my clinical commitments.
What do you see as JBJS’ role in shaping the future of orthopaedics?
JBJS plays a vital role to inspire definition of the current practice for
excellence and drive the future direction of the clinical landscape in trauma and orthopaedic surgery.
Alastair Younger, MD
University of British Columbia
What do you like best about reviewing for JBJS?
Reviewing journal articles makes me read the article in depth to
understand the thoughts of the authors. It compels me to remain
current in publications. It is a way of giving back to the orthopaedic
community by assisting other authors in their publications. It helps
me and my co-authors write better papers. Reviewing keeps me
curious and asking questions about how can we help patients better.
How do you find time to review for JBJS?
The reviewing process takes less time with experience. The critical
points or lack of them become apparent. However, like most of my
academic activities the time is found between cases, or at nights and
weekends. Like all other academic activities, it is the desire to do it
that makes you find the time.
What do you see as JBJS’ role in shaping the future of orthopaedics?
JBJS remains the premier journal for peer-reviewed orthopaedic
publications. JBJS and orthopaedics will be challenged as our field
becomes more sub-specialized and as we try to keep up with the ever
increasing knowledge in the field. JBJS and the physicians involved will play a major role in leading the future.
JBJS to Launch Elite Reviewers Program
Peer review is the basic underpinning of scientific publication, and The Journal’s reviewers are key components in our 127-year history of publishing the highest quality of evidence-based information. JBJS reviewers volunteer their time and expertise to serve the orthopaedic community and enhance the quality of care for patients.
To formally recognize the outstanding contributions of our very best reviewers, beginning in January 2016, JBJS will implement an Elite Reviewer Program. Elite Reviewers will be recognized based on measurements of their response time, quality of reviews, and reliability.
We expect that authors will benefit from the program with even higher-quality and more prompt peer review, and we hope the program inspires all of our reviewers to meet Elite Reviewer standards.
For more information about the program, read the JBJS editorial, or visit the Elite Reviewers Program page on our website.
Publisher’s Note: Launch of PRE-val supports “Excellence Through Peer Review”
A year ago we debuted the “peer-review statement” in The Journal to emphasize our commitment to pre-publication peer review and to the rigorous, double-blind peer-review process that is integral to our editorial standards.
Today we are happy to announce our participation in PRE-val, the flagship service offered by PRE (Peer Review Evaluation). Our readers will notice the PRE-val badge above the article title for most JBJS articles published on our website in the past 12 months. Clicking on the badge reveals the PRE-val window, which provides detail about the peer review for that particular article. We know that your confidence in the reliability of the information published in The Journal will be increased by the enhanced transparency of our peer-review process.
As a result of the commitment to peer review shared by JBJS and PRE, our Board of Trustees approved the acquisition of PRE in 2014. We are excited about this launch, and we look forward to the implementation of this valuable service on the sites of our partner publishers over the coming months. You can learn more about PRE here. Of course, we welcome your feedback; please let us know what you think of this initiative by writing to us at info@jbjs.org.
Medical publishing continues to evolve-sometimes to keep up with technology, sometimes due to financial constraints, and, unfortunately, sometimes in ways that make some of us uncomfortable-but readers of JBJS can be assured that our commitment to peer review and the quality it helps us to achieve will not waver. “Excellence Through Peer Review” will always remain a critical element of our core mission.
–Mady Tissenbaum, Publisher, JBJS
JBJS Print Readership Steady, Online Access Up
According to a recent JBJS readership study among more than 1,000 orthopaedic surgeons, print readership of JBJS remained fairly consistent from 2012 to 2014; 91% accessed the JBJS print edition frequently/occasionally in 2012, and 87% did so in 2014. The percentage of respondents accessing JBJS online access jumped significantly from 58% in 2012 to 74% in 2014. Roughly 30% cited reading all/most of their issue in both the 2012 and 2014 studies.
Additionally, the survey revealed a slight increase in relevancy of JBJS to clinical practice/research compared to 5 years ago. In 2012, 33% stated JBJS was more relevant compared to the past 5 years; in 2014 36% rated JBJS more clinically relevant that it was 5 years prior.
JBJS Deputy Editor Gives Publishing Tips to Chinese Researchers
The number of manuscripts submitted to The Journal of Bone & Joint Surgery (JBJS) from physicians and researchers in China has been steadily increasing, but the overall acceptance rate is relatively low. While the quality of the research performed in China is rapidly improving, orthopaedic researchers in China recognize the need for education related to experimental design, manuscript preparation and manuscript review. In June 2014, Dr. Thomas Bauer, JBJS deputy editor for research, participated in a two-day workshop in Shanghai that focused on helping Chinese researchers prepare and submit high-quality journal manuscripts.
During an afternoon workshop, Dr. Bauer and three other experienced Chinese editors/reviewers provided “face-to-face” reviewing with individual researchers who had provided and presented draft manuscripts. Dr. Bauer’s subsequent lectures included recommendations with respect to selecting the most appropriate journal for a specific paper, the contents of each section of a scientific manuscript, tables and figures, and how to respond to a manuscript review. He also described the general review process at JBJS and discussed “misbehaving authors,” including issues related to attempted duplicate publication, fraud, image manipulation, and plagiarism.
Based on review of 50 manuscripts from China that had been rejected, Dr. Bauer tabulated the reviewers’ comments to identify the most frequent reasons for manuscript rejection. The most frequent criticism reflected insufficient information about the number of patients or specimens and the lack of an explanation for sample size. Dr. Bauer also illustrated several recent manuscripts from China that have been published in JBJS and in JBJS Case Connector. Several other speakers also discussed issues related to experimental design and statistics.
The lively discussion from the audience of more than 100 researchers reflected intense desire to publish the best possible work in JBJS. We anticipate a striking increase in the number and quality of manuscripts from China in the near future.
Recent Comments