|
|
J Gen Intern Med. 2006 March; 21(3): 281–284. doi: 10.1111/j.1525-1497.2006.00354.x. | PMCID: PMC1828092 |
Copyright © 2006 by the Society of General Internal Medicine. All rights reserved How to Be an Outstanding Reviewer for the Journal of General Internal Medicine … and Other Journals Carlos Estrada, MD, MS,1 Adina Kalet, MD, MPH,2 Wally Smith, MD,3 and Marshall H Chin, MD, MPH4 1University of Alabama at Birmingham and the Birmingham Veterans Affairs Medical Center, Birmingham, AL, USA 2Division of General Internal Medicine, New York University School of Medicine, New York, NY, USA 3Division of Quality Health Care Virginia Commonwealth University, Richmond, VA, USA 4University of Chicago, Chicago, IL, USA Peer reviewers are essential to maintaining the excellence and relevance of the Journal of General Internal Medicine ( JGIM). We salute the many volunteers who consistently provide high-quality peer reviews of manuscripts submitted for publication. As advisors to the authors and Editors, your work allows our journal to maintain high standards and provide a well-respected venue for publication of manuscripts essential to the work of the members of the Society of General Internal Medicine. In this editorial, we hope to increase the number of you who feel empowered to participate as peer reviewers. We briefly summarize what is known about peer reviews and reviewers, make transparent the peer review process at the Journal, and highlight the benefits of participating as a reviewer. In this spirit, we provide specific suggestions about how to become an effective reviewer, comments that we hope will assist current and prospective reviewers in preparing their responses. Additional resources are available elsewhere. 1WHAT DO WE KNOW ABOUT REVIEWS AND REVIEWERS? While the peer review process is widely used and provides face validity to the publication process, evidence supporting using peer review to improve the quality of manuscripts is limited. We know that reviews appear to improve communication of the precision of results, improve discussion sections, and temper sometimes overreaching conclusions. 2 Such improvements are most striking for manuscripts judged as low quality initially. 2 Not surprisingly, technical editing improves manuscripts, 3 abstract quality, readability, and accuracy of references. 4 Despite these advantages, the quality of peer reviews varies widely, 5 and efforts to improve peer review by blinding the review process 6–8 or training the reviewers 9,10 by brief workshops or giving feedback does not seem to provide a measurable sustained benefit. 11Better quality reviews have been associated with a number of difficult to modify rater characteristics like being younger, of lower academic rank, from strong academic institutions, having previous research training, additional postgraduate degrees, being known to the editors, and blinded to the identity of the manuscript's authors. 5,12 But so much about the peer review process is still to be elucidated. For example, what is the impact of participating in the peer review process to the reviewer? What is the best way to provide feedback to reviewers? What is the best training for reviewers?. WHAT IS THE REVIEW PROCESS AT JGIM? Upon submission of a manuscript, 1 of the 2 Editors forwards manuscripts to a Deputy Editor (see ). Manuscripts other than Editorials are rarely accepted without peer review. Usually, they are sent for peer review, or rejected without external review. In the latter case, the manuscript is reviewed by one of the Co-Editors or Deputy Editors who provides feedback to the author to help improve the article before it is sent to another journal. Deputy Editors select reviewers using a database of reviewers maintained by the Journal, or based on their knowledge of experts in the field. The reviewer database includes areas of expertise or interest, numbers of reviews made by each reviewer, the date of each reviewer's last review, and the timeliness of prior reviews. If a reviewer has performed a review, their name is blocked from selection for 3 months to assure that we do not overly burden anyone. We attempt to engage 3 reviewers for each manuscript. Importantly, Deputy Editors are asked to rate the quality of each review they receive using a 6-point Likert scale (1=poor, 6=excellent). The quality of past reviews assists Deputy Editors with the decision to use a given reviewer. It is also used along with the number of reviews and timeliness of returning reviews in JGIM's annual recognition of outstanding reviewers. Reviewers who agree to review the manuscript within a defined timeline receive a blinded copy of the manuscript. Once the reviews are returned to the Journal, the Deputy Editor evaluates the manuscript and makes a decision to reject, accept, or ask the author to revise the manuscript. For many other journals, this decision is made either by the Editor or together by the Editor and the Deputy/Associate Editors. But for JGIM, this decision is delegated to the Deputy Editors who can (and often do) seek input from the Co-Editors or other Deputy Editors. The Deputy Editor highlights important areas of the reviews that should be addressed for manuscripts invited for resubmission. Authors receive a blinded copy of the reviews. At the end of the process, reviewers receive a copy of the decision letter and other reviewers' comments. We acknowledge reviewers by publishing their names yearly, and in 2005, the best reviewers received special recognition in an Editorial. 13WHY SHOULD I BECOME A REVIEWER? Based on our own experiences, we believe that participating as a peer reviewer has many personal benefits. It motivates one to write, is fun and intellectually satisfying, provides an opportunity to be creative, and demystifies the academic publishing process. Also, being a peer reviewer is a discrete and manageable way to contribute to the generation of higher quality scholarship that shapes our profession. Expert review adds to the integrity of the decision to publish submitted manuscripts and therefore is an explicit way to participate in the self-regulation of our profession. A peer reviewer for the Journal has the opportunity to shape, clarify, and highlight important aspects of the communication of discovery in General Internal Medicine and to benefit personally from this process. First, reviewers learn about their field as they review. Second, reviewers effectively mentor authors in their field by giving them constructive feedback. Third, by shaping how manuscripts finally appear to readers, reviewers contribute to the quality of research, education, practice, and policy making. Participating in peer review is evidence of scholarly activity and qualitatively indicates regional or national prominence. The perceived value of peer review activities makes them a criterion for promotion and tenure in most academic centers and of advancement in other organizations. Many reviewers have included letters from the JGIM editors among their external reviews for promotion. Finally, reviewing is also a very efficient tutorial in writing style (“What are they trying to say?”), research methods (“I never thought of that—what a great idea”), and content area (“Hey I could be doing this kind of work also”)—thus sharpening your skills as clinician, researcher, or administrator. WHAT MAKES A GOOD REVIEW? A good review can enhance the quality of the manuscript on many levels. Below are tips for constructing a good review for JGIM. In general, a good review discusses the importance of a research question and its originality, identifies the strengths and weaknesses of the methods, provides specific and constructive suggestions, and comments on the interpretation of results. 14 Of course, writing a good review does not guarantee that either the manuscript's author or editor will agree with your review. But a good review should clarify any points of disagreement. The The Appendix (available online) contains annotated samples of reviewers' comments that illustrate some of these tips. Some reviews are deemed good while others are in need of improvement. For more tips, refer to sources that summarize the responsibilities of reviewers, 15 provide additional suggestions 1,16 and guidelines, 17,18 and provide a validated instrument to measure the quality of a review. 10,14- Cover major areas and be concise: Make your review 1 to 3 pages in length, and answer the questions outlined in Table 1.
- Prioritize and organize your concerns: Prioritizing helps both the author and the editor to know which major issues must be addressed, and which are minor and not as critical to the publication or revision of a paper. Provide brief statements, number and organize the suggestions, and be clear and direct.
- Be specific and give examples from the manuscript: General statements such as “not a significant contribution,”“clinically unimportant,” and “poorly analyzed” are not very useful without more details. Vague reviews frustrate authors because they are forced to guess what parts of their manuscript need to be improved.
- Suggest corrective actions if possible: Specific suggestions for analysis, writing, formatting, or summarizing the information to enhance the manuscript are helpful. Point to fundamental flaws that are unlikely to be overcome.
- Be respectful of the authors: Scholarship is difficult and criticism is not pleasant to receive. Be fair and critical but make your comments in the tone that you would want to receive.
- Act as the expert: Especially for new reviewers, do not be shy. You were chosen because you have some expertise, either as a reader of the literature or a contributor to the field. Do not hold back. One reason for maintaining the anonymity of reviewers is that we believe it encourages frank feedback.
- Use guidelines or checklists as appropriate: Guidelines are available to assist you in your review and in giving feedback to authors. Published guidelines exist for educational interventions,19 meta-analysis of randomized,20 or observational studies,21 economic evaluations,22,23 diagnostic tests studies,24 randomized-controlled trials,25 and nonrandomized interventions.26,27
- Follow the instructions: Once you have agreed to review, stay within the timeline. It is not fair to the authors to delay. Provide scoring requested on the reviewer's report form such as appropriateness to the audience, scientific rigor, and clarity of the writing. Also, indicate or suggest whether the manuscript is appropriate for an editorial. Maintain confidentiality and describe any conflicts of interest.
- Be frank when making confidential comments to the editor: This section allows you to make comments that the author does not see. Unlike the comments returned to the authors, these comments may include your opinions about whether the manuscript is ultimately worth publishing. Comments made to the author are not worth repeating in this section.
- Do not convey different messages to the author and editor: It is confusing and sometimes even hurtful to an author to have a reviewer “be kind” or even positive in his or her comments to the author and harsh (rejecting the manuscript) to the editor. It may create a communication problem that the editor will need to fix. Provide unbridled but accurate, constructive criticism to the author.
Like all peer-reviewed journals, JGIM depends on your participation. If you have not volunteered already, register online at the Journal website to be considered for the JGIM reviewer database. Although the literature on this subject is limited, peer review is the best system we have currently for improving manuscript and journal quality. Peer review needs everyone's participation to be valuable and viable. A well-done review is beneficial to the reviewer, Editor, and author. When preparing your review, we encourage you to keep the concepts outlined above in mind, as well as reviewing the annotated samples of good reviews available in the online version of this Editorial. We thank Drs. Lisa Willett, Terrance Shaneyfelt, and others for their helpful suggestions. Appendix I Example of three reviews completed on the same manuscript. Appendix II Example of two reviews completed on the same manuscript. 1. Laine C, Mulrow C. Peer review: integral to Science and indispensable to Annals. Ann Intern Med. 2003;139:1038–40. [PubMed] 2. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at annals of internal medicine. Ann Intern Med. 1994;121:11–21. [PubMed] 3. Wager E, Middleton P. Effects of technical editing in biomedical journals: a systematic review. JAMA. 2002;287:2821–4. [PubMed] 4. Roberts JC, Fletcher RH, Fletcher SW. Effects of peer review and editing on the readability of articles published in Annals of Internal Medicine. JAMA. 1994;272:119–21. [PubMed] 5. Evans AT, McNutt RA, Fletcher SW, Fletcher RH. The characteristics of peer reviewers who produce good-quality reviews. J Gen Intern Med. 1993;8:422–8. [PubMed] 6. Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. 1998;280:237–40. [PubMed] 7. van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA. 1998;280:234–7. [PubMed] 8. Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D. Does masking author identity improve peer review quality? A randomized controlled trial. PEER investigators. JAMA. 1998;280:240–2. [PubMed] 9. Callaham ML, Wears RL, Waeckerle JF. Effect of attendance at a training session on peer reviewer quality and performance. Ann Emerg Med. 1998;32:318–22. [PubMed] 10. Schroter S, Black N, Evans S, Carpenter J, Godlee F, Smith R. Effects of training on quality of peer review: randomised controlled trial. BMJ. 2004;328:673. Epub 2004 Mar 2. [PubMed] 11. Callaham ML, Knopp RK, Gallagher EJ. Effect of written feedback by editors on quality of reviews: two randomized trials. JAMA. 2002;287:2781–3. [PubMed] 12. Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA. 1998;280:231–3. [PubMed] 13. The Editors. Distinguished reviewers. J Gen Intern Med. 2005;20(1199) 14. van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol. 1999;52:625–9. [PubMed] 16. Bordage G, Caelleigh AS, Steinecke A, et al. Review criteria for research manuscripts. Acad Med. 2001;76:897–978. [PubMed] 19. Reed D, Price EG, Windish DM, et al. Challenges in systematic reviews of educational intervention studies. Ann Intern Med. 2005;142:1080–9. [PubMed] 20. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354:1896–900. [PubMed] 21. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis of observational studies in epidemiology (MOOSE) group. JAMA. 2000;283:2008–12. [PubMed] 22. Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. BMJ. 1996;313:275–83. [PubMed] 23. Pignone M, Saha S, Hoerger T, Lohr KN, Teutsch S, Mandelblatt J. Challenges in systematic reviews of economic analyses. Ann Intern Med. 2005;142:1073–9. [PubMed] 24. Rennie D. Improving reports of studies of diagnostic tests: the STARD initiative. JAMA. 2003;289:89–90. [PubMed] 25. Moher D, Schulz KF, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA. 2001;285:1987–91. [PubMed] 26. Des Jarlais DC, Lyles C, Crepaz N. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Am J Public Health. 2004;94:361–6. [PubMed]
|
