Mentorship Series: How to Peer-Review a Manuscript

Knowledge translation and dissemination is a critical and typically ultimate step in the life cycle of a research project before the work ideally goes on to influence paradigms, policies, and practices. Such knowledge is generally communicated in both written and oral formats through dissemination tools including abstract presentation at scientific conferences; lay-language print, news, and social media; webinars, podcasts, and other continuing education events; and, most commonly, publication in a scientific journal. Fundamental to the production of such academic deliverables – or, as some would say, “academic currency” – is objective peer-review, wherein members of the larger community within one’s sphere of competence and expertise evaluate the suitability of the knowledge product for dissemination. As the Editor-in-Chief of the BMC journal Tropical Diseases, Travel Medicine and Vaccines (www.tdtmvjournal.com), and Associate Editor/Editorial Board member of several other scientific journals, I am frequently asked by trainees across academic levels for guidance on how to peer review a manuscript. Thus, I have developed the approach below based on my own personal experience as an author, ad-hoc peer-reviewer, editorial board member, and editor. The approach reflects my own opinions, which should be construed as neither endorsed nor shared by my employer or professional affiliations. I hope that trainees find this guidance helpful!

The first step after agreeing to serve as a peer-reviewer, is to organize questions and tasks into discrete work packages. For a manuscript reporting the findings of an original research project – the most common article type that you are likely to be invited to review – your work packages will usually constitute a critique of the paper’s: writing, presentation, organization, and flow; originality, scientific accuracy, and potential impact; and suitability for the particular audience that you are representing as a peer-reviewer. Elements of the paper to consider through that critical lens include: study design, hypothesis, and objectives; data collection and analysis; and, synthesis of findings and interpretation. The work packages and elements can then be conceived as a matrix, into which your article-specific questions will fit. A sample matrix with near-universal questions to be answered through peer-review of an original research article is provided below:

Elements		Work Packages - Critiques of:
	Writing, presentation, organization, and flow	Originality, scientific accuracy, and potential impact	Suitability for journal's audience
Study design, hypothesis, objectives (key sections include Title, Abstract, Introduction)	Does the title make sense? If the authors have a hypothesis, is it clearly stated? Are the objectives communicated in the Abstract and Introduction? Does the Introduction avoid an exhaustive topical review and focus concisely on the background that is most relevant to the authors' question? Does the manuscript require editing for language throughout?	Is there a clear justification for why the work was needed and is timely? Have the study objectives been situated in the larger context of the field? Is the study design appropriate to address the objectives? Are sex/gender considerations addressed in the study design? If not, is justification provided? Is the work novel? Does the work aim to logically extend what is already known in the field? Is the Introduction appropriately referenced?	Is the paper within scope of the journal? Will the general topic be of interest and relevance to the audience? Have papers using the same or similar study design been published in the journal previously? If not, would publication of a new study design for the journal be justified?
Data collection and analysis (key sections include Methods and Results)	Are the methods reported in sufficient detail so as to enable replication? Are the sources, handling, storage, and protection of data and materials identified? Are human and animal subjects considerations addressed? Are the Methods organized into a logical sequence of steps? Are statistical analyses clearly described in the Methods? Does the reporting of results logically follow the flow of the methods? Does the Results section erroneously include text better suited to Methods? Are the results appropriately communicated in either text, tabular, or figure format? Are results presented in the Abstract consistent with those in the Results text, Tables, and Figures? Are new tables, figures, or text required to improve readability and comprehension of the paper? Are supplemental Methods, Tables, or Figures required?	Are the methods appropriate to the study design? Were experiments or tests conducted in a transparent, replicable, and precise manner? Were experiments or tests conducted in accordance to international standards? Were reporting guidelines followed and presented (e.g., STROBE, STARD, CONSORT)? Were standard analytic guidelines and tools utilized (e.g., QUADAS, GRADE)? Are methods appropriately referenced? Are results reported fully? If not, is this absence explicitly justified by the authorship? Where feasible, are results reported according to intersectional demographics? If not, is justification provided? Are there errors in Tables and Figures? Is the scope or scale of the work likely to translate into meaningful impact?	Will the methodological approach be understood by the readership? Are the analytic tools and reporting frameworks utilized likely to be understood by and relevant to the readership?
Synthesis of findings and interpretation (key sections include Discussion / Conclusions and References)	Do you understand what the authors have concluded? Do you understand how the authors have interpreted their data? Are all results discussed first presented in the results section (either tables, figures, or text)? Are specific numeric results extensively recapitulated (including p-values!) and erroneously included in the Discussion section? Are all references listed in the reference section also cited in the text? Similarly, are all references cited in the text listed in the reference section?	Do the conclusions extend from the results? Has the interpretation of findings been over-generalized or over-extended? Are interpretations stated that could not possibly be concluded from the study design? Are the findings appropriately situated within the context of the broader and specific literature? Are all key references included? Are the limitations of the work identified (including external validity of the findings to the intersectional demographic of the representative population)? Were models or populations used experimentally that were of such homogeneity as to prevent generalizability to the larger population? If so, is this limitation clearly identified? In reviewing the contributions, acknowledgements, and disclosures sections, does it appear that interpretation of results may have been influenced by external factors? Will the work inform or change current or future approaches, practices, policies, or paradigms? Do the authors identify next steps?	Are the findings of sufficiently high priority and impact to justify publication in that journal at that time? If the journal is a niche area (special interest), is the interpretation of sufficient scientific depth to truly advance the literature in that particular niche? If the journal is of general scientific or medical interest, have the results been appropriately contextualized and situated within the general interest literature (breadth focus rather than depth focus)?

Having this kind of an approach to peer-review will enable you to complete your assignments in an organized, efficient manner, and arm you with a fighting chance of meeting the laughable deadlines imposed at the editorial level. Sticking to the approach will also prevent you from getting sucked into the trap of policing formatting, correcting minor typographical errors, and identifying grammatical calamities. This is what copy-editors do. Your job is to identify areas for improvement to structure, form, content, and interpretation in as objective a manner as possible. Threats to one’s objectivity include current or past collaborative or adversarial relationships with the authorship; shared workplace with one or more listed authors; financial conflicts of interest; time pressure; and, quite importantly, implicit and explicit biases. Consideration and recognition of factors that might imperil your objectivity are incumbent upon you as a scientist with integrity. Mitigation strategies to be executed in advance of agreeing to the peer-review assignment include: detailed review of the authorship (if unblinded); allocation of sufficient time to complete the review by blocking one’s calendar; quick review of the article’s abstract (often provided) for red flags around possible financial or other conflicts of interest; and completion of implicit bias testing and/or training. One such implicit bias tool is the Harvard Implicit Association Test, which can be found at: https://implicit.harvard.edu/implicit. Peer-reviewers should also be mindful of the fact that authors from low- and middle-income countries are at a natural disadvantage when it comes to publishing in the English language literature. This is why much scientific knowledge housed in equatorial parts of the world lives in anecdotal, oral historical, and unpublished form. If the manuscript is of potential high quality but requires editing for English, so state that in your review; it is not your job to rewrite the paper, but inherent disadvantage due to language barrier should not preclude a solid piece of science from appearing in the peer-reviewed literature.

Additional resources that will assist you as you develop your peer-reviewing skills are listed below. Participation in the peer-review process should be viewed as an opportunity to return a valuable service to your scientific community and to improve your own writing, data representation, and communication skills. Being invited to peer-review the work of colleagues is both an honour and a privilege, and is foundational to the social contract of life as an academic. You will be a better scientist for it!

Additional Resources:
Peer Review Process: https://www.biomedcentral.com/getpublished/peer-review-process
International Committee of Medical Journal Editors (ICMJE) “Recommendations for the Conduct, Reporting, Editing and Publication of Scholarly Work in Medical Journals”:
http://www.icmje.org/recommendations

Mentorship Series: How to Peer-Review a Manuscript

1 thought on “Mentorship Series: How to Peer-Review a Manuscript”

Leave a Reply to Adele Rutter Cancel reply