How to review a journal article

One of the things I've had to do recently is to write anonymous reviews for manuscripts that are submitted to journals. It was something that I never received any real training on, and I had to pick up from looking at the reviews of my own manuscripts I had received previously, chatting with colleagues, and simply thinking about the things I would want a reviewer to tell me.

Manuscript... REJECTED!
Ha ha ha! Huzzah for anonymity!

One of my friends who is still in grad school just received a request to review a manuscript for an article, and asked if I had any advice on how to do that. I figured this would be a great topic for a blog post. So without further ado, here is my (relatively new, fairly untested) advice about how to review a manuscript for a journal. Remember that this is what I find works for me as an anonymous review, and what I like to see in a review. Your mileage may vary.





1. Start by putting the manuscript into whatever format you typically read journal articles.
I find it much easier to critique an article for a manuscript when I approach it as if I was reading a regular article already published in a journal. When I read most of my regular journal articles, I download the PDF, print it out, and put it in my 3-ring binder of things I need to read. I can then take the binder to the couch, on the porch, on the bus, or wherever I decide to get some reading done, rather than being tethered to my computer.

Yes, I realize that my
reviewing method is indirectly
responsible for the death of
many trees.

This also makes it easier for me to read the manuscript and think about its contents, rather than being distracted by its odd format, as I would be if I tried reading it on the computer in the format originally sent to me by the editor (typically a MS Word or Adobe PDF). I can also make notes much easier on the printed out pages than I can using the MS Word or PDF electronic comments and markups, and it's easier for me to review my comments later on when I write the summary (more about this later in the post).


2. Relatedly, read the article as if it was already published.
Read the article as if it was coming out in the latest issue of the journal. Don't cut it any slack because it's in 'draft' form. As you read, make marks in the margins, on the text, or in a separate list about things you could add into your review.

For instance:

• What do you agree with? What did the author do well?
• What do you disagree with? Note that this is not whether you like or don't like the outcome of the results. Instead, do you disagree with things like:
• The way the literature review was setup
• The type of analytical model used
• The statistics reported or not reported in the tables
• The interpretation of results
• The link to policy/practice 
• Is the literature review fairly comprehensive and does it set up the analyses and discussion well? Or, instead, is it missing a key piece of the literature, key citations in the field, or does it read as a very lopsided argument, ignoring any prior literature that contradicts the theory/hypotheses being tested by the authors.
• Related to the literature review, does it misinterpret results from prior studies? Does it say that study X found Y, when it really found Z?
• Does the study have a 'consistent read,' or does it instead read like two (or more) people wrote different pieces independently and just stitched them together at the end?

Dear editor: Me strongly disagree
with reviewer two, who say me writing
read like two people be 'stitched together.'
Me feel much unhappy and me want to
squeeze reviewer two's neck until dead.
P.S. Fire bad.

3. Do you spot any errors?
Do you see anything that obviously needs to be fixed? Any typos, formatting errors, missing table labels, citations that you can't find in the references section, etc. Regarding the citations, you don't obviously have to check every one as that will be spotted in the proofing stage, but if a study sticks out in the text and you can't find the citation in the reference section, note that. If the reference section is obviously missing something important based on the journal's style (DOI's where required, first initial/names, etc.), note that.


4. Are there any citations that seem fishy? 

Why yes I did attend the
1932 presentation I cited
in my manuscript

One of my pet peeves when reviewing manuscripts is when the authors cite a conference presentation that's more than a few years old. First, I think "Were they actually THERE, or are they just citing the abstract they found online?" I then think "If this presentation was so important, why hasn't it been turned into a paper yet?" Either way, if you're citing a presentation that's more than, say, 3 years old, you probably should either find the resulting paper or find a different source, in my opinion.

If you see 12 different citations for
studies by Manatee and colleagues,
you're probably reviewing a paper
by Manatee and colleagues...

Do the authors cite websites that aren't scholarly (e.g., Wikipedia, Urban Dictionary, Blog Posts, etc.), but cite them as if they are scholarly sources of information? Do you notice any 'unpublished' citations (if so, they probably shouldn't be citing them unless those manuscripts are actually 'in press').

If the discussion and

conclusion sections sound

like these guys wrote them,

they probably don't deserve

a revise and resubmit..

5. Do you feel the article ties the lit review/analyses/etc. all together into a coherent final discussion and conclusions? 
When you finish the manuscript, does it make you go "Ah ha! That's the point!" or does it make you shrug your shoulders and go "So what? Who cares?" If the authors don't do a good job setting up the front end of the article, then they're probably going to run into a problem at the back end when trying to tie everything together into a coherent point. If you can see a glimmer of hope in the article, and the authors simply didn't do a great job at getting across the 'so what' about the article, then you should note that in requesting a rewrite of the discussion/conclusions. Relatedly, does the author miss a big piece of the results? Do they downplay an important finding that should be highlighted? Note these in the review, explaining exactly what you think the author missed or should otherwise correct, and why they should do this.


6. Overall, does it seem like the manuscript is a good match for the journal you're reviewing for? 
For instance, if the journal is focused on linking research to practice and policy (e.g., Justice Quarterly, Criminology and Public Policy), does the manuscript talk about implications for policy and practice? If it doesn't, and the main takeaway doesn't seems to have a clear implication for policy and practice (even after a revision), then perhaps the authors should instead submit to a less policy-focused journal (which obviously means they shouldn't get an R&R at the current journal...). The same goes for journals focused on quantitative research methods (Journal of Quantitative Criminology), theoretical aspects (Theoretical Criminology), Policing, or other specializations. If the manuscript is simply not a good fit for the journal, then you should note what journal might be a better fit.

Interesting conclusion,
Dr. Crazy-Sweater...
Revise and Resubmit.

7. Is the paper written in a neutral way, or does it come across as an advocacy piece?
If the paper is meant to be a scholarly research manuscript, it probably shouldn't read like an advocacy piece. For example, if the manuscript examines the relationship between religiosity and crime/deviance, it shouldn't read like something that could have been written by either the Vatican or the Freedom from Religion Foundation. If it does, then it probably needs to be scaled back in terms of tone so as to come across as more neutral. Note that this doesn't apply to all scholarly articles, as some forums (like the response essays in Criminology and Public Policy) are geared more for discussing certain viewpoints or implications related to the findings of the research manuscript.

Unless this guy wrote
the paper, there are
probably at least some
limitations to mention.

8. Does the author note the important limitations of the study?
All studies should mention important limitations. For example, if the study is about a single county, the generalizability of the study should be one of its limitations. If the manuscript doesn't mention this limitation, then it should. Related to this, does the study stretch the limits of generalizability? Does the one-county-one-state study try to pass off results as valid across the entire US or even the entire state? If so, something's wrong that needs more fixing than just adding something in the limitations section.


9. Time to make the master list.
Once you have all of the things noted on the manuscript (or otherwise written down) that you think are either 'wrong' with the article or need some type of revision, and at least a few things that are well done, you are almost ready to write your actual review.

I like to go back through the manuscript and make a final list of everything I didn't like, want changed, or liked, and order it by page in the manuscript. I like this method, because it lets the author more easily address comments and change requests, as they'll be able to quickly find what the review is talking about in each point.

While Smith and Fluffy (2011)
raise interesting points, I believe
they should give less emphasis
to the government mind-ray
theory in their manuscript.

If there are a number of things that all fall under the same category (e.g., a phrase you think should be changed throughout the manuscript, a misinterpretation of results, a typo that keeps popping up, etc.), then simply note the first occurrence of it on your list and indicate that this is a problem in other places as well.


10. Time to summarize everything.
After you have your final list with all of the things you're going to note to the authors, you now should write a few paragraphs (maybe a page or so) about your overall impressions of the article. A lot of the reviews I've seen begin with a short (2-3 sentence) summary of the article: what was the research question, theory, findings, and conclusions? If you choose to do this, great, but the 'meat' of the review comes after this.

While perhaps a correct
review, try for something
a little more scholarly.

When you write up the summary of what you think of the article, you should remember to emphasize what was really well done, and not just focus on what the major problems/fixes. Remember the complement sandwich! If something needs a little more research, feel free to suggest citations to use, but if you keep suggesting the author incorporate citations by "[Your Name] and Colleagues," you're unlikely to remain an anonymous reviewer for long.

The summary, in my opinion, should read as if you read a friend's dissertation and are giving the most important feedback up front to them. Don't mention every minor typo on your list (the list will speak for itself later), but if there are a bunch of typos throughout, mention that the author needs to do some major proofreading in the next draft.

You don't want to come across
as "the special reviewer"

Make sure that your summary comes across fairly neutral in tone, especially if your comments are geared towards the negative. You don't want to blow the author out of the water with nastiness, nor do you want to be kissing their ass. Praise the manuscript where you feel it's appropriate, and take the author to task for things that are egregious errors, but make it a fairly evenhanded summary.

The final paragraph of summary (before you provide the list of changes/notes/etc.) should indicate whether you feel the article is good enough to publish (pending a minor R&R of course), or if you instead feel that it's not an appropriate manuscript for the journal. If you think it's not a good fit for this journal, but a perfect match for another one, then say so.


11. Making anonymous comments to the editor.
You will also probably have a chance to write directly to the editor about the manuscript. Your comments in this special section will remain anonymous to the author of the manuscript. You should end up with perhaps a couple paragraphs of private comments to the editor that state what you really feel about the article, but don't feel comfortable saying to the authors. For example, if the article reads like an undergraduate student wrote the article, gave it to a professor to quickly read through and sprinkle in citations to his own work, and then shipped it off to the journal, egregious spelling mistakes and all, then say so. If the study is one of the best you've ever read on the topic, and that you think the study should be published with very few corrections, then say so. If you can't be honest when writing directly to the editor, when the author won't see your comments, then you probably need to reconsider reviewing the manuscript.

This is what happens when
you don't proofread things...

12. Proofread your damn review!
One last thing you should do is to proofread your entire review. You should be reading it both for grammar/spelling mistakes (especially if you call out the author for making spelling errors in their manuscript), and also to ensure you don't come across as either a complete jerk or as a clueless reviewer. You can say a manuscript shouldn't be accepted and still give the author some good feedback and a gentle letdown. Alternatively, you can say the manuscript should be published and is a great example of scholarly work, while still providing insightful feedback to incorporate before the final acceptance. If you read your own review and think "I wish I could get reviews (negative or positive) that read like this!" then you've done a good job.

Addendum: My friend has this bit of advice to add (Thanks Dr. Liz!): 

No matter how terrible the paper you review, tone down negative criticism. Avoid inflammatory language or personal attacks. Don't use exclamation points. Also, reviewing papers is an amazing learning experience about a) what journals ask reviewers to comment on and what kind of scale - from reject to accept - they use and b) what a good paper "needs" (this can help your own writing).

You mean I can review things
for you, for free, and not receive
any credit other than a line on my
C.V. that won't really help me get
a job or tenure? Hell yes I'd be
interested!

Postscript: How to become a reviewer for a journal
When I first started my PhD program, I wasn't really sure how people became eligible to review for journals. Sometimes, once you have an article accepted by a journal, you might automatically be added to their 'list' of potential reviewers. However, if you want to review for a journal you haven't been accepted at before, you can simply send the journal editor a short email explaining your interest in reviewing for them, and what your area(s) or expertise are. They will likely be happy to have you volunteer, and you might receive something to review within a few weeks (although don't hold your breath, it all depends on what is submitted to the journal.)





Final Note: A good link from a major journal publisher explaining what they are looking for in a journal review can be found here.

Do you have any other tips for reviewing manuscripts?  If so, add them below in the comments or send me a message!
---------------------------------------------------------
Photo Credits: Bag_Head Clearing Frankenstein Old_Man Thoughtful_Manatee Dumb_and_Dumber Protester Mr._Perfect Tinfoil Proofreading You_Suck Owls Phone_Guy (Yes, I'm addicted to this photo...)