Saturday, November 12, 2011

How to 'do' research (Part 1)

[Sorry about the long wait time since the last post.  The first-year-on-the-tenure-track mid-semester crunch really slapped the hell out of me these past month or so.]

I was chatting with a new Ph.D. student the other day about their research ideas and (eventual) dissertation topic.  They said that they had plenty of statistical training (they have two master's degrees), but haven't had much training (yet) in the way of research methods.  Midway through the conversation about their research interests and background, they asked me one hell of a question: "How do you actually do research?  Like, how do you find the literature, figure out a research question, write up a literature review, choose a dataset, that kind of stuff?"

I think 'doing research' is closer
to this than anything else...
The answer to that question, I think, is both very complicated and very personal.  I assume that everyone does their research a little differently from anyone else, and it likely depends on whose research methods courses you took in grad school, which articles and books about research you read (and understood/liked), your own personal preferences and habits, the kind of research you're doing, and a whole lot of other things that aren't immediately coming to mind.  So, I don't think it's as simple as "Do X, then Y, then Z, and viola!  Top-tier pub here we come!"

Still, it's a fair (and good) question to ask, especially as a new Ph.D. student whose largely expected to hit the ground running.  While my answer right then wasn't really well thought out, insightful, or likely even coherent, I've had a little more time to think about it.  Thus, this is the way that I typically conduct research (or at least part of it -- I'm sure there will be more to come, hence the 'Part 1' in the title).  It seems to work pretty well for me thus far, so feel free to steal anything that looks like it might help you, since I'm sure I stole most of this from other people along the way.

1. Read the literature for stuff that interests you.

Researching uninteresting stuff
is like watching Golf on TV:
Both make me want to stab my
eyes out with a spoon.
Note the second part of that statement -- 'for stuff that interests me.'  I think one of the biggest mistakes people make as Ph.D. students and new researchers is choosing research topics only because it's what's 'hot' in the field, or because their advisor does it, or because they can be funded through a project to do it, or for any number of other reasons that don't involve being genuinely interested in it. (Yes, there's obviously something to be said for doing research that will eventually get you a good job, but you should be able to balance the marketability of a particular research subject with your interest in it.)

There's a reason I left my stable, fairly well paying office job to earn my Ph.D.: I got to the point where couldn't see myself doing something, as my career, that I didn't absolutely love.  I still feel that way, but now instead of that unloved 'thing' being the 9-5 office job of shuffling papers, it is now conducting research that I really couldn't give two sh*ts about (or even one sh*t about). So I try not to do that on a day-to-day basis, and I think I've been fairly successful at it thus far.

So, I guess the precursor to step 1 would be 'figure out what interests me.'  This likely involves reading a lot of broader literature in your field to find the things that you want to read more about.  Once you've done that, and you know that you're really, really interested in X and Y, go ahead and look up more literature on X and Y and read it.

Keep in mind that this isn't simply reading the latest 3 years of top-tier journal articles that used X and Y as keywords, but also involves figuring out who the big names are (and were) in the field, what the major studies on X and Y are that get cited over and over again are, and what types of methods, data, etc. are generally used in those studies.  You should also read studies that might not be on X or Y, but relate to it somehow (similar methods, similar units of analysis, similar theoretical models, etc.).  All of this reading allows you to talk coherently in your research papers about what's been done before, who has done it, how they did it, and how your stuff is similar.

So, now that you're knee deep in the literature of the stuff that interests you, what's next?  Well, it's something that takes place while you're reading the literature...

2. Figure out what people have been doing poorly, or not at all, in those studies (i.e., find the 'gaps' in the literature).

Similar to the UK warnings,
if you don't mind the gaps in
the literature, you'll be hit by a
metro train called 'peer review'
This really only comes from reading study after study after study.  Invariably you'll be able to spot the limitations of studies that you read, whether it's generalizability, unsophisticated methods, not using Theory A, not studying area B, etc. etc. etc.  Sometimes the authors will do you a solid and list the 'real' limitations of their studies in their respective limitations section.  From what I've seen, however, most of these so-called limitation sections are really just a list of one or two of the most minor problems that barely constitute an actual limitation, with no mention of the more serious limitations of the study.  So read the limitations sections with a grain of salt, but if something keeps popping up over and over again (e.g., 'this study only used cross-tabs and not multiple regression', 'we couldn't study X, so we had to study a proxy of it, Y', 'we could only examine one county of data, so generalizability is a concern', 'historical/longitudinal data were unavailable for X, so this was cross-sectional in design'), then it's likely one of the main gaps in the current literature.  You want to figure out what these gaps are, because they can help you either craft a research question or craft your study's methods section.

For instance, if you find that prior studies have all examined individual states/counties/cities, and no one's compared multiple places, that could be the source of a new study.  Your research question might be as simple as "What are the state/county/city characteristics that influence X", and even though this exact same question might have been asked by multiple prior studies, if your study does something those other studies didn't (e.g., examine multiple places rather than just one), you'll probably be making an important contribution to the literature.  If studies have all ignored Theory A, and you think Theory A could be important in explaining X or Y, then your research question might be "Does Theory A explain the occurrence of X?"

Ok, so you've read (and are continuing to read) the literature on your topic, you've found some things that are missing in the current literature that you might be able to address, and you've incorporated that into your research question.  Now what?

3. Write a good, but concise, literature review. 

You should by now have a handle on who the big names are in your field, what the major studies are, and what the general findings of the literature are on topic X.  Now comes what is, for me, one of the hardest and most boring things: writing the literature review. There are a number of guides on this floating around the interview (see one example here), and I guess now there's one more...

Note that I'm talking not about the 40+ page literature reviews that get published as stand-alone manuscripts in journals. Rather, I mean the shorter, ~5-10 page reviews you see at the front of a 'regular' journal article.

The literature review helps lay the foundation for your study - it talks about what people have done, how they did it, and what they found, but it also connects those findings into a coherent narrative, as well as exposing the gaps that you will be filling with your study. That said, the literature review can vary tremendously depending on how your study is set up. I have most of my experience with quantitative, policy-oriented studies, so that's the type of literature review I'll be discussing. I'm sure this varies for qualitative, historical, theoretical, etc. studies, so buyer beware.

In any case, the point of your literature review should be to give the readers a crash course in what's already been done in the field that relates to your research. Sometimes this will be fairly easy and direct, since a lot of people will have done studies before you on a similar or same issue.

Another research idea:
The effect of access to hoodies
on rates of juvenile delinquency.
For example, suppose you're writing about the relationship between doing poorly in school and juvenile delinquency. If there were a number of studies, or even a specific theory, that can be easily applied to this issue (let's go with Strain Theory, since apparently everybody in my field wants to test this at one time or another), then much of your literature review might be discussing the theory, what it says, and how these studies have supported (or not supported) this theory. So you might want to set up your literature review by first discussing Strain theory generally, and then noting how it easily applies to your specific situation (i.e., doing poorly in school leads to delinquency). Then you can cite prior studies on school achievement and delinquent outcomes, and note whether they support Strain Theory or not. You would note the problems with the prior research, and how more research on this issue is sorely needed (although, really, it isn't in this case... so please stop doing these studies, folks...). This sets up your study by placing it in relation to the prior research and theory, and also helps distinguish it since you've addressed the limitations of prior research.

However, sometimes you're paving new ground, and need to discuss two or three different types of research and blend them together into a coherent story.  For example, maybe you're trying to examine whether access to a cell phone influences juvenile delinquency. It makes sense, since with it you can chat with your friends wherever you are, and you can more easily decide where to meet up to vandalize the school, or burgle the liquor store, etc. Maybe no one has previously looked at this specific issue (if not, I call dibs...). BUT, people HAVE looked at how communicating with friends (in person) can influence delinquency. So, you review this prior research. Then, you realize that socio-economic status (SES) is going to influence who has a cellphone in the first place, and so you need to review the research on how SES can influence delinquency. Now that you've done this, you finish your literature review by pulling together these two prior research fields (communication and delinquency, SES and delinquency), and relate this to juveniles communicating via cellphone, and support why you think this would (or would not) have the same influence on delinquency as face-to-face communication with friends, controlling for SES.

I can't be too much more specific on how to actually organize your literature review, because the organization is going to depend on a number of factors specific to what you're studying, how you're studying it, and what's been done in the past. I recommend reading a lot of articles, mostly from top journals, that have looked at similar things and see how those were set up. This should give you an idea on how to set up your own review section.

Side rant: Tone
Let me take just a minute to talk a little about tone when discussing the prior literature. I'm assuming that the ultimate goal of doing this research will be some type of scholarly journal article that will need to survive the peer review process. If this is the case, then the journal will send your work to other 'experts' in field. While this might include grad students that are just beginning their careers, it might also include some experienced professors whose past research you've cited and discussed in your literature review. How you discussed their research, then, matters a great deal...

Let's consider the following examples:

Example 1: Standing on the shoulders of giants.
Example 1:
"The prior research has been
tremendously important, but
has suffered from some minor
issues which I will address"
Imagine you're one of those distinguished professors reading an article for a journal. The author discusses your article in a neutral fashion, commends certain things that it (you) did well, and even-handedly notes what was missing or otherwise could have been improved upon. The author is acknowledging the contributions of your work, and indirectly suggests that they are 'standing on the shoulders of giants' with their present study. This sets the tone for your thoughts on the article. You think to yourself "Good call, sir! I should have used more than three years of data. I'm glad you're taking the ball and running with it, and glad my research could contribute!" as you sip your brandy and puff on your Meerschaum pipe while sitting by a roaring fire in your study (or where ever/however distinguished professors review journal articles...). The rest of the article is well done, so you send off a positive revise and resubmit recommendation to the editor, with encouraging words for the author.

Example 2: All my predecessors were idiots. 
Example 2:
"The prior work is total
doo-doo done by smelly
caca-heads. My research
is good because it isn't
poop based." 
Now, imagine you're that same distinguished professor reviewing a very similar article. The author cites your work, but does so in a dismissive, critical tone. It's clear that they think your prior research was garbage, and that you were obviously a drooling idiot and/or untrained monkey. While not specifically saying this, they imply that their research can 'save the field' from its past horrific, worthless studies conducted by lesser researchers. "Humph and habberdashery!" you exclaim as you quickly type up your review to the editor with the strong recommendation to reject this horrible, insulting manuscript on the spot. "Little sh*t thinks my research is garbage, does he? Well good luck getting into [top journal of the field], muahahahahaha!" you cackle as you fire off your recommendation to the journal and casually toss the reviewed manuscript into your fireplace.

So kiddos, what did we learn from these examples? Well, first, apparently there are still some people in the world who use the phrase 'habberdashery.' Second, and more importantly, the tone you use to discuss the prior literature can be very important. Yes, you need to point out the gaps in prior studies so you can set up reasons for why your research is an important contribution. However, praising the prior researchers for their contributions, and implying that you're using their research as a basis for yours to help further the field, is a very, very different story compared to one where you're bashing the idiocy of prior researchers to promote your (obviously) much more sophisticated study. You want your manuscript to come across as 'standing on the shoulders of giants,' rather than 'chopping the giants off at the kneecaps.'  So tone matters.

Hopefully what I've written so far will be somewhat helpful to those of you just starting the research process. I'll be posting more on this topic in the future, but this should get you started.

Image credits: GolfHoodies, Old ScholarBaby Scholar, Mind the GapThen a miracle occurs (Harris, 1977)

Friday, September 30, 2011

Writing an Acknowledgement Page in the Dissertation

One thing you'll have to do near the end of your dissertation writing days is to come up with an acknowledgement page.  This can be a tough thing to write, but it's your chance to put some real personality into your writing.

Even though you might feel
like this near the end of the
dissertation process, try to
refrain from thanking your
dissertation directly as if it
were a human being.  Save
the crazy for after tenure.
If you haven't really thought about writing the acknowledgement page yet, but it's getting close, I'd recommend checking out a few recent dissertations and seeing what THEY wrote.  You'll likely read some that you think are horrible/boring/sterile, and some that are really touching/interesting/funny.  It's really up to you whether you want to go with the standard business-like thank yous or make it more personal, though I recommend the latter.

So, once you decide the type of page you want to write, you have to come up with a list of people to thank.
Here's some ideas that might help you remember everyone who helped you along the way.

1. Your dissertation advisor/committee -- Obviously these are the people that worked with you to get your dissertation finished, and in my opinion they should be either the first people you thank in the acknowledgments, or the last people you thank.  I went with first, saving the last thank yous for my family.  You should single out your dissertation advisor (DA) from the other committee members (CMs), as the DA likely put the most effort into your dissertation compared with the other CMs.

2. Your other 'advisors' in the program.  You may have been assigned an advisor early on in your Ph.D. program that ended up not becoming your DA, or even a CM, but provided a lot of early advice and encouragement.  If so, they should be included in your acknowledgements.  If they are one of your CMs, then single them out in addition to your DA, and if they are you DA, then be sure to mention the non-dissertation support they provided for you.

3. The other faculty you frequently bugged for help.  When I was in the Ph.D. program, there were a few faculty that I frequently bothered for advice on a number of topics, despite having no formal mentor/mentee relationship with them.  The advice ranged from detailed methodological/statistical advice to inform career advice.  I'm sure by the time I was ready to graduate, they had developed an inadvertent eye-twitch whenever they heard me knocked on their door and/or start a conversation with the phrase: "So I've got a quick question for you..."  Even though they may not have directly helped on my dissertation, they still helped me navigate life as a Ph.D. student, and so they deserve to be thanked.

4. The staff you bugged for help.  There is likely at least one staff member who not only knew you by name, but did things above and beyond the call of duty to help you out.  This might have involved helping you file all of the misc. paperwork for a grant application at the last minute, helping find you a last-minute assistantship position to keep your health insurance in tact, or even printing on their fancy high-quality laser printer 20 sheets of stick-on beer labels for your newly brewed batch of Maple Ale.  If anything like this applies to a staff member know you, then you should thank them.

There's a reason the show wasn't
just called Garfield. Dude needed
help filling the other 15 minutes.
5. Your friends.  The Ph.D. program can be very isolating and lonely.  If you're lucky, you will find a tight-knit group of friends to commiserate with, whether they are part of the program or not.  Even if they don't know the first thing about your research and never read a draft, they had a small part in helping you get through the program and finish the dissertation.  You should thank them in your acknowledgement page (by name for at least some, with a catch all 'all others who I haven't yet thanked').

6. The people who got you to grad school.  I know that if it weren't for the people who wrote my recommendation letters, I wouldn't have been accepted to the grad program that I graduated from. Therefore, I felt I should thank them for the time they put into me both as an undergrad student of theirs, as well as for the letter writing that they did for my application packet.

8. Other support networks.  If you have other support networks, like a grad student forum that you've used for advice over the years, writing support, etc. don't forget to include them if it makes sense to.

8. Your family.  This is an obvious one, and who you specifically mention will vary based on your own circumstances.  In my own acknowledgement page, I put my family (parents, sibling) next to last, and put my wife last, as I felt the first and last positions held the most importance.

Final thoughts: Finally, just a couple words of wisdom regarding the acknowledgement page.

First, I don't personally believe this is something that your dissertation committee needs to see.  It is personal and doesn't affect the merit of your dissertation work in terms of defensible or not.  So, if you don't feel you want the committee to see this page until after the dissertation is bound and printed, then I see no concerns with slipping it into the dissertation after the final defense and before the final product is sent to the university's graduate office for final approval and publication.  (Your mileage may vary, however).

Second, you should spell check your acknowledgement page very, very, very carefully.  This is because the rest of the dissertation has probably been read dozens of times by various people, but this is something that, probably, very few people will have a chance to read before it's in print.

Dr. Seuss had many friends.
Dr. Seuss liked to use pens.
Those who mattered minded not.
Those who did he f*cking shot.
Third, I think there's nothing wrong with adding some humor to this page.  To crudely paraphrase Bernard Baruch and/or Dr. Seuss, "those who matter won't mind and the rest can go f*ck themselves."  I would much rather have someone read my acknowledgement page and laugh (or cry, if you're my wife -- yes, it was that good/touching/etc.), than have someone read it, shrug, and go "eh, pretty good."

Finally, you should be proud of your acknowledgement page when you're done.  My acknowledgement page, I think, is pretty damn good, as well as interesting, funny, and touching.  Plus, where else in my dissertation could I have used phrases such as "hobo salary," "purchase the 'good' beer that individuals of stature and sophistication drink," "ridiculously strong back muscles," "totally-should-be-subject-to-freedom-of-information-laws-but-it-is-not," "slow-but-steady financial support (one poker game at a time),"  "squeezing the bejesus out of him," "defend myself from an onslaught of up to four enemy samurai," and "zombie apocalypse."
...See, don't you feel at least a little interested in reading my acknowledgement page now?  That's what I was going for.
Image credits: Hugging_Words, GarfieldDr. Seuss,

Saturday, September 10, 2011

How to respond to an R&R

[Note: Some of the follow suggestions I picked up from my faculty advisors and colleagues while in grad school, others I came up with on my own, and still others came from Wendy Belcher's excellent book: Writing Your Journal Article in 12 Weeks]

So, you submitted a manuscript to a journal, and the editor liked it enough to give you an R&R... now what?

First, congrats! An R&R is about as good as you can expect for a regular journal, since very few manuscripts are accepted 'as is.'

Depending on the editor's letter and reviewers' comments, it might be an easy R&R that won't take much time, or a very hard R&R that will take weeks or months to address. Also, depending on the journal, an R&R might mean the manuscript is basically accepted, pending a little polishing, or alternatively that there's a good chance you'll still get rejected later on in the process. In any case, your goal should be to address as many of the recommended changes as is reasonably possible in the revision to make the paper better, without accidentally 'messing up' your paper (or your main points) in the process. Hopefully this post helps guide you there.

So the first thing you should do is read the editor's letter in its entirely (you probably opened the letter/email and just skimmed the page(s) until you saw 'revise and resubmit'...).  The editor's job is to read the reviews, make a decision on your manuscript (in this case, R&R), and then (in my opinion) give you guidance on what are the most important things to address in the revised manuscript. If the editor didn't do that, or gave you a generic 'please address the reviewers' comments in your revised submission,' then your job will be a little harder than it probably should be.

Anyhow, read the editor's letter and see if s/he gave you specific instructions as to what to address. The editor might note the important of a missing variable in your model, as mentioned by reviewer 2, or that your treatment of the prior literature was lacking in X area, as noted by reviewers 1 and 3. Whatever the editor says in their letter regarding potential changes should be your first priority in the revision. What I like to do is open a new word document and make a bulleted list of the changes the editor specifically mentioned in the letter (these bullets go under an 'Editor' heading). This will become part of your 'potential revision' list/spreadsheet that you will continue building in the future. (More about this in a bit).

Next, read through all of your reviewers' comments, without taking any notes. Make sure you have your submitted manuscript draft in front of you, so you can reference it when a reviewer mentions something on page X or in Table Y. Keep an eye out for comments that come up from more than one reviewer, such as the suggestion to use robust standard errors in your model, to provide a correlation matrix table, or to cite the work of authors A and B.

Chances are, you won't agree with everything the reviewers recommend, and you'll form the following opinions about the reviewers [note: for ease of comparisons, all three reviewers are assumed to be male in this example]:

Reviewer 1: Even my
bowel movements are
high impact.

Reviewer 1: You'll probably think this reviewer is a genius, particularly since he gave you the most useful recommendations, and other than a couple minor suggestions, loved your entire manuscript. His suggestions are relevant, fairly easy to implement, and will really help boost the quality of your manuscript. This reviewer, while rarely seen in the wild, is the best you can hope for.

Reviewer 2: "Derp?"

Reviewer 2: You'll probably think the second reviewer is an idiot, but a fairly harmless one. He probably doesn't really know anything about either your analytical methods or the literature related to your topic, but tried to fake it so the editor didn't get angry. Most of the suggestions he offered aren't very useful and/or make no sense given your model, argument, and findings. Still, you can easily address his main concerns, and easily dismiss his unhelpful and misguided recommendations.

Reviewer 3: My comb-over
is horrible, I can't eat sweets
due to my diabeetus, and I'm
going to take my anger out
on you and your manuscript.

Reviewer 3: Upon reading reviewer 3's comments, your first thought will probably be "what a f*cking d*ckhead." You'll probably think that every suggestion by reviewer 3 is complete garbage, that he misunderstood your entire argument, knows nothing of the methods you used, and decided to use his "anonymous" review as an outlet for his deep-seated inner frustrations concerning sexual impotency and male pattern baldness. You'll likely question his academic and genetic pedigree, and wonder how the editor could have chosen such an arrogant jerk. This reaction is completely normal, and has a long historical precedent.

In fact, Einstein even developed a theory related to reviewer comments that largely supports these three general categories, as previously noted in my post on how to get over manuscript rejections:
More useful than relativity.
However, these categories are not exhaustive, nor are they exclusive. You may, for instance, get a review from a reviewer that combines traits of 2 (idiot) and 3 (jerk).

 Regardless of the types of reviewers you get, or the content of their reviews, the fact that you received an R&R meant the editor saw some promise in your manuscript. Therefore, now that you've surveyed all of the reviews, you need to get to work addressing the specific suggestions and critiques, and revising your manuscript for resubmission.

So the next step is to go back through the reviews and add to an excel spreadsheet (or equivalent) every specific suggestion or critique, from each reviewer, that requires you to change something (or might require you to say why you didn't change something.) Each suggestion/comment should be its own entry on a separate line. I recommend five columns in the spreadsheet: ID # (increments up), Reviewer (who made the comment), Comment (what was the change/critique), Importance (How important does this seem to your overall revised manuscript, or to the editor/reviewer?), and Ease (How long and/or how much work is this going to take to address). I also like to color code the importance and ease cells using the an easy to distinguish color scheme - red for important/hard, yellow for medium/moderate, and green for low/easy. If you set it up like this, your spreadsheet should look something like the following (excluding the earlier comments from the editor):

Once you've finished adding the reviewers' comments to your spreadsheet, add the editor comments you previously listed out if they're not already covered by a reviewer, and mark anything the editor said to do as high importance. If the editor says something a reviewer already said, then add 'editor' to the reviewer cell as a reminder, and mark it as high importance regardless of your original importance rank. If the editor specifically says something is not important to address, then mark it as low importance, and if they say not to do it at all, then change the font style for the associated entry to strikeout.

After everything is down on your spreadsheet, you can get a sense of the overall work that's ahead of you by looking at the importance/ease categories for each entry. A roadmap to these combinations, and what they mean for your revision, is below:

High/Hard: These are the changes that will give you the most headaches. You need to do them (high importance), but they're going to take a long time and generally be a P.I.T.A. (hard to do).  Doing these first will make your life much easier later on, and if they're major changes, some of the less important changes may be obsolete once you do these. If you decide not to do one of these, you need a very good reason why not.

High/Moderate: These are changes that will be a little easier than the High/Hard changes. You still need to do them, and they're going to take you some time, but they won't be absolutely horrible to address.

High/Easy: These are the changes that will make you feel most productive, since you can address these important 'need to do' changes very quickly and without much work.

Medium/Moderate: You should probably do these changes, especially if you have the time.  If you don't do one of these change requests, then you should definitely address why you didn't in your cover letter (more about the cover letter in a minute).

Medium/Easy or Low/Easy: Since these are easy to do, you might as well do them, especially for the ones of medium importance. If you choose not to do these, you should at least acknowledge why you're not doing the medium importance ones, if not the low importance ones as well (since not doing the latter will make you look lazy).

Low/Moderate or Low/High: These might be changes you decide not to do, given their relatively low importance compared to the moderate-to-high amount of work they'd require of you. Again, you may want to address why you didn't do these in your cover letter, but given their low importance, if you miss discussing some of these it probably won't be the end of the world.

Ok, so you have your potential changes all in order, you've ranked their importance and ease, you know which ones you need to address, which you should address, which you can easily address, and which you're not wasting your time with. Now comes the hard part: Make these changes in the manuscript. That's about as specific as I can get for this step, but this is what will take the longest.

Once you're done making (or not making) the various changes, you need to reread your manuscript from start to finish to make sure things still make sense. This is especially true if you had to do some serious changing of text. Edit as necessary.

"Dear Editor: Thank you for
the opportunity to revise
my manuscript. Unfortunately,
as I am not fluent in dumbass, I
have been unable to interpret
Reviewer 3's critiques."
After your revised manuscript is re-polished into a final version, you need to draft your cover letter to the editor. I recommend starting the letter with a brief paragraph thanking the editor for the opportunity to revise and resubmit your manuscript for their journal. You should also thank the editor and reviewers for their helpful suggestions and critiques (if even you think most of the suggestions were garbage). Then I recommend stating something along the lines of "I've provided specific responses to each of the suggestions and critiques below. I look forward to hearing your final decision." This will provide a transition between your formal paragraph and the subsequent list of changes that's about to follow, as well as obviate the need to write a closing paragraph after your list.

Skip a line or two, and then list out each change you want to address (all of the high importance ones, most/all of the ones of medium importance, some/most of the low importance ones). Briefly summarize the suggestion or critique, identify who said it, its location in the manuscript, and what you did to address it (or, alternatively, why you did not address it). Examples of wording for this section are provided below:

Reviewers 1 and 2 recommended adding r^2 values to Table 3 (p. 21). This has been done for all three models.
 Reviewer 2 recommended revising the paper to focus on racial threat theory. While this is a valid alternative theoretical focus, I feel that racial threat is not appropriate for the current study given the current focus on non-racial factors, as well as the comments of Reviewer 1 regarding the appropriateness of the current theoretical model. Further, using racial threat would generate the problem of..."
 Reviewer 2 questioned the need for spatial lag measurements in the final model. I have expanded the justification for this measurement on page 4, but believe that the inclusion of this measurement is essential in order to control for the spatial autocorrelation of the observations that was indicated by the Moran's I tests discussed on page 12, footnote 3. As such, this measurement has been left in the final model.
Reviewer 3 recommended deleting the third paragraph on page 2, as it was considered unnecessary. On the contrary, I believe the key points of that paragraph are essential to understanding why the study includes measurements for different types of vehicles as well as the differences in operator characteristics. Without it, the subsequent three paragraphs appear disjointed from the rest of the literature review. As such, this paragraph was left in the article as was originally written.
 Make sure you come across as respectful in your comments, especially when discussing suggestions that you didn't implement, and certainly don't word your responses as attacks on the reviewers' or editor's suggestions. If it's clear that one of the reviewers is clearly confused on a certain point, model, or statistic, spin the wording of your response in such a way so that it seems like it was your fault for not being clear, and then justify your original point/method:

Reviewer 2 seemed confused by my use of the term "spatial proximity" on pages 3 and 6. In order to address this potential point of confusion for readers, I have added footnote 2 on page 3 so as to clarify what this term precisely means for this study. 

When you're done, reread your cover letter to make sure, one more time, that you come across as respectful, attentive, and appreciative of all of the suggestions. When you're satisfied that everything sounds good, send the revised manuscript and cover letter to the editor (or upload them to the journal website, as appropriate), and cross your fingers. With any luck, you'll hear back in a few weeks (or a few months) with good news! If not, then see if you can revise the manuscript any further, and then submit it to a different journal to start the process over again.

Image Credits: The Most Interesting Man in the WorldMr. Bean, Reviewer 3Einstein, Letter Writer

Thursday, September 1, 2011

How to handle a journal rejection

In one of my prior posts I discussed how to write reviews for journals. Sometimes those reviews result in good news from the editor, like a revise and resubmit (R&R) decision. The majority of reviews, though, will probably result in a rejection letter. This is the result of basic math, given that top-tier journals have acceptance rates well below 50%. Having recently experienced two back-to-back article rejections myself (same journal, I might add), I figured it might be useful to talk about how I, personally, handle having my manuscript rejected, as well as some thoughts on how to effectively deal with it.

One of Einstein's more relevant theories

First off, this is typically the timeframe I go through with my rejections, and my actions/thoughts along the way:

Day 1: The editor and reviewers at journal X are complete idiots!! Efffffffff Yooooooooooo reviewers!!!

Day 2: Ugh, I can't read this review again without getting angry.  Efff Yoooo!

Day 3-14: [Do something else]

Day 15: [After rereading the manuscript and making comments on the draft]  Hmmm, maybe my submission wasn't as strong as I thought, and maybe the editor and reviewers are only mildly stupid (Eff Yoo!). Still, they might have brought up a couple things that I might be able to fix.  ...And maybe the manuscript was a little long and did have a few too many appendices.  I'll work on the revision tomorrow.

Day 16-???: [Revise manuscript]

Day ??? + 1: Ok, the manuscript has been revised, it's much better than the previous submission, so now where should I send it?  Obviously I can't resubmit to journal X (eff u), so lemme look into journal X...


Does this above sound like a good timeline/plan that might help you get over a manuscript rejection?  Did you just get a rejection and are still stewing with righteous anger?  If so, then read my step-by-step instructions below!

First, when you initially get the rejection email or letter, you'll likely feel like this:

'Generalizability Concerns?'
Well Effff Yoooooo!!! least, I typically do.  I bet there are few people who can receive a rejection letter and feel fine about it.

I recommend the very first thing you do is print and read the rejection comments (from the editor and the reviewers, if included) as soon as possible, preferably that same day.  

Next, do one (or more) of the following to the pages you just read:

  • Burn them (watch out for smoke detectors)
  • Smash them into a ball and punch it with your fist
  • Shred it in the paper shredder.  Throw in a blank CD-R at the same time for a feel-good crunching sound
  • Tear it to bits and toss it in the air like confetti (make sure to do this in a common area that you don't have to clean up)
  • Stomp on it (works better if you first crunch it into a ball), and then kick the smushed ball across the room.
  • Use them for gun/archery/whatever target practice (this one is probably the most satisfying)

Tony recently had his manuscript rejected from
the Journal of Drug Control Policy...
They didn't like his theoretical model I guess.
...feel better?  Only a little?  Ok, good enough for now.

Now that you're hopefully feeling a little better, reward yourself with a treat.  You've earned it, so treat yourself to a box of cookies (or a drink, or a steak, whatever).  You might be asking: "But my manuscript just got rejected?!? What did I do to earn a reward?"  Ahh, it's what you didn't do that earned you the reward.  Specifically you (probably) didn't:
  • Fire off a snarky email to the editor telling them off
  • Punch a hole in your LCD monitor
  • Throw your laptop across the room and/or into the face of a coworker
  • Curl into a little ball and cry (though it's ok if you did this one... you can still have a cookie)
So, treat yourself with cookies or something.
C is also for Craptastic Reviewers
Ok, now that you're hopefully a little calmer and feeling a little better, this next part is very, very important.  If you're not under an extreme deadline to get the paper published (e.g., you're racing against a competing academic so you don't get scooped), then take a week or two without looking at either the paper, the associated data, or (ESPECIALLY) the rejection email/reviews.  Just work on something else for a week or two.

After a couple weeks, you should be feeling a little more like this:
A little less pissed off... that's good!, still a little hurt, and a little pissed off, but you're getting there.  Now, read through your original submission (the one that got rejected), and try and read it as if you're a reviewer and that is not your own writing.  As soon as you're done reading what you submitted, then read the rejection letter and reviews again.

Having read what you submitted, do the reviews seem a little more legitimate in their critiques? Oh really, your submission was perfect?  THEN READ IT AGAIN.  Ok, did you find something that might be worthy of revising a little?  Good.  So, how do you feel towards the reviewers/editor now?

Notice the smile?  That's the spirit!

Ok, that's good!  So, you probably noticed that your masterpiece of a manuscript had a few places that you could improve.  If so, great!  Get to work revising the manuscript, trying to incorporate into the revised version as many of the valid critiques from the reviewers and/or editor as you feel comfortable with.  Eventually, you'll have a finished revision that you can submit to a different journal, hopefully with a better outcome this time (if not, start back at the beginning of this post).

Regardless of whether your coping strategies are similar or different (or the advice above works for you), you should at least be willing to revise the manuscript and send it somewhere else.  Don't let a single rejection result in tossing the manuscript in the garbage (maybe after 3 rejections, but not after the first...).  Also, you have to keep in mind that any decent journal is going to reject more manuscripts than they accept, so chances are if you send them something else, they won't hold your prior rejection against you, and so you shouldn't hold the prior rejection against them (eventually... give yourself some time to be pissed with them).

Image Credits: Finger Old_Finger Young_Finger Tony Cookie_Monster Einstein

Sunday, August 28, 2011

Having a creative outlet outside of academia

Yes, when I get stressed, I basically
turn into Tweak from Southpark.
One of the things I think is incredibly important for making it through grad school (and hopefully through the tenure track) is to have a creative outlet that doesn't involve academia.  These outlets let you get away (even temporarily) from thinking about classes, about the dissertation, about teaching, about publications, about the job market, and all of those other things that caused my eyelid to twitch from thinking about them too much. (Seriously, the eyelid twitch is God's way of saying "Look, you've got to calm down before I give you a stroke.")

When I was in grad school I had a number of different hobbies, some short-lived, some not, that let me escape from the grad school, homework (when I still had classes), research, and academia in general.  Due to my self-diagnosed hobby A.D.D., the assortment of things I used as creative outlets and general stress relievers was pretty varied. Here's a list of three things I did (and may still do from time to time), along with the reasons I found them to be enjoyable, and things to consider if you wanted to take any of these up in the future.

This is why I hate swimming.
Thanks Speilberg.
1. Kayaking
My wife and I took up kayaking after her parents took us kayaking a couple times during a summer visit. While I am not a strong swimmer, and generally am terrified of swimming in natural bodies of water (holla back if you saw Jaws as a six year old as well!), I found I actually enjoyed floating on top of the water, and being able to easily control where the kayak went.

The best part of kayaking, I think, is being forced to go out into nature and actually do something.  Kayaking gets you out of the house, let's you see and explore the natural surroundings in your area, and is very relaxing. Note that I am not talking about whitewater kayaking, which requires a helmet, fast moving water, an updated will, and the ability to not freak the f*ck out when you flip over and are strapped into your death trap of a boat. I don't think I would find that relaxing, and since I'm allergic to drowning, I don't think I'll be trying it anytime soon. Rather, I'm talking about kayaking on slow moving rivers and calm lakes.

Yeah... not this kind of kayaking...
The worst thing about kayaking is it's very weather dependent, and in the northeast that means about six months of the year it is too cold (or frozen) to go, and is also a no go when it's raining.  The startup costs are also one of the downsides, at least initially.

Kayaking has pretty high startup costs, especially if you want to buy your equipment and not rent it every time from someplace like L.L. Bean (which gets expensive quickly anyway).  A good entry level kayak will start at about $250, and you'll need a paddle ($50), a life vest ($25+), a carrier of some type for your vehicle ($50-$200), associated straps and ropes ($25), and things like dry bags ($10-$30) and other accessories.  It will probably cost at least $500 to get into kayaking (per person, if you go in as a couple like my wife and I did).  The nice thing is once you've bought your stuff, the actual kayaking trips will be virtually free, minus gas to drive places and maybe admission to state parks.

2. Iaido
My friend Matt convinced me during my first year in the Ph.D. program to join him for an iaido class. His sales pitch was basically "Do you like swords? You get to swing them around in Iaido."  Since I do, in fact, like swords, and wanted to swing them around without someone calling the po-po, I decided to give it a try.

Thanks to Iaido, I can theoretically
defend myself in this situation.
Very useful on the college campus.
Iaido means 'the art of the sword,' and involves learning the appropriate techniques for drawing a samurai sword (an iaito), making clear and precise cuts against imagined opponents, while portraying a level of zanshin (roughly a conciousness and energy), for certain scenarios (kata).  These scenarios stemmed from various attacks that a samurai may have encountered and had to defend against in his daily life in Japan during the Edo (and similar) period.  It does not involve attacking actual people.  For that, look into kendo, which uses body armor (bogu) and bamboo 'swords' (shinai).  Kendo also comes with a lot more bruises than Iaido, so keep that in mind.

Iaido is like this, but with
more 'wooshing' sounds
The best part of Iaido, in my opinion, is that it's very easy to let non-Iaido things slip out of your mind when you're doing the kata.  You have to have a LOT of focus to do the moves correctly, as it's not just where you swing the sword, but how you swing it, your stance, your 'energy,' etc.  It's also a decent low impact workout, and the people who do it are generally interesting people who are fun to hang out with.

The worst part is probably that it requires constant training if you want to be any good at it.  It's not really something you can pick up for the month, put down for six months, and pick back up for a month.  Like any martial art, to learn Iaido you likely need to find a school to join (run by a sensei), attend regularly, pay school dues, and practice outside of class.

Iaido has low initial startup costs, as you can practice in gym clothes (no shoes) and use a wooden sword called a bokken ($15?).  Many schools will let you practice with these for the first few classes until you decide that Iaido is something you want to continue to pursue.  When you get more into it, you can spring for the metal iaito sword ($100 - $1000+), and the samurai clothing (a gi and a hakama, $100+ for both).  The main expense from week to week will be the class fees, which could be free (unlikely), or could be  more than $50/week.  Finding an instructor may also be a challenge depending on where you live.

3. Home-Brewing Beer 
So let me say up front that I don't drink a lot of beer.  I drink beer, I just don't drink a lot of beer.  This, apparently, is not a good trait for being a home brewer, as you will quickly end up with hundreds of bottles (literally) of beer that you won't be able to possibly drink yourself, and with friends who are tired of drinking your homebrew beer.  That said, I attended a party where someone brought some homebrew beer, and I was intrigued.  It seemed like an interesting hobby where you could get creative and tweak recipes to brew what you liked.  When I got the opportunity some time later to purchase my own homebrew kit, I dove in.

It's best not to think
about what this looks
like and just appreciate
the finished product...
Homebrewing is a fairly simple process, but involves a lot of steps that are all equally important to getting a tasty (or at least drinkable) resulting beer.  The process starts by making what is essentially a hot soup of sugary liquid (whether extracting it from grain yourself, or buying pre-made sugar extract), bringing it to a boil with a lot of water, adding specialty grains to give flavor and color to the mixture, throwing in hops at certain times during the boil to give bitterness, taste, and aroma, cooling the mixture down, adding specialty yeast, and putting everything in a large glass bottle (carboy) or plastic bucket to sit for a few weeks while the yeast turns the sugary liquid (called wort) into beer.

The best part of homebrewing I think is the ability to customize your recipes to what suits your tastes.  For example, I'm a fan of big malty beers, so I tweak recipes for barleywines, porters, and stouts.  Other homebrewers love hops, and so they focus on brewing beers like IPAs.  Thus, you can make what you like, and in fairly large quantities, for about the same cost as buying beer in the store.  This gives you a creative outlet to make, tweak, and try out different recipes, and you have a tangible product at the end of it (albeit about a month later) that you can hold in your hand and say "I made this!"

The worst part of homebrewing, probably, is that you don't actually 'save' any money by homebrewing.  For instance, a good recipe for a high alcohol beer might cost upwards of $50 in ingredients, and produce about 4 cases (48 bottles) of beer, for a cost of a little over $1 per bottle.  That's not that much less than buying decent microbrew ($8/6 pack?) and is a lot more expensive than buying a 30 pack of cheaper beer.  Actually, if you want to brew a basic America Lager (think Budweiser or Coors), you're going to end up spending significantly more money than if you had simply bought the same type of beer in the store.  It also takes a lot of patience to brew the wort (5+ hours?), and then wait for the beer to ferment in the carboy or bucket (2-3+ weeks).  So if you're one of those 'immediate gratification' people... well.. you probably shouldn't be in a Ph.D. program for one, but you also will probably find it hard to homebrew, for two.

"After I had to mortgage my house to afford
all of my homebrew equipment, my wife left
me and took the kids.  BEST DAY EVER!"
Homebrewing has moderate startup costs, as you can get a startup 'kit' for around $100 (or more, depending on how fancy you want to make it).  The kit will come with a couple brewing buckets, hoses and other tools, and a basic ingredient kit (you supply a large pot).  You can also buy ingredient kits for a basic low alcohol beer for around $30-$40, more if you want something like a Barleywine, Stout, or Porter.  You can also customize your homebrew setup with larger brew pots, more carboys or buckets, higher quality tools, etc., so you can quickly drop a LOT of money into homebrewing if you're not careful.  If you stick with the basic setup, then you can figure somewhere between $30 and $60 per 5 gallon (~48 bottle) batch of beer.

"My 'hobby' is academic research... as well as
heart attacks... lots and lots of heart attacks."
In conlusion: There are plenty of other hobbies out there that range from costing almost nothing (e.g., hiking, volunteering, blogging), to being very expensive (e.g., collecting antiques, skydiving); that range from being relatively quick (swimming, working out) to relatively slow (knitting, reading).  The trick is to find something that interests you, fits in your budget (both in terms of time and money), and gives you a way to get away from academia, if only for a little while.  Having a non-academic-related hobby will help keep you sane, make you well-rounded, and give you perspective.  I highly recommend everyone taking up something of interest to them that is outside of the academic world.
Picture Credits: Tweak Jaws Kayaking Iaido_1 Iaido_2 Carboy Beer_Dad Professor_Heart_Attack

Wednesday, August 17, 2011

Watch out for the post defense blahs!

So this post is a little more serious than some of the previous ones.  Around 4 months ago, I defended my dissertation.  I wasn't really aware of what to expect immediately afterwards, and hopefully this post helps some people who are in a similar boat.

My actual dissertation defense was fairly easy, with no curve balls from the committee or moments of doubt once the defense started.  While I stayed fairly calm in the weeks leading up to the defense, the day before I was ABSOLUTELY. FREAKING. OUT.  I felt that I was going to forget some key piece of my dissertation (like one of my research questions or main findings), and/or that I would completely choke under the pressure.

Everyone I knew who had already gone through it (including my own dissertation committee members) said the defense would be fine, and would be more like a friendly chat about my research and future directions than an actual defense of the work I completed.  While I "heard" what they said, I wasn't able to really internalize it and overcome my self doubt.

I imagine I looked a little like this
during the defense.  I blame coffee.
In any event, I wasn't able to sleep much the night before the defense, and so I went into my defense running on about 2 hours of sleep and a pot of coffee.  Despite being tired to the point of almost falling over, the defense went fine, and I became a 'Dr.' (which, from what I've been told, is conferred upon successful defense, while the 'Ph.D.' usage comes after successful conferral of the degree itself.)

Immediately after the defense, I felt pretty damn good. I was finished, and even if I got hit by a car and killed walking back to the parking lot, I'd go out as a Doctor (sorta).  After going back to my (on-campus) office and receiving some congratulations from coworkers, and calling my wife and my mom, I went home early to crash.  I slept for about six hours, got up, and then felt... well... nothing much.  Maybe a little relieved, but otherwise just sort of empty.  I knew I was basically finished (barring some minor formatting things required by the University beancounters), and was already a 'doctor' in title, but I still just felt a little... blah... if that makes sense.  I had a 'so what' kind of feeling about the successful defense, the culmination of what I'd spent the last 5 years pursuing, and the last 2 years directly working on.

If you DO see a sad
little bubble following
your every move, you
should probably call
a real doctor...
Unfortunately, this 'blah' feeling stuck with me for probably a month or so.  I had occasional flashes of 'F*ck Yeah!' feelings that would last a few minutes or maybe an hour (like after walking across the stage at graduation), but nothing constant like I had imagined prior to the defense.  It wasn't full-blown depression by any means (at least, I didn't see a sad little floating bubble following me).  However, there definitely were occasional feelings of sadness at the 'race' of the dissertation being over, spiked with a little 'Oh Sh*t' feelings at what I needed to do next (i.e., get tenure... eventually...).

After a while, the 'blah' feelings subsided, and were replaced with other feelings and thoughts, like stressing about upcoming deadlines, travel plans, and work, my upcoming cross-country move, and day-to-day things like working on research projects. I began to get more and more excited about other things, too, like submitting (revised) pieces of my dissertation to journals for publishing, planning out my fall syllabus, looking into housing options where I would be moving to, etc.  The feelings that were at the forefront of my thoughts, namely my new status as a doctor, eventually became a feeling in the background of my thoughts, that I now forget about more often than not.  A somewhat similar experience happened after I finished my Master's degree, but it was on a much smaller scale compared to finishing my Ph.D.

I'm writing this blog post not as a review of what everyone always experiences after defending their dissertation, nor is it meant as a warning of impending post-dissertation depression.  I simply want to throw out the idea that you might feel a little 'blah,' a little sad, a little depressed, a little angry, a little scared, or feel nothing much in the days and weeks after you defend.  So, if you know this *might* happen, you can hopefully deal with it if it occurs, whether that's by surrounding yourself with friends and family, going on a relaxing trip, diving into your next project, rewarding yourself with a post-defense gift, or even talking with a professional if need be.

Now that it's been about 4 months since I defended, on a typical day I usually don't remember that I'm a Doctor/Ph.D. unless something or someone reminds me (like when someone refers to me as 'Doctor.')  While I'm very proud of my accomplishment, 'being a Doctor' doesn't define who I am as much as I thought it would. That might change the more time I spend in my new position as an assistant professor, but until that point, I feel I'm just someone that successfully survived the dissertation and made it through the post-defense blahs.

Picture Credits: Beaker Zoloft_Bubble

Tuesday, August 9, 2011

How to drown out noises and concentrate

One of my previous posts noted that I like to use active noise cancellation headphones to help concentrate on writing tasks.What I didn't really emphasize is that even with the noise cancellation technology turned on, the headphones still don't always keep all of the various sounds out, especially if people are talking nearby.

This is what happens when I start listening to the
music playing through my headphones... it's not pretty

With some music playing through the headphones, it's much easier to drown everything out, but then I'm stuck listening to the music, which itself gets distracting since I start listening to the lyrics (I was never one of those people who could fall asleep with the TV on).  I then start tapping my fingers/feet to the music, and pretty soon I'm completely distracted.

Hanson actually created
what is widely considered
to be the first 'White Noise.'
One way around this is to play white noise through the headphones along with the noise cancelling turned on.  White noise can also be used to help fall asleep, meditate, etc.  Here's a very useful YouTube 'video' of 12 hours of just white noise:

However, listening to YouTube requires internet, and sometimes you won't have access to it (like when you're on a plane with your laptop).  If you're somewhat technologically savvy, you can use a special program (or if you're not, use this website) to capture that white noise YouTube video as a (very, very long) .mp3 sound file. Then you can cut it down to ~5 minutes using any sound editing program, and set it to loop on your desktop, laptop, or portable music player.  Voila!  A small, non-internet-based white noise audio file on an infinite loop.

Enjoy!  (And remember to turn off the noise cancelling option when you're done with it!  On a related note, the dollar store is great for finding cheap packs of AAA batteries that you will no doubt burn through fairly quickly with the headphones.)

Picture Credits: Headphones Hanson Jay_and_Silent_Bob

Monday, August 1, 2011

How to review a journal article

One of the things I've had to do recently is to write anonymous reviews for manuscripts that are submitted to journals. It was something that I never received any real training on, and I had to pick up from looking at the reviews of my own manuscripts I had received previously, chatting with colleagues, and simply thinking about the things I would want a reviewer to tell me.

Manuscript... REJECTED!
Ha ha ha! Huzzah for anonymity!
One of my friends who is still in grad school just received a request to review a manuscript for an article, and asked if I had any advice on how to do that. I figured this would be a great topic for a blog post. So without further ado, here is my (relatively new, fairly untested) advice about how to review a manuscript for a journal. Remember that this is what I find works for me as an anonymous review, and what I like to see in a review. Your mileage may vary.

1. Start by putting the manuscript into whatever format you typically read journal articles.
I find it much easier to critique an article for a manuscript when I approach it as if I was reading a regular article already published in a journal. When I read most of my regular journal articles, I download the PDF, print it out, and put it in my 3-ring binder of things I need to read. I can then take the binder to the couch, on the porch, on the bus, or wherever I decide to get some reading done, rather than being tethered to my computer.

Yes, I realize that my
reviewing method is indirectly
responsible for the death of
many trees.
This also makes it easier for me to read the manuscript and think about its contents, rather than being distracted by its odd format, as I would be if I tried reading it on the computer in the format originally sent to me by the editor (typically a MS Word or Adobe PDF). I can also make notes much easier on the printed out pages than I can using the MS Word or PDF electronic comments and markups, and it's easier for me to review my comments later on when I write the summary (more about this later in the post).

2. Relatedly, read the article as if it was already published.
Read the article as if it was coming out in the latest issue of the journal. Don't cut it any slack because it's in 'draft' form. As you read, make marks in the margins, on the text, or in a separate list about things you could add into your review.

For instance:

  • What do you agree with? What did the author do well?
  • What do you disagree with? Note that this is not whether you like or don't like the outcome of the results. Instead, do you disagree with things like:
    • The way the literature review was setup
    • The type of analytical model used
    • The statistics reported or not reported in the tables
    • The interpretation of results
    • The link to policy/practice 
  • Is the literature review fairly comprehensive and does it set up the analyses and discussion well? Or, instead, is it missing a key piece of the literature, key citations in the field, or does it read as a very lopsided argument, ignoring any prior literature that contradicts the theory/hypotheses being tested by the authors.
  • Related to the literature review, does it misinterpret results from prior studies? Does it say that study X found Y, when it really found Z?
  • Does the study have a 'consistent read,' or does it instead read like two (or more) people wrote different pieces independently and just stitched them together at the end?

Dear editor: Me strongly disagree
with reviewer two, who say me writing
read like two people be 'stitched together.'
Me feel much unhappy and me want to
squeeze reviewer two's neck until dead.
P.S. Fire bad.

3. Do you spot any errors?
Do you see anything that obviously needs to be fixed? Any typos, formatting errors, missing table labels, citations that you can't find in the references section, etc. Regarding the citations, you don't obviously have to check every one as that will be spotted in the proofing stage, but if a study sticks out in the text and you can't find the citation in the reference section, note that. If the reference section is obviously missing something important based on the journal's style (DOI's where required, first initial/names, etc.), note that.

4. Are there any citations that seem fishy? 
Why yes I did attend the
1932 presentation I cited
in my manuscript
One of my pet peeves when reviewing manuscripts is when the authors cite a conference presentation that's more than a few years old. First, I think "Were they actually THERE, or are they just citing the abstract they found online?" I then think "If this presentation was so important, why hasn't it been turned into a paper yet?" Either way, if you're citing a presentation that's more than, say, 3 years old, you probably should either find the resulting paper or find a different source, in my opinion.

If you see 12 different citations for
studies by Manatee and colleagues,
you're probably reviewing a paper
by Manatee and colleagues...
Do the authors cite websites that aren't scholarly (e.g., Wikipedia, Urban Dictionary, Blog Posts, etc.), but cite them as if they are scholarly sources of information? Do you notice any 'unpublished' citations (if so, they probably shouldn't be citing them unless those manuscripts are actually 'in press').

If the discussion and
conclusion sections sound
like these guys wrote them,
they probably don't deserve
a revise and resubmit..
5. Do you feel the article ties the lit review/analyses/etc. all together into a coherent final discussion and conclusions? 
When you finish the manuscript, does it make you go "Ah ha! That's the point!" or does it make you shrug your shoulders and go "So what? Who cares?" If the authors don't do a good job setting up the front end of the article, then they're probably going to run into a problem at the back end when trying to tie everything together into a coherent point. If you can see a glimmer of hope in the article, and the authors simply didn't do a great job at getting across the 'so what' about the article, then you should note that in requesting a rewrite of the discussion/conclusions. Relatedly, does the author miss a big piece of the results? Do they downplay an important finding that should be highlighted? Note these in the review, explaining exactly what you think the author missed or should otherwise correct, and why they should do this.

6. Overall, does it seem like the manuscript is a good match for the journal you're reviewing for? 
For instance, if the journal is focused on linking research to practice and policy (e.g., Justice Quarterly, Criminology and Public Policy), does the manuscript talk about implications for policy and practice? If it doesn't, and the main takeaway doesn't seems to have a clear implication for policy and practice (even after a revision), then perhaps the authors should instead submit to a less policy-focused journal (which obviously means they shouldn't get an R&R at the current journal...). The same goes for journals focused on quantitative research methods (Journal of Quantitative Criminology), theoretical aspects (Theoretical Criminology), Policing, or other specializations. If the manuscript is simply not a good fit for the journal, then you should note what journal might be a better fit.

Interesting conclusion,
Dr. Crazy-Sweater...
Revise and Resubmit.

7. Is the paper written in a neutral way, or does it come across as an advocacy piece?
If the paper is meant to be a scholarly research manuscript, it probably shouldn't read like an advocacy piece. For example, if the manuscript examines the relationship between religiosity and crime/deviance, it shouldn't read like something that could have been written by either the Vatican or the Freedom from Religion Foundation. If it does, then it probably needs to be scaled back in terms of tone so as to come across as more neutral. Note that this doesn't apply to all scholarly articles, as some forums (like the response essays in Criminology and Public Policy) are geared more for discussing certain viewpoints or implications related to the findings of the research manuscript.

Unless this guy wrote
the paper, there are
probably at least some
limitations to mention.
8. Does the author note the important limitations of the study?
All studies should mention important limitations. For example, if the study is about a single county, the generalizability of the study should be one of its limitations. If the manuscript doesn't mention this limitation, then it should. Related to this, does the study stretch the limits of generalizability? Does the one-county-one-state study try to pass off results as valid across the entire US or even the entire state? If so, something's wrong that needs more fixing than just adding something in the limitations section.

9. Time to make the master list.
Once you have all of the things noted on the manuscript (or otherwise written down) that you think are either 'wrong' with the article or need some type of revision, and at least a few things that are well done, you are almost ready to write your actual review.

I like to go back through the manuscript and make a final list of everything I didn't like, want changed, or liked, and order it by page in the manuscript. I like this method, because it lets the author more easily address comments and change requests, as they'll be able to quickly find what the review is talking about in each point.

While Smith and Fluffy (2011)
raise interesting points, I believe
they should give less emphasis
to the government mind-ray
theory in their manuscript.

If there are a number of things that all fall under the same category (e.g., a phrase you think should be changed throughout the manuscript, a misinterpretation of results, a typo that keeps popping up, etc.), then simply note the first occurrence of it on your list and indicate that this is a problem in other places as well.

10. Time to summarize everything.
After you have your final list with all of the things you're going to note to the authors, you now should write a few paragraphs (maybe a page or so) about your overall impressions of the article. A lot of the reviews I've seen begin with a short (2-3 sentence) summary of the article: what was the research question, theory, findings, and conclusions? If you choose to do this, great, but the 'meat' of the review comes after this.

While perhaps a correct
review, try for something
a little more scholarly.
When you write up the summary of what you think of the article, you should remember to emphasize what was really well done, and not just focus on what the major problems/fixes. Remember the complement sandwich! If something needs a little more research, feel free to suggest citations to use, but if you keep suggesting the author incorporate citations by "[Your Name] and Colleagues," you're unlikely to remain an anonymous reviewer for long.

The summary, in my opinion, should read as if you read a friend's dissertation and are giving the most important feedback up front to them. Don't mention every minor typo on your list (the list will speak for itself later), but if there are a bunch of typos throughout, mention that the author needs to do some major proofreading in the next draft.

You don't want to come across
as "the special reviewer"
Make sure that your summary comes across fairly neutral in tone, especially if your comments are geared towards the negative. You don't want to blow the author out of the water with nastiness, nor do you want to be kissing their ass. Praise the manuscript where you feel it's appropriate, and take the author to task for things that are egregious errors, but make it a fairly evenhanded summary.

The final paragraph of summary (before you provide the list of changes/notes/etc.) should indicate whether you feel the article is good enough to publish (pending a minor R&R of course), or if you instead feel that it's not an appropriate manuscript for the journal. If you think it's not a good fit for this journal, but a perfect match for another one, then say so.

11. Making anonymous comments to the editor.
You will also probably have a chance to write directly to the editor about the manuscript. Your comments in this special section will remain anonymous to the author of the manuscript. You should end up with perhaps a couple paragraphs of private comments to the editor that state what you really feel about the article, but don't feel comfortable saying to the authors. For example, if the article reads like an undergraduate student wrote the article, gave it to a professor to quickly read through and sprinkle in citations to his own work, and then shipped it off to the journal, egregious spelling mistakes and all, then say so. If the study is one of the best you've ever read on the topic, and that you think the study should be published with very few corrections, then say so. If you can't be honest when writing directly to the editor, when the author won't see your comments, then you probably need to reconsider reviewing the manuscript.

This is what happens when
you don't proofread things...
12. Proofread your damn review!
One last thing you should do is to proofread your entire review. You should be reading it both for grammar/spelling mistakes (especially if you call out the author for making spelling errors in their manuscript), and also to ensure you don't come across as either a complete jerk or as a clueless reviewer. You can say a manuscript shouldn't be accepted and still give the author some good feedback and a gentle letdown. Alternatively, you can say the manuscript should be published and is a great example of scholarly work, while still providing insightful feedback to incorporate before the final acceptance. If you read your own review and think "I wish I could get reviews (negative or positive) that read like this!" then you've done a good job.

Addendum: My friend has this bit of advice to add (Thanks Dr. Liz!): 
No matter how terrible the paper you review, tone down negative criticism. Avoid inflammatory language or personal attacks. Don't use exclamation points. Also, reviewing papers is an amazing learning experience about a) what journals ask reviewers to comment on and what kind of scale - from reject to accept - they use and b) what a good paper "needs" (this can help your own writing).

You mean I can review things
for you, for free, and not receive
any credit other than a line on my
C.V. that won't really help me get
a job or tenure? Hell yes I'd be
Postscript: How to become a reviewer for a journal
When I first started my PhD program, I wasn't really sure how people became eligible to review for journals. Sometimes, once you have an article accepted by a journal, you might automatically be added to their 'list' of potential reviewers. However, if you want to review for a journal you haven't been accepted at before, you can simply send the journal editor a short email explaining your interest in reviewing for them, and what your area(s) or expertise are. They will likely be happy to have you volunteer, and you might receive something to review within a few weeks (although don't hold your breath, it all depends on what is submitted to the journal.)

Final Note: A good link from a major journal publisher explaining what they are looking for in a journal review can be found here.

Do you have any other tips for reviewing manuscripts?  If so, add them below in the comments or send me a message!
Photo Credits: Bag_Head Clearing Frankenstein Old_Man Thoughtful_Manatee Dumb_and_Dumber Protester Mr._Perfect Tinfoil Proofreading You_Suck Owls Phone_Guy (Yes, I'm addicted to this photo...)