On Wiser People and Doubts

"The whole problem with the world is that fools and fanatics are always so certain of themselves, but wiser people so full of doubts." 
- Bertrand Russell

This is probably one of my most favorite quotes of all time.  It also, I think, accurately reflects much of what happens when you have earned (or are in the process of earning) a Ph.D.

The Ph.D. process is meant to make you both a jack of some trades and a master of one.  That is, you are expected to have a general knowledge about an entire field (basic as that knowledge may be), but also to become one of the (if not THE) worldwide experts on a little, tiny bit of subject matter within your field.

For instance: Because of my dissertation work, I am probably now one of the top researchers on the one specific type of policy that my dissertation examined, at least to the handful of academics that actually care about said policy.

...You have no idea how many times I revised that last sentence because I thought it was stretching the truth, changing, adding, or deleting phrases such as "top five", "top ten", "maybe" "definitely", "probably", "in America", "I can state with confidence", etc. etc. etc.).  In the end, I still pussed-out and added the "at least to..." clause to the sentence. That, right there, is part of the problem.

"I am an expert on my own dissertation topic!!!"

That is, this provides an excellent example of what Bertrand Russell was talking about.  The more knowledge you have about something, the less confident you feel about your abilities. (Conversely, 'fools and fanatics' are absolutely certain of their talent and abilities.)

This feeling is referred to as impostor syndrome, and I've also seen it referred to as the 'phony police' (as in, the phony police are going to come and arrest me for being a phony scholar).  Even writing a sentence on my own personal blog about something that is probably more true than not (that I'm an expert on my own damn dissertation topic) requires me to put in tons of qualifiers until I feel comfortable typing it.  In truth, the sentence as is still makes me feel a little uncomfortable, but as Bill O'Reilly said: "F*ck it!" (video link, nsfw language).

He made it through grad school
...without experiencing any
feelings of inadequacy.

Now I'm sure there are people who finish their dissertations and never experience any self doubts about their abilities as a scholar.  I've just never met any of those people, and like the majestic Sasquatch, the only evidence that they exist are blurry photos and television shows.  (Side note: Those people are typically referred to as 'assholes').  I'm also willing to bet that feelings of inadequacy/'faking it' are more common among Ph.D.s. (and especially Ph.D. students) than not.  This is one reason why I tell my friends in grad school that it takes at least three years to realize you're not a complete idiot.  (You still feel like a phony, just not so much of an idiotic one).  As such, it seems that it's normal to think of yourself as 'faking it', which really isn't of much consolation.

Giving a Hammer to a Ph.D.
On a related note, in researching for this post, I came across an article on how to spot a 'fake' Ph.D. (that is, someone who got their degree from a diploma mill).  One thing in the article jumped out at me: "Give a hammer to a phony Ph.D. and he'll treat everything he sees as if it is a nail. Trained minds don't do this."  I get what the author is saying: Fake Ph.D.s think they know everything about anything, and their research findings are more important (and more generalizable) than they really are.

However, I think the reverse is true as well: "Give a hammer to a 'real' Ph.D. and he'll realize that his hammer is only good on certain nails, used on certain boards, under certain environmental conditions, which will never actually be in the same location at the same time. Sane minds don't do this."  

That is, during the Ph.D. process you're taught to cite anything you say that could possibly be attributed to someone else.  You have to qualify statements with phrases such as "The evidence suggests that...", and "While not confirmatory, it appears that..."   As such, I think you internalize the idea that you aren't an original thinker, and are not a 'real' scholar who is able to be an 'expert' on something. (Or maybe not.  Again, who the f*ck am I?)

Constant Judging as Part of the Process

"I've seen better
theoretical models
selling toothpaste
on TV!  Hrumph!"

It also doesn't help that every aspect of obtaining your Ph.D. degree
(classes, comps, dissertation) involves being judged by 'senior' members of the field.  Even after you become a Ph.D., most of your work is subject to peer-review from anonymous 'experts' in your field (and given the low acceptance rates from top journals, is subject to frequent rejection from those same experts).   The entire process of getting a Ph.D. and being an academic involves multiple and repeated cases of being judged by someone with *supposedly* more knowledge than you have, about your work on a topic that you're supposedly an expert scholar in.  (Hence the title of this blog).  All of this leads to a very f*cked up view of your own abilities, and frankly, is extremely exhausting.

Semi-related note: This post also helps shed light onto one of the reasons my posting schedule is infrequent and completely random.  I swing between the extremes of "Hey, I know what I'm talking about!  Maybe I should post something that might help someone in the future!" to the other end of "Post on a academic topic?  Who the f*ck am I?  A nobody who couldn't tell his ass from a hole in the ground!"  Unfortunately, the latter kinds of feelings are (still) more frequent than the former.  I'm still working on getting over this, and I'm not sure I'll ever really be free of these feelings/thoughts. (However, buying and drinking good quality beer seems to help just a little bit).

That's not entirely true. I can, in fact,
tell my ass from a hole in the ground.
[Insert joke about being full of crap]

----------------------------------------------------------------------------------------
Image Credits: Russell, O'Reilly, The Most Interesting Man in the World, Hole

Note: The actual Russell quote is: "The fundamental cause of the trouble is that in the modern world the stupid are cocksure while the intelligent are full of doubt."  It is from Russell's 1933 essay "The Triumph of Stupidity".  However, I think the paraphrased version is easier to understand and more snappy.

How to 'do' research (Part 1)

[Sorry about the long wait time since the last post.  The first-year-on-the-tenure-track mid-semester crunch really slapped the hell out of me these past month or so.]

I was chatting with a new Ph.D. student the other day about their research ideas and (eventual) dissertation topic.  They said that they had plenty of statistical training (they have two master's degrees), but haven't had much training (yet) in the way of research methods.  Midway through the conversation about their research interests and background, they asked me one hell of a question: "How do you actually do research?  Like, how do you find the literature, figure out a research question, write up a literature review, choose a dataset, that kind of stuff?"

I think 'doing research' is closer
to this than anything else...

The answer to that question, I think, is both very complicated and very personal.  I assume that everyone does their research a little differently from anyone else, and it likely depends on whose research methods courses you took in grad school, which articles and books about research you read (and understood/liked), your own personal preferences and habits, the kind of research you're doing, and a whole lot of other things that aren't immediately coming to mind.  So, I don't think it's as simple as "Do X, then Y, then Z, and viola!  Top-tier pub here we come!"

Still, it's a fair (and good) question to ask, especially as a new Ph.D. student whose largely expected to hit the ground running.  While my answer right then wasn't really well thought out, insightful, or likely even coherent, I've had a little more time to think about it.  Thus, this is the way that I typically conduct research (or at least part of it -- I'm sure there will be more to come, hence the 'Part 1' in the title).  It seems to work pretty well for me thus far, so feel free to steal anything that looks like it might help you, since I'm sure I stole most of this from other people along the way.

1. Read the literature for stuff that interests you.

Researching uninteresting stuff
is like watching Golf on TV:
Both make me want to stab my
eyes out with a spoon.

Note the second part of that statement -- 'for stuff that interests me.'  I think one of the biggest mistakes people make as Ph.D. students and new researchers is choosing research topics only because it's what's 'hot' in the field, or because their advisor does it, or because they can be funded through a project to do it, or for any number of other reasons that don't involve being genuinely interested in it. (Yes, there's obviously something to be said for doing research that will eventually get you a good job, but you should be able to balance the marketability of a particular research subject with your interest in it.)

There's a reason I left my stable, fairly well paying office job to earn my Ph.D.: I got to the point where couldn't see myself doing something, as my career, that I didn't absolutely love.  I still feel that way, but now instead of that unloved 'thing' being the 9-5 office job of shuffling papers, it is now conducting research that I really couldn't give two sh*ts about (or even one sh*t about). So I try not to do that on a day-to-day basis, and I think I've been fairly successful at it thus far.

So, I guess the precursor to step 1 would be 'figure out what interests me.'  This likely involves reading a lot of broader literature in your field to find the things that you want to read more about.  Once you've done that, and you know that you're really, really interested in X and Y, go ahead and look up more literature on X and Y and read it.

Keep in mind that this isn't simply reading the latest 3 years of top-tier journal articles that used X and Y as keywords, but also involves figuring out who the big names are (and were) in the field, what the major studies on X and Y are that get cited over and over again are, and what types of methods, data, etc. are generally used in those studies.  You should also read studies that might not be on X or Y, but relate to it somehow (similar methods, similar units of analysis, similar theoretical models, etc.).  All of this reading allows you to talk coherently in your research papers about what's been done before, who has done it, how they did it, and how your stuff is similar.

So, now that you're knee deep in the literature of the stuff that interests you, what's next?  Well, it's something that takes place while you're reading the literature...


2. Figure out what people have been doing poorly, or not at all, in those studies (i.e., find the 'gaps' in the literature).

Similar to the UK warnings,
if you don't mind the gaps in
the literature, you'll be hit by a
metro train called 'peer review'

This really only comes from reading study after study after study.  Invariably you'll be able to spot the limitations of studies that you read, whether it's generalizability, unsophisticated methods, not using Theory A, not studying area B, etc. etc. etc.  Sometimes the authors will do you a solid and list the 'real' limitations of their studies in their respective limitations section.  From what I've seen, however, most of these so-called limitation sections are really just a list of one or two of the most minor problems that barely constitute an actual limitation, with no mention of the more serious limitations of the study.  So read the limitations sections with a grain of salt, but if something keeps popping up over and over again (e.g., 'this study only used cross-tabs and not multiple regression', 'we couldn't study X, so we had to study a proxy of it, Y', 'we could only examine one county of data, so generalizability is a concern', 'historical/longitudinal data were unavailable for X, so this was cross-sectional in design'), then it's likely one of the main gaps in the current literature.  You want to figure out what these gaps are, because they can help you either craft a research question or craft your study's methods section.

For instance, if you find that prior studies have all examined individual states/counties/cities, and no one's compared multiple places, that could be the source of a new study.  Your research question might be as simple as "What are the state/county/city characteristics that influence X", and even though this exact same question might have been asked by multiple prior studies, if your study does something those other studies didn't (e.g., examine multiple places rather than just one), you'll probably be making an important contribution to the literature.  If studies have all ignored Theory A, and you think Theory A could be important in explaining X or Y, then your research question might be "Does Theory A explain the occurrence of X?"

Ok, so you've read (and are continuing to read) the literature on your topic, you've found some things that are missing in the current literature that you might be able to address, and you've incorporated that into your research question.  Now what?

3. Write a good, but concise, literature review. 

You should by now have a handle on who the big names are in your field, what the major studies are, and what the general findings of the literature are on topic X.  Now comes what is, for me, one of the hardest and most boring things: writing the literature review. There are a number of guides on this floating around the interview (see one example here), and I guess now there's one more...

Note that I'm talking not about the 40+ page literature reviews that get published as stand-alone manuscripts in journals. Rather, I mean the shorter, ~5-10 page reviews you see at the front of a 'regular' journal article.

The literature review helps lay the foundation for your study - it talks about what people have done, how they did it, and what they found, but it also connects those findings into a coherent narrative, as well as exposing the gaps that you will be filling with your study. That said, the literature review can vary tremendously depending on how your study is set up. I have most of my experience with quantitative, policy-oriented studies, so that's the type of literature review I'll be discussing. I'm sure this varies for qualitative, historical, theoretical, etc. studies, so buyer beware.

In any case, the point of your literature review should be to give the readers a crash course in what's already been done in the field that relates to your research. Sometimes this will be fairly easy and direct, since a lot of people will have done studies before you on a similar or same issue.

Another research idea:
The effect of access to hoodies
on rates of juvenile delinquency.

For example, suppose you're writing about the relationship between doing poorly in school and juvenile delinquency. If there were a number of studies, or even a specific theory, that can be easily applied to this issue (let's go with Strain Theory, since apparently everybody in my field wants to test this at one time or another), then much of your literature review might be discussing the theory, what it says, and how these studies have supported (or not supported) this theory. So you might want to set up your literature review by first discussing Strain theory generally, and then noting how it easily applies to your specific situation (i.e., doing poorly in school leads to delinquency). Then you can cite prior studies on school achievement and delinquent outcomes, and note whether they support Strain Theory or not. You would note the problems with the prior research, and how more research on this issue is sorely needed (although, really, it isn't in this case... so please stop doing these studies, folks...). This sets up your study by placing it in relation to the prior research and theory, and also helps distinguish it since you've addressed the limitations of prior research.

However, sometimes you're paving new ground, and need to discuss two or three different types of research and blend them together into a coherent story.  For example, maybe you're trying to examine whether access to a cell phone influences juvenile delinquency. It makes sense, since with it you can chat with your friends wherever you are, and you can more easily decide where to meet up to vandalize the school, or burgle the liquor store, etc. Maybe no one has previously looked at this specific issue (if not, I call dibs...). BUT, people HAVE looked at how communicating with friends (in person) can influence delinquency. So, you review this prior research. Then, you realize that socio-economic status (SES) is going to influence who has a cellphone in the first place, and so you need to review the research on how SES can influence delinquency. Now that you've done this, you finish your literature review by pulling together these two prior research fields (communication and delinquency, SES and delinquency), and relate this to juveniles communicating via cellphone, and support why you think this would (or would not) have the same influence on delinquency as face-to-face communication with friends, controlling for SES.

I can't be too much more specific on how to actually organize your literature review, because the organization is going to depend on a number of factors specific to what you're studying, how you're studying it, and what's been done in the past. I recommend reading a lot of articles, mostly from top journals, that have looked at similar things and see how those were set up. This should give you an idea on how to set up your own review section.

Side rant: Tone
Let me take just a minute to talk a little about tone when discussing the prior literature. I'm assuming that the ultimate goal of doing this research will be some type of scholarly journal article that will need to survive the peer review process. If this is the case, then the journal will send your work to other 'experts' in field. While this might include grad students that are just beginning their careers, it might also include some experienced professors whose past research you've cited and discussed in your literature review. How you discussed their research, then, matters a great deal...

Let's consider the following examples:

Example 1: Standing on the shoulders of giants.

Example 1:
"The prior research has been
tremendously important, but
has suffered from some minor
issues which I will address"

Imagine you're one of those distinguished professors reading an article for a journal. The author discusses your article in a neutral fashion, commends certain things that it (you) did well, and even-handedly notes what was missing or otherwise could have been improved upon. The author is acknowledging the contributions of your work, and indirectly suggests that they are 'standing on the shoulders of giants' with their present study. This sets the tone for your thoughts on the article. You think to yourself "Good call, sir! I should have used more than three years of data. I'm glad you're taking the ball and running with it, and glad my research could contribute!" as you sip your brandy and puff on your Meerschaum pipe while sitting by a roaring fire in your study (or where ever/however distinguished professors review journal articles...). The rest of the article is well done, so you send off a positive revise and resubmit recommendation to the editor, with encouraging words for the author.


Example 2: All my predecessors were idiots. 

Example 2:
"The prior work is total
doo-doo done by smelly
caca-heads. My research
is good because it isn't
poop based." 

Now, imagine you're that same distinguished professor reviewing a very similar article. The author cites your work, but does so in a dismissive, critical tone. It's clear that they think your prior research was garbage, and that you were obviously a drooling idiot and/or untrained monkey. While not specifically saying this, they imply that their research can 'save the field' from its past horrific, worthless studies conducted by lesser researchers. "Humph and habberdashery!" you exclaim as you quickly type up your review to the editor with the strong recommendation to reject this horrible, insulting manuscript on the spot. "Little sh*t thinks my research is garbage, does he? Well good luck getting into [top journal of the field], muahahahahaha!" you cackle as you fire off your recommendation to the journal and casually toss the reviewed manuscript into your fireplace.

So kiddos, what did we learn from these examples? Well, first, apparently there are still some people in the world who use the phrase 'habberdashery.' Second, and more importantly, the tone you use to discuss the prior literature can be very important. Yes, you need to point out the gaps in prior studies so you can set up reasons for why your research is an important contribution. However, praising the prior researchers for their contributions, and implying that you're using their research as a basis for yours to help further the field, is a very, very different story compared to one where you're bashing the idiocy of prior researchers to promote your (obviously) much more sophisticated study. You want your manuscript to come across as 'standing on the shoulders of giants,' rather than 'chopping the giants off at the kneecaps.'  So tone matters.

Hopefully what I've written so far will be somewhat helpful to those of you just starting the research process. I'll be posting more on this topic in the future, but this should get you started.

--------------------------------
Image credits: Golf, Hoodies, Old Scholar, Baby Scholar, Mind the Gap, Then a miracle occurs (Harris, 1977)

How to respond to an R&R

[Note: Some of the follow suggestions I picked up from my faculty advisors and colleagues while in grad school, others I came up with on my own, and still others came from Wendy Belcher's excellent book: Writing Your Journal Article in 12 Weeks]

So, you submitted a manuscript to a journal, and the editor liked it enough to give you an R&R... now what?

First, congrats! An R&R is about as good as you can expect for a regular journal, since very few manuscripts are accepted 'as is.'

Depending on the editor's letter and reviewers' comments, it might be an easy R&R that won't take much time, or a very hard R&R that will take weeks or months to address. Also, depending on the journal, an R&R might mean the manuscript is basically accepted, pending a little polishing, or alternatively that there's a good chance you'll still get rejected later on in the process. In any case, your goal should be to address as many of the recommended changes as is reasonably possible in the revision to make the paper better, without accidentally 'messing up' your paper (or your main points) in the process. Hopefully this post helps guide you there.

So the first thing you should do is read the editor's letter in its entirely (you probably opened the letter/email and just skimmed the page(s) until you saw 'revise and resubmit'...).  The editor's job is to read the reviews, make a decision on your manuscript (in this case, R&R), and then (in my opinion) give you guidance on what are the most important things to address in the revised manuscript. If the editor didn't do that, or gave you a generic 'please address the reviewers' comments in your revised submission,' then your job will be a little harder than it probably should be.

Anyhow, read the editor's letter and see if s/he gave you specific instructions as to what to address. The editor might note the important of a missing variable in your model, as mentioned by reviewer 2, or that your treatment of the prior literature was lacking in X area, as noted by reviewers 1 and 3. Whatever the editor says in their letter regarding potential changes should be your first priority in the revision. What I like to do is open a new word document and make a bulleted list of the changes the editor specifically mentioned in the letter (these bullets go under an 'Editor' heading). This will become part of your 'potential revision' list/spreadsheet that you will continue building in the future. (More about this in a bit).

Next, read through all of your reviewers' comments, without taking any notes. Make sure you have your submitted manuscript draft in front of you, so you can reference it when a reviewer mentions something on page X or in Table Y. Keep an eye out for comments that come up from more than one reviewer, such as the suggestion to use robust standard errors in your model, to provide a correlation matrix table, or to cite the work of authors A and B.

Chances are, you won't agree with everything the reviewers recommend, and you'll form the following opinions about the reviewers [note: for ease of comparisons, all three reviewers are assumed to be male in this example]:

Reviewer 1: Even my
bowel movements are
high impact.

Reviewer 1: You'll probably think this reviewer is a genius, particularly since he gave you the most useful recommendations, and other than a couple minor suggestions, loved your entire manuscript. His suggestions are relevant, fairly easy to implement, and will really help boost the quality of your manuscript. This reviewer, while rarely seen in the wild, is the best you can hope for.

Reviewer 2: "Derp?"

Reviewer 2: You'll probably think the second reviewer is an idiot, but a fairly harmless one. He probably doesn't really know anything about either your analytical methods or the literature related to your topic, but tried to fake it so the editor didn't get angry. Most of the suggestions he offered aren't very useful and/or make no sense given your model, argument, and findings. Still, you can easily address his main concerns, and easily dismiss his unhelpful and misguided recommendations.

Reviewer 3: My comb-over
is horrible, I can't eat sweets
due to my diabeetus, and I'm
going to take my anger out
on you and your manuscript.

Reviewer 3: Upon reading reviewer 3's comments, your first thought will probably be "what a f*cking d*ckhead." You'll probably think that every suggestion by reviewer 3 is complete garbage, that he misunderstood your entire argument, knows nothing of the methods you used, and decided to use his "anonymous" review as an outlet for his deep-seated inner frustrations concerning sexual impotency and male pattern baldness. You'll likely question his academic and genetic pedigree, and wonder how the editor could have chosen such an arrogant jerk. This reaction is completely normal, and has a long historical precedent.



In fact, Einstein even developed a theory related to reviewer comments that largely supports these three general categories, as previously noted in my post on how to get over manuscript rejections:

More useful than relativity.

However, these categories are not exhaustive, nor are they exclusive. You may, for instance, get a review from a reviewer that combines traits of 2 (idiot) and 3 (jerk).

 Regardless of the types of reviewers you get, or the content of their reviews, the fact that you received an R&R meant the editor saw some promise in your manuscript. Therefore, now that you've surveyed all of the reviews, you need to get to work addressing the specific suggestions and critiques, and revising your manuscript for resubmission.

So the next step is to go back through the reviews and add to an excel spreadsheet (or equivalent) every specific suggestion or critique, from each reviewer, that requires you to change something (or might require you to say why you didn't change something.) Each suggestion/comment should be its own entry on a separate line. I recommend five columns in the spreadsheet: ID # (increments up), Reviewer (who made the comment), Comment (what was the change/critique), Importance (How important does this seem to your overall revised manuscript, or to the editor/reviewer?), and Ease (How long and/or how much work is this going to take to address). I also like to color code the importance and ease cells using the an easy to distinguish color scheme - red for important/hard, yellow for medium/moderate, and green for low/easy. If you set it up like this, your spreadsheet should look something like the following (excluding the earlier comments from the editor):

Once you've finished adding the reviewers' comments to your spreadsheet, add the editor comments you previously listed out if they're not already covered by a reviewer, and mark anything the editor said to do as high importance. If the editor says something a reviewer already said, then add 'editor' to the reviewer cell as a reminder, and mark it as high importance regardless of your original importance rank. If the editor specifically says something is not important to address, then mark it as low importance, and if they say not to do it at all, then change the font style for the associated entry to strikeout.

After everything is down on your spreadsheet, you can get a sense of the overall work that's ahead of you by looking at the importance/ease categories for each entry. A roadmap to these combinations, and what they mean for your revision, is below:

High/Hard: These are the changes that will give you the most headaches. You need to do them (high importance), but they're going to take a long time and generally be a P.I.T.A. (hard to do).  Doing these first will make your life much easier later on, and if they're major changes, some of the less important changes may be obsolete once you do these. If you decide not to do one of these, you need a very good reason why not.

High/Moderate: These are changes that will be a little easier than the High/Hard changes. You still need to do them, and they're going to take you some time, but they won't be absolutely horrible to address.

High/Easy: These are the changes that will make you feel most productive, since you can address these important 'need to do' changes very quickly and without much work.


Medium/Moderate: You should probably do these changes, especially if you have the time.  If you don't do one of these change requests, then you should definitely address why you didn't in your cover letter (more about the cover letter in a minute).


Medium/Easy or Low/Easy: Since these are easy to do, you might as well do them, especially for the ones of medium importance. If you choose not to do these, you should at least acknowledge why you're not doing the medium importance ones, if not the low importance ones as well (since not doing the latter will make you look lazy).

Low/Moderate or Low/High: These might be changes you decide not to do, given their relatively low importance compared to the moderate-to-high amount of work they'd require of you. Again, you may want to address why you didn't do these in your cover letter, but given their low importance, if you miss discussing some of these it probably won't be the end of the world.


Ok, so you have your potential changes all in order, you've ranked their importance and ease, you know which ones you need to address, which you should address, which you can easily address, and which you're not wasting your time with. Now comes the hard part: Make these changes in the manuscript. That's about as specific as I can get for this step, but this is what will take the longest.

Once you're done making (or not making) the various changes, you need to reread your manuscript from start to finish to make sure things still make sense. This is especially true if you had to do some serious changing of text. Edit as necessary.

"Dear Editor: Thank you for
the opportunity to revise
my manuscript. Unfortunately,
as I am not fluent in dumbass, I
have been unable to interpret
Reviewer 3's critiques."

After your revised manuscript is re-polished into a final version, you need to draft your cover letter to the editor. I recommend starting the letter with a brief paragraph thanking the editor for the opportunity to revise and resubmit your manuscript for their journal. You should also thank the editor and reviewers for their helpful suggestions and critiques (if even you think most of the suggestions were garbage). Then I recommend stating something along the lines of "I've provided specific responses to each of the suggestions and critiques below. I look forward to hearing your final decision." This will provide a transition between your formal paragraph and the subsequent list of changes that's about to follow, as well as obviate the need to write a closing paragraph after your list.

Skip a line or two, and then list out each change you want to address (all of the high importance ones, most/all of the ones of medium importance, some/most of the low importance ones). Briefly summarize the suggestion or critique, identify who said it, its location in the manuscript, and what you did to address it (or, alternatively, why you did not address it). Examples of wording for this section are provided below:

Reviewers 1 and 2 recommended adding r^2 values to Table 3 (p. 21). This has been done for all three models.

 Reviewer 2 recommended revising the paper to focus on racial threat theory. While this is a valid alternative theoretical focus, I feel that racial threat is not appropriate for the current study given the current focus on non-racial factors, as well as the comments of Reviewer 1 regarding the appropriateness of the current theoretical model. Further, using racial threat would generate the problem of..."

 Reviewer 2 questioned the need for spatial lag measurements in the final model. I have expanded the justification for this measurement on page 4, but believe that the inclusion of this measurement is essential in order to control for the spatial autocorrelation of the observations that was indicated by the Moran's I tests discussed on page 12, footnote 3. As such, this measurement has been left in the final model.

Reviewer 3 recommended deleting the third paragraph on page 2, as it was considered unnecessary. On the contrary, I believe the key points of that paragraph are essential to understanding why the study includes measurements for different types of vehicles as well as the differences in operator characteristics. Without it, the subsequent three paragraphs appear disjointed from the rest of the literature review. As such, this paragraph was left in the article as was originally written.

 Make sure you come across as respectful in your comments, especially when discussing suggestions that you didn't implement, and certainly don't word your responses as attacks on the reviewers' or editor's suggestions. If it's clear that one of the reviewers is clearly confused on a certain point, model, or statistic, spin the wording of your response in such a way so that it seems like it was your fault for not being clear, and then justify your original point/method:

Reviewer 2 seemed confused by my use of the term "spatial proximity" on pages 3 and 6. In order to address this potential point of confusion for readers, I have added footnote 2 on page 3 so as to clarify what this term precisely means for this study. 

When you're done, reread your cover letter to make sure, one more time, that you come across as respectful, attentive, and appreciative of all of the suggestions. When you're satisfied that everything sounds good, send the revised manuscript and cover letter to the editor (or upload them to the journal website, as appropriate), and cross your fingers. With any luck, you'll hear back in a few weeks (or a few months) with good news! If not, then see if you can revise the manuscript any further, and then submit it to a different journal to start the process over again.

---------------------------------------------------------------
Image Credits: The Most Interesting Man in the World, Mr. Bean, Reviewer 3, Einstein, Letter Writer

How to handle a journal rejection

In one of my prior posts I discussed how to write reviews for journals. Sometimes those reviews result in good news from the editor, like a revise and resubmit (R&R) decision. The majority of reviews, though, will probably result in a rejection letter. This is the result of basic math, given that top-tier journals have acceptance rates well below 50%. Having recently experienced two back-to-back article rejections myself (same journal, I might add), I figured it might be useful to talk about how I, personally, handle having my manuscript rejected, as well as some thoughts on how to effectively deal with it.

One of Einstein's more relevant theories

First off, this is typically the timeframe I go through with my rejections, and my actions/thoughts along the way:

Day 1: The editor and reviewers at journal X are complete idiots!! Efffffffff Yooooooooooo reviewers!!!

Day 2: Ugh, I can't read this review again without getting angry.  Efff Yoooo!

Day 3-14: [Do something else]

Day 15: [After rereading the manuscript and making comments on the draft]  Hmmm, maybe my submission wasn't as strong as I thought, and maybe the editor and reviewers are only mildly stupid (Eff Yoo!). Still, they might have brought up a couple things that I might be able to fix.  ...And maybe the manuscript was a little long and did have a few too many appendices.  I'll work on the revision tomorrow.

Day 16-???: [Revise manuscript]

Day ??? + 1: Ok, the manuscript has been revised, it's much better than the previous submission, so now where should I send it?  Obviously I can't resubmit to journal X (eff u), so lemme look into journal X...

******************************

Does this above sound like a good timeline/plan that might help you get over a manuscript rejection?  Did you just get a rejection and are still stewing with righteous anger?  If so, then read my step-by-step instructions below!

First, when you initially get the rejection email or letter, you'll likely feel like this:

'Generalizability Concerns?'
Well Effff Yoooooo!!!

...at least, I typically do.  I bet there are few people who can receive a rejection letter and feel fine about it.

I recommend the very first thing you do is print and read the rejection comments (from the editor and the reviewers, if included) as soon as possible, preferably that same day.  

Next, do one (or more) of the following to the pages you just read:

• Burn them (watch out for smoke detectors)
• Smash them into a ball and punch it with your fist
• Shred it in the paper shredder.  Throw in a blank CD-R at the same time for a feel-good crunching sound
• Tear it to bits and toss it in the air like confetti (make sure to do this in a common area that you don't have to clean up)
• Stomp on it (works better if you first crunch it into a ball), and then kick the smushed ball across the room.
• Use them for gun/archery/whatever target practice (this one is probably the most satisfying)

Tony recently had his manuscript rejected from
the Journal of Drug Control Policy...
They didn't like his theoretical model I guess.

...feel better?  Only a little?  Ok, good enough for now.

Now that you're hopefully feeling a little better, reward yourself with a treat.  You've earned it, so treat yourself to a box of cookies (or a drink, or a steak, whatever).  You might be asking: "But my manuscript just got rejected?!? What did I do to earn a reward?"  Ahh, it's what you didn't do that earned you the reward.  Specifically you (probably) didn't:

• Fire off a snarky email to the editor telling them off
• Punch a hole in your LCD monitor
• Throw your laptop across the room and/or into the face of a coworker
• Curl into a little ball and cry (though it's ok if you did this one... you can still have a cookie)

So, treat yourself with cookies or something.

C is also for Craptastic Reviewers

Ok, now that you're hopefully a little calmer and feeling a little better, this next part is very, very important.  If you're not under an extreme deadline to get the paper published (e.g., you're racing against a competing academic so you don't get scooped), then take a week or two without looking at either the paper, the associated data, or (ESPECIALLY) the rejection email/reviews.  Just work on something else for a week or two.

After a couple weeks, you should be feeling a little more like this:

A little less pissed off... that's good!

...so, still a little hurt, and a little pissed off, but you're getting there.  Now, read through your original submission (the one that got rejected), and try and read it as if you're a reviewer and that is not your own writing.  As soon as you're done reading what you submitted, then read the rejection letter and reviews again.


Having read what you submitted, do the reviews seem a little more legitimate in their critiques? Oh really, your submission was perfect?  THEN READ IT AGAIN.  Ok, did you find something that might be worthy of revising a little?  Good.  So, how do you feel towards the reviewers/editor now?

Notice the smile?  That's the spirit!

Ok, that's good!  So, you probably noticed that your masterpiece of a manuscript had a few places that you could improve.  If so, great!  Get to work revising the manuscript, trying to incorporate into the revised version as many of the valid critiques from the reviewers and/or editor as you feel comfortable with.  Eventually, you'll have a finished revision that you can submit to a different journal, hopefully with a better outcome this time (if not, start back at the beginning of this post).

Regardless of whether your coping strategies are similar or different (or the advice above works for you), you should at least be willing to revise the manuscript and send it somewhere else.  Don't let a single rejection result in tossing the manuscript in the garbage (maybe after 3 rejections, but not after the first...).  Also, you have to keep in mind that any decent journal is going to reject more manuscripts than they accept, so chances are if you send them something else, they won't hold your prior rejection against you, and so you shouldn't hold the prior rejection against them (eventually... give yourself some time to be pissed with them).

-----------------------------------------------------------
Image Credits: Finger Old_Finger Young_Finger Tony Cookie_Monster Einstein

How to review a journal article

One of the things I've had to do recently is to write anonymous reviews for manuscripts that are submitted to journals. It was something that I never received any real training on, and I had to pick up from looking at the reviews of my own manuscripts I had received previously, chatting with colleagues, and simply thinking about the things I would want a reviewer to tell me.

Manuscript... REJECTED!
Ha ha ha! Huzzah for anonymity!

One of my friends who is still in grad school just received a request to review a manuscript for an article, and asked if I had any advice on how to do that. I figured this would be a great topic for a blog post. So without further ado, here is my (relatively new, fairly untested) advice about how to review a manuscript for a journal. Remember that this is what I find works for me as an anonymous review, and what I like to see in a review. Your mileage may vary.





1. Start by putting the manuscript into whatever format you typically read journal articles.
I find it much easier to critique an article for a manuscript when I approach it as if I was reading a regular article already published in a journal. When I read most of my regular journal articles, I download the PDF, print it out, and put it in my 3-ring binder of things I need to read. I can then take the binder to the couch, on the porch, on the bus, or wherever I decide to get some reading done, rather than being tethered to my computer.

Yes, I realize that my
reviewing method is indirectly
responsible for the death of
many trees.

This also makes it easier for me to read the manuscript and think about its contents, rather than being distracted by its odd format, as I would be if I tried reading it on the computer in the format originally sent to me by the editor (typically a MS Word or Adobe PDF). I can also make notes much easier on the printed out pages than I can using the MS Word or PDF electronic comments and markups, and it's easier for me to review my comments later on when I write the summary (more about this later in the post).


2. Relatedly, read the article as if it was already published.
Read the article as if it was coming out in the latest issue of the journal. Don't cut it any slack because it's in 'draft' form. As you read, make marks in the margins, on the text, or in a separate list about things you could add into your review.

For instance:

• What do you agree with? What did the author do well?
• What do you disagree with? Note that this is not whether you like or don't like the outcome of the results. Instead, do you disagree with things like:
• The way the literature review was setup
• The type of analytical model used
• The statistics reported or not reported in the tables
• The interpretation of results
• The link to policy/practice 
• Is the literature review fairly comprehensive and does it set up the analyses and discussion well? Or, instead, is it missing a key piece of the literature, key citations in the field, or does it read as a very lopsided argument, ignoring any prior literature that contradicts the theory/hypotheses being tested by the authors.
• Related to the literature review, does it misinterpret results from prior studies? Does it say that study X found Y, when it really found Z?
• Does the study have a 'consistent read,' or does it instead read like two (or more) people wrote different pieces independently and just stitched them together at the end?

Dear editor: Me strongly disagree
with reviewer two, who say me writing
read like two people be 'stitched together.'
Me feel much unhappy and me want to
squeeze reviewer two's neck until dead.
P.S. Fire bad.

3. Do you spot any errors?
Do you see anything that obviously needs to be fixed? Any typos, formatting errors, missing table labels, citations that you can't find in the references section, etc. Regarding the citations, you don't obviously have to check every one as that will be spotted in the proofing stage, but if a study sticks out in the text and you can't find the citation in the reference section, note that. If the reference section is obviously missing something important based on the journal's style (DOI's where required, first initial/names, etc.), note that.


4. Are there any citations that seem fishy? 

Why yes I did attend the
1932 presentation I cited
in my manuscript

One of my pet peeves when reviewing manuscripts is when the authors cite a conference presentation that's more than a few years old. First, I think "Were they actually THERE, or are they just citing the abstract they found online?" I then think "If this presentation was so important, why hasn't it been turned into a paper yet?" Either way, if you're citing a presentation that's more than, say, 3 years old, you probably should either find the resulting paper or find a different source, in my opinion.

If you see 12 different citations for
studies by Manatee and colleagues,
you're probably reviewing a paper
by Manatee and colleagues...

Do the authors cite websites that aren't scholarly (e.g., Wikipedia, Urban Dictionary, Blog Posts, etc.), but cite them as if they are scholarly sources of information? Do you notice any 'unpublished' citations (if so, they probably shouldn't be citing them unless those manuscripts are actually 'in press').

If the discussion and

conclusion sections sound

like these guys wrote them,

they probably don't deserve

a revise and resubmit..

5. Do you feel the article ties the lit review/analyses/etc. all together into a coherent final discussion and conclusions? 
When you finish the manuscript, does it make you go "Ah ha! That's the point!" or does it make you shrug your shoulders and go "So what? Who cares?" If the authors don't do a good job setting up the front end of the article, then they're probably going to run into a problem at the back end when trying to tie everything together into a coherent point. If you can see a glimmer of hope in the article, and the authors simply didn't do a great job at getting across the 'so what' about the article, then you should note that in requesting a rewrite of the discussion/conclusions. Relatedly, does the author miss a big piece of the results? Do they downplay an important finding that should be highlighted? Note these in the review, explaining exactly what you think the author missed or should otherwise correct, and why they should do this.


6. Overall, does it seem like the manuscript is a good match for the journal you're reviewing for? 
For instance, if the journal is focused on linking research to practice and policy (e.g., Justice Quarterly, Criminology and Public Policy), does the manuscript talk about implications for policy and practice? If it doesn't, and the main takeaway doesn't seems to have a clear implication for policy and practice (even after a revision), then perhaps the authors should instead submit to a less policy-focused journal (which obviously means they shouldn't get an R&R at the current journal...). The same goes for journals focused on quantitative research methods (Journal of Quantitative Criminology), theoretical aspects (Theoretical Criminology), Policing, or other specializations. If the manuscript is simply not a good fit for the journal, then you should note what journal might be a better fit.

Interesting conclusion,
Dr. Crazy-Sweater...
Revise and Resubmit.

7. Is the paper written in a neutral way, or does it come across as an advocacy piece?
If the paper is meant to be a scholarly research manuscript, it probably shouldn't read like an advocacy piece. For example, if the manuscript examines the relationship between religiosity and crime/deviance, it shouldn't read like something that could have been written by either the Vatican or the Freedom from Religion Foundation. If it does, then it probably needs to be scaled back in terms of tone so as to come across as more neutral. Note that this doesn't apply to all scholarly articles, as some forums (like the response essays in Criminology and Public Policy) are geared more for discussing certain viewpoints or implications related to the findings of the research manuscript.

Unless this guy wrote
the paper, there are
probably at least some
limitations to mention.

8. Does the author note the important limitations of the study?
All studies should mention important limitations. For example, if the study is about a single county, the generalizability of the study should be one of its limitations. If the manuscript doesn't mention this limitation, then it should. Related to this, does the study stretch the limits of generalizability? Does the one-county-one-state study try to pass off results as valid across the entire US or even the entire state? If so, something's wrong that needs more fixing than just adding something in the limitations section.


9. Time to make the master list.
Once you have all of the things noted on the manuscript (or otherwise written down) that you think are either 'wrong' with the article or need some type of revision, and at least a few things that are well done, you are almost ready to write your actual review.

I like to go back through the manuscript and make a final list of everything I didn't like, want changed, or liked, and order it by page in the manuscript. I like this method, because it lets the author more easily address comments and change requests, as they'll be able to quickly find what the review is talking about in each point.

While Smith and Fluffy (2011)
raise interesting points, I believe
they should give less emphasis
to the government mind-ray
theory in their manuscript.

If there are a number of things that all fall under the same category (e.g., a phrase you think should be changed throughout the manuscript, a misinterpretation of results, a typo that keeps popping up, etc.), then simply note the first occurrence of it on your list and indicate that this is a problem in other places as well.


10. Time to summarize everything.
After you have your final list with all of the things you're going to note to the authors, you now should write a few paragraphs (maybe a page or so) about your overall impressions of the article. A lot of the reviews I've seen begin with a short (2-3 sentence) summary of the article: what was the research question, theory, findings, and conclusions? If you choose to do this, great, but the 'meat' of the review comes after this.

While perhaps a correct
review, try for something
a little more scholarly.

When you write up the summary of what you think of the article, you should remember to emphasize what was really well done, and not just focus on what the major problems/fixes. Remember the complement sandwich! If something needs a little more research, feel free to suggest citations to use, but if you keep suggesting the author incorporate citations by "[Your Name] and Colleagues," you're unlikely to remain an anonymous reviewer for long.

The summary, in my opinion, should read as if you read a friend's dissertation and are giving the most important feedback up front to them. Don't mention every minor typo on your list (the list will speak for itself later), but if there are a bunch of typos throughout, mention that the author needs to do some major proofreading in the next draft.

You don't want to come across
as "the special reviewer"

Make sure that your summary comes across fairly neutral in tone, especially if your comments are geared towards the negative. You don't want to blow the author out of the water with nastiness, nor do you want to be kissing their ass. Praise the manuscript where you feel it's appropriate, and take the author to task for things that are egregious errors, but make it a fairly evenhanded summary.

The final paragraph of summary (before you provide the list of changes/notes/etc.) should indicate whether you feel the article is good enough to publish (pending a minor R&R of course), or if you instead feel that it's not an appropriate manuscript for the journal. If you think it's not a good fit for this journal, but a perfect match for another one, then say so.


11. Making anonymous comments to the editor.
You will also probably have a chance to write directly to the editor about the manuscript. Your comments in this special section will remain anonymous to the author of the manuscript. You should end up with perhaps a couple paragraphs of private comments to the editor that state what you really feel about the article, but don't feel comfortable saying to the authors. For example, if the article reads like an undergraduate student wrote the article, gave it to a professor to quickly read through and sprinkle in citations to his own work, and then shipped it off to the journal, egregious spelling mistakes and all, then say so. If the study is one of the best you've ever read on the topic, and that you think the study should be published with very few corrections, then say so. If you can't be honest when writing directly to the editor, when the author won't see your comments, then you probably need to reconsider reviewing the manuscript.

This is what happens when
you don't proofread things...

12. Proofread your damn review!
One last thing you should do is to proofread your entire review. You should be reading it both for grammar/spelling mistakes (especially if you call out the author for making spelling errors in their manuscript), and also to ensure you don't come across as either a complete jerk or as a clueless reviewer. You can say a manuscript shouldn't be accepted and still give the author some good feedback and a gentle letdown. Alternatively, you can say the manuscript should be published and is a great example of scholarly work, while still providing insightful feedback to incorporate before the final acceptance. If you read your own review and think "I wish I could get reviews (negative or positive) that read like this!" then you've done a good job.

Addendum: My friend has this bit of advice to add (Thanks Dr. Liz!): 

No matter how terrible the paper you review, tone down negative criticism. Avoid inflammatory language or personal attacks. Don't use exclamation points. Also, reviewing papers is an amazing learning experience about a) what journals ask reviewers to comment on and what kind of scale - from reject to accept - they use and b) what a good paper "needs" (this can help your own writing).

You mean I can review things
for you, for free, and not receive
any credit other than a line on my
C.V. that won't really help me get
a job or tenure? Hell yes I'd be
interested!

Postscript: How to become a reviewer for a journal
When I first started my PhD program, I wasn't really sure how people became eligible to review for journals. Sometimes, once you have an article accepted by a journal, you might automatically be added to their 'list' of potential reviewers. However, if you want to review for a journal you haven't been accepted at before, you can simply send the journal editor a short email explaining your interest in reviewing for them, and what your area(s) or expertise are. They will likely be happy to have you volunteer, and you might receive something to review within a few weeks (although don't hold your breath, it all depends on what is submitted to the journal.)





Final Note: A good link from a major journal publisher explaining what they are looking for in a journal review can be found here.

Do you have any other tips for reviewing manuscripts?  If so, add them below in the comments or send me a message!
---------------------------------------------------------
Photo Credits: Bag_Head Clearing Frankenstein Old_Man Thoughtful_Manatee Dumb_and_Dumber Protester Mr._Perfect Tinfoil Proofreading You_Suck Owls Phone_Guy (Yes, I'm addicted to this photo...)