Question of the day: Does the journal change the way you review?

Mar 14 2013 Published by under [Education&Careers]

One thing I have begun to appreciate more and more recently is that reviewing manuscripts is not a "one size fits all" endeavor. The majority of my reviewing is done at the society journal level (IF 2-10), related to my field. Generally this is a straight forward exercise of reporting back the issues one sees with the research described. Are there flaws in the study? Did the authors miss literature that might change their interpretation? Are there additional data (within reason, folks) that might make a borderline case stronger?

But the process is very different when reviewing or being reviewed for the Big Journals. The tone of the review is critical to the editorial decision. I've seen reviewers be very positive about a manuscript, while sneaking in enough "suggestions" that the revisions would be considered "major" by most standards. Nevertheless, these manuscripts make it through the gauntlet. I've watched as reviewers with little negative to say have framed their review in a manner that has been a death knell for manuscripts. There really seems to be a reviewer-fu art to selling a manuscript to the editor while still making significant changes to the finished paper. In retrospect it makes total sense given the scope of some journals, but I've no one ever mentioned to me as an author or reviewer.

So, Dear Readers, do you change the way you review a manuscript based on the journal, or does the way in which the manuscript is written change the perspective you use to review it?

5 responses so far

  • LD says:

    I have found that if you are reviewing and like a ms submitted to a Big journal, you need to sell it to the editors. This can take considerable effort, but usually works. Same, of course, is true for grant proposals when serving on a panel. But in answer to the bigger question, yes, I do review differently depending on the journal, because after all, the audience is different and different journals serve different purposes.

  • newProf says:

    The journal affects my review enormously. As you say, enthusiasm becomes increasingly important as you move up in IF; it's not enough for the study to be correct or even elegant, it must also overturn something or surprise me. Especially once you're dealing with nonacademic editors or editors who are too broad/famous to know a subfield, it becomes very, very important to place the work in context and explain why it's important and exciting or important but fairly obvious and belongs someplace else. I remember my grad adviser once pointing out that although a manuscript I was reviewing wasn't necessarily the best study on the subject, I might want to consider the impact for the subfield of having a study in a big journal. All sorts of dynamics are at play.

  • Terry says:

    I will identify the same major issues and concerns regardless of the journal.

    However, my recommendation about whether these shortcomings are permissible varies with the venue.

    I attempt to be as factual as possible when stating the strengths and weaknesses. Kind of like reviewing for PLoS ONE, I just try to say whether it's done well and whether it's overstating its claims. I'm sure that I must unconsciously use language or hints one way or another, but I try not to. I'm just evaluating if it's done well and right.

    Even if it's a very low-tier journal, I'll say that it's okay, but here is a fancier analysis - or design - that you could choose to do (which might be required for a higher tier).

    Then, in a separate part of the review, I address relevance to the journal.

    If I see a manuscript more than once, this spares me the trouble of rewriting a whole review.

  • Confounding says:

    Yes, but it has nothing to do with Impact Factor or other percieved ranking.

  • qaz says:

    Of course.

    My job as reviewer is to (1) provide a check to determine if the authors have successfully substantiated their claims [provided the right controls, not made a math error, made the paper as strong as it could be (within reason), etc.], and (2) to make a recommendation to the journal as to whether the paper is appropriate for the journal. As Terry says above, it is important to keep these aspects separate, so that one can say "There's nothing technically wrong with this paper, but it's not particularly surprising." or "If the author's claims are correct, this would rock the field to its core. However, without the additional control X, the authors have not substantiated their claims. The authors should be given the opportunity to do control X."

    I try to keep part 1 (correctness) the same across all journals. I don't want to pollute the scientific literature with incorrect results. However, part 2 (appropriateness) depends on the journal. "Appropriateness" does include some aspect of "should the journal spend pages on this paper" which involves some correlation to journal impact/importance, but it also includes whether the journal is the right one for the paper. For example, I would probably say that the new report from the LHC that the Higgs boson really is a Higgs boson is not really appropriate for Psychopharmacology.

Leave a Reply