Thursday, October 21, 2010

Advice for article reviewers: what is best practice?

Readers may be familiar with my "Publishing Advice for Graduate Students" which addressed issues from publishing book reviews and conference proceedings to replies, full length articles, and submitting book contracts successfully. I have been genuinely thrilled by its reception as it struck me that there was a real dearth of helpful advice on the subject available. Students only had to hope for an insighful supervisor to teach them the ropes previously.

I am now beginning work on "How to Peer Review" which will address substantive, practical advice on how to best conduct reviews of journal articles and book proposals. This seems to be the new area where good information is lacking.

A question then for readers: what advice should be offered? All comments will be gratefully acknowledged in the final piece.


Joe Morrison said...

Advice: you really really need to consider saying 'no'.

New reviewers can be kinda flattered by being invited to review something for a journal - it feels like a first step into The Academy, finally doing some service, having your opinion taken seriously, being treated as a peer. And since you're desperate to put something extra on your CV (for all the difference it actually makes = none), it's very tempting to say 'yes' to anything that comes along. But hold up: do you really know what you're talking about. Sure, you're pretty good at following what goes on in seminars that you go along to, and sometimes you notice that you've thought up the same objection that someone else actually asks - you're able to think critically about pretty much anything put in front of you... Is that sufficient? Are you really in a position to weigh in on someone's original research? Are you informed enough?

I'm confident that most decent postgraduates could probably say something slightly useful about something that's perhaps-only-tenuously-related to their research interests. But that doesn't mean that they should do so: peer review, done properly, should involve some perspective on the material - familiarity with the issue, even an angle or opinion about the debate can be useful, but bringing just critical ability isn't enough. Another concern is that the eager reviewer is often too keen to be critical, to score points. That kind of attitude can bring about some misreadings of otherwise straightforward (and even orthodox) claims. So the advice: say no, unless you're certain that you're in the right position to do it properly.

Of course, Tom, you're going to have to explain what 'doing it properly' involves. Perhaps before someone starts peer-reviewing they should have gone through the process a few times, so that they've some experience of what its like to be on the receiving end.

A neat idea to accompany/promote a book on peer-review would be to request and compile a bunch of (instructive, anecdotal) horror-story examples. I'm sure you could do it while maintaining proper anonymity etc. Get a request out, see what you haul in. I've heard loads as after-dinner yarns at conferences, there's a sort of one-upmanship about it as well.

KateNorlock said...

New peer reviewers are not all equally confident about the following, so consider suggesting that they write back to the sometimes overly brief email solicitations to peer review and actually ask the editor for clear parameters if they are lacking. Not all editors have been equally excellent about telling me if this is doubly anonymous, if there's a time frame, if they are looking for constructive advice regarding revision, if they require written comments in detail or just a thumbs-down, etc.

enzo rossi said...

There's a useful discussion of this important topic here:

Ben said...

One, perhaps fairly obvious, thing - remember that being able to think of substantive objections is not necessarily reason to reject a paper (and not being able to think of any isn't reason to accept it!)

Lee said...

The best peer reviews I've received have been concerned to highlight what is original and good about my work and then offer advice on how to strengthen these elements so the articles are the best they can be. The worst peer reviews I've received have clearly been written by people who either don't comprehend or simply don't like the arguments being made; these reviews try to pick holes in the argument in order to weaken the piece and force it to be converted into something more acceptable to the reviewer's intellectual and/or political persuasion.

For me the best piece of advice you could give is: never, ever become the latter type of reviewer. Do this by understanding your role to be one of quality control. The journal editors want you to make sure that only quality, rigorous research is published in their journal. You should not be afraid to recommend refusal of publication if work doesn't meet the required standard of research, evidence, logic, writing, etc. However, the editors do not want you to police the boundaries of the discipline or regulate what arguments can and cannot be aired in print. (If they do, you should not review for them.) To ensure you do this job, instead of illegitimately 'gate-keeping' the discipline, on top of your quality-control role, adopt the attitude of a firm but fair supervisor who has received a draft of work from a student. The good supervisor doesn't try to censor a student; they try to assist the student to make the work as good as it can be. If you adopt this approach to peer review, you will be helpful and supportive, while upholding the quality standards of the journal.

Ben said...

The blogpost imported as facebook note generated the following discussion:

David Barnard-Wills Review what has been written, not what you assume has been written?

Karl Widerquist I haven't gotten much feedback on my reviewing either. So, most of what I know I've learned from being on the other side as well. I go with this general rule: The less you like a paper, the more you should write about it. If you're going to keep them from getting published you should at the very least give them a thorough explanation why.

Ben Saunders Karl, I see some merits in that proposal but I don't think it can be applied literally. Firstly, with a really bad paper it would be rather demanding, when arguably the paper shouldn't have been sent for review at all. More importantly though, your comments aren't only for the author but really for the editor. If you like a paper, you should make clear why it ought to be published, in case the other referee disagrees.

Ben said...

Al Wilson My strategy:
If the paper is hopeless then concentrate on saying so, and why, as quickly as possible.
If it could with suitable changes be worth publishing, then give comments on it for the author as you would on a colleague's work - eg detailed, friendly and constructive. Once you've done that, a brief note to the editors highlighting the most relevant points from your comments for the author and giving an overall assessment should be enough. This takes some time (~3hrs) but I figure it's good karma.

Karl Widerquist The most worthless thing that reviewers do is to summarize the paper. I've seen these as both an editor and a person being reviewed, and from either perspective, I haven't found any value in it. I either wrote or read the paper. Cut to the evaluation & skip the summary.

Ben Saunders I've heard it said that the purpose of summarizing the paper is largely to show that they read and understood it (though it's perhaps also helpful to the editor if s/he hasn't read the paper closely/at all). I'd imagine that if the referee's summary shows blatant misunderstanding - e.g. taking you to be endorsing a position that was actually intended as a reductio - that would be good grounds for appeal to the editor.
Also, sometimes it highlights passages or stages of the argument that aren't clear. And it may be easier for them to have the summary to refer back to in the later critical comments.

Neil said...

Best advice: do it quickly or turn it down. Median review time in medical journals is 29 days. No reason it has to be so bad in philosophy. Medicos are not less busy than us.

KateNorlock said...

Well said, Lee! I've kept a mantra up while reviewing, "Let the author be the author." The aim of a really constructive review is to assess whether or not the author has done a high-quality job of featuring their own held arguments, and suggest further ways they might. I've gotten one too many reviews positing that I ought to change my mind about, e.g., feminism, or justice, which is as frustrating as can be.

(May I just add that my word-verification was "couthro"? I cannot stop picturing a nattily dressed, refined Jethro.)

The Brooks Blog said...

Could someone please add the link to the Facebook note? I'd enjoy seeing it and reading the discussion. Thanks to whomever for helping to further discussion!

Ben said...

Do you mean the facebook post I refer to? It's here:
(Sorry for not being clear what I was referring to; I linked to this post on my blog, which then got imported as my facebook note)

Anca Gheaus said...

There are interesting recent discussions on refereeing here:

and here: