I was waiting for a topic to come along that would inspire me to write something, and finally one did, so for an inaugural post, here goes. I should add that I think that the title of this article may very well be one that I return to frequently, because, I see a lot of problems in the review process, and I’m not alone in that.
For those who don’t know, the review process typically works as follows. A manuscript is submitted by authors for publication to one chosen journal. Submitting to more than one at a time is taboo. The manuscript is assigned to one of a number of topic-specific editors the journal maintains, who in turn sends out email requests to review the paper, to two or more people considered specialists in the field. When two agree (sometimes three), the review process starts. Typically the reviewers are requested to have their review in within a month, though often they don’t comply with that, because there’s no enforcement of anything here–it’s all completely voluntary. Reviewers are not paid for their time.
With that as background, copied below is the text of a comment I made over at Dynamic Ecology earlier today, on a post dealing with review times, edited slightly. Below it is a copy of a letter I sent to an editor of a journal, as described in the comment.
Two days ago I received a message from the editorial office of a journal that I completed a review for in late July. The message was simply to inform me that the manuscript had been rejected. There was only one other reviewer, and his/her review was included with the message.
My review had been supportive, saying the work was fundamentally sound, a useful addition to the subject area (bias in the taxonomic composition of trees from land surveyors data in the 1800s) and then detailing some weaknesses I thought should be fixed. The manuscript was quite nice–tightly written and nicely integrating both simulation analyses and application of a new technique to a large set of empirical data. My review was a mix of general comments and line by line specific comments.
The other review consisted of two short paragraphs of about 3 sentences each. The reviewer was completely wrong in all comments, and did not even seem to grasp the most basic goal of the paper. It was a complete travesty, just a hodgepodge shotgun assortment of wrong statements and confusion. Pretty clearly, this review had just come in, almost two months after mine, and on its basis the paper was rejected. I should note that this field has been plagued by a number of long-standing analytical problems that both the author of the manuscript and I have been trying to correct for several years now. The nature of these problems is very basic, but I won’t get into them here.
This was something of a straw that broke the back for me in terms of outrage with respect to the review process.
I immediately composed a letter that I sent to the coordinating editor, explaining that the other reviewer’s review was a travesty, and that he/she was either grossly negligent or incompetent. I requested an explanation for exactly why the paper was being rejected, and also that my response letter be passed anonymously to the other reviewer, before I would consider any future reviews for the journal. I made it clear that I was not happy with the review process and that it impacted the likelihood that I myself would submit a future manuscript to the journal. I also made it clear that I did not know the author and had never met him/her and that my concerns were based 100% on the scientific issues at hand. Which is 100% true.
I cc’ed this letter to the author, whom I think was kind of shocked, but pleasantly surprised, and who responded with his/her own letter detailing some of the specifics of why the reviewer was egregiously wrong.
We’ll see what happens, but as for me, I’ve just decided that I’m not going to take this kind of blatant malfeasance anymore. I’ve seen too much of it, too many times. In fact, I’ve decided that I’m going to detail the most egregious of the examples I’ve experienced on my new blog and/or wherever else seems appropriate. If you don’t shine a light on this stuff, too many people just quietly accept it like good little boys and girls, and so the same behavior gets repeated over and over again.
Below is a copy of the letter itself, edited so as to make everything anonymous.
Dear Dr. ______,
Thank you for your message. However, I think you are making a mistake in your decision on [manuscript number], which since I supported generally, may therefore be based upon the review by reviewer #2. If the text you included below constitutes the full review by this reviewer, then it is woefully lacking, both in quality and quantity. It badly misunderstands the nature and importance of the paper, as well as the importance of [Anonymous] (2010), and quite frankly, the field overall. I can detail why this is so should you want to hear it. To add insult to injury, it apparently took two months for this reviewer to write such a short and wrong review, thereby delaying the entire publication process, including the author’s ability for timely submission to another journal. It is not at all encouraging to see this, either as a reviewer, or a potential submitter of future manuscripts myself to the journal.
I want to make it perfectly clear that I do not know Dr. ______ and have never met, or interacted with, him/her in any way. My concerns expressed here are based solely on the quality of the manuscript, the general quality of other studies in this field, and the review by reviewer #2. As I mentioned in my review, this field has been fraught with a number of serious analytical problems for a long time (and still is), and Dr _______ is one of the very few people who has been working to correct these issues using sound analytical approaches to the detection and correction of various types of bias. The paper is fundamentally sound and any decisions not to publish it in [your journal] should therefore be based on other considerations the journal may have, and not on the quality of the work itself. I stated this clearly in my review of the paper, along with weaknesses I felt should be addressed before final acceptance. My review is much more detailed and specific in these respects than is that of reviewer #2.
I will add one last point. Dr.________’s manuscript is based on an approach that I myself took  a few years ago, which I referred to in a publication of mine, but which I never published. Therefore, if I wanted to “shoot down” this paper so that I could publish my analyses first, for purely selfish reasons, I could endeavor to do that. We all know that a reviewer can stop a manuscript from being published if he so chooses, by giving a biased or misleading review. But that is not the way an honest scientific review process should work, and I do not believe that reviewer 2 has made a good faith effort to properly understand this paper. Alternatively, the reviewer is simply incompetent to judge the work. Neither of these are acceptable situations.
I respectfully request that (1) my comments here be passed, anonymously, to reviewer #2 and to the coordinating editor of the review process and (2) that the reasons for the rejection of this manuscript be made clear, before I will consider any future reviews for [your journal].