Discussing reviews of scientific manuscripts: where?

I’ve had my doubts about why I’ve bothered to spend time here discussing the serious problems in dendroclimatology, including getting into the specifics of the review of my PNAS manuscript. Why should I bother? I do wish at this point that I’d re-submitted it elsewhere sooner, and time spent writing articles here could have gone toward that end, but that’s not the main point here. The main point is this: there’s no convenient place for scientists to easily and openly discuss problems in science, and especially, problems in the review process in science. This is a big problem in my view.

I’ll focus on the latter issue. If you talk to seemingly almost anybody in science, you’ll hear a story, often more than one, about a bad (often atrocious) review they experienced, wherein reviewers rejected a manuscript for illegitimate or very questionable reasons. Those reasons vary, but often the reviewers just simply didn’t understand the paper in question, because they either didn’t spend the time to read it carefully, or didn’t really understand the topic area as well as they needed to. Sometimes the reason given for rejection is just frankly a convenient wrapper, such as “author(s) do/does not provide an adequate defense for assertions” or “authors’ tone reveals an underlying bias and is not acceptable”, or similar vague, general and/or unsupportable or irrelevant statements.

That’s a big problem just in itself of course. But what’s just as interesting to me is what happens next in the process. Interesting, as in, symptomatic of a deeper problem. I’m guessing that in at least 99.99% of all cases, authors of rejected papers simply move on to another journal and re-submit it there. What else can you do, you want to get it published somewhere, right? The problem is that this course of action leaves entirely unaddressed the quality and efficiency of the peer review process; it just sweeps those problems under the rug. For whatever reason, there has never really been an outlet for discussing reviews of manuscripts. Such discussions are universally “underground”–the stories you hear when you talk to individuals at any venue where the self-defined rules of “proper” scientific discourse cannot be enforced and people are thus free to speak their minds openly. Well that situation rubs me the wrong way; I don’t like the thought of people having to discuss things underground that ought to be discussed above-ground, nor the idea of things being swept under rugs and ignored. At all.

Like many things in life, it’s hard to know why this situation exists, which likely entails knowing how it originated. One issue here is the potential for anonymous retribution, and this one is big indeed. If you openly criticize somebody by exposing the quality of their review, they could retaliate in any number of ways in the future (future manuscripts (including the same one when you re-submit elsewhere), grants, and job applications to name three big ones). This problem is severely exacerbated by the one-way anonymity at most journals: you as a manuscript author have no idea who the reviewer is you are publicly criticizing, but they most certainly know who you are. That is a BAD situation to be in! Two-way anonymity is better, and some journals have that, but it’s not an ideal remedy either, because the authors can often be inferred based on the topic, writing style and reference list, whereas the reviewers can remain perfectly anonymous.

There are cases of good open discussions in the scientific literature, for example Climate of the Past, where the reviewers are not anonymous and the full exchange between authors and reviewers during the review process is published along with the paper itself (as a “Discussion” paper). So, this is a big improvement over the typical situation, especially the identification of all involved. But even it only applies to papers that actually get published, so it’s still a limited solution, [NOTE: No, I’m wrong on that, see the comment with links by Paul S in the comments section] and I know of no outlet devoted solely to detailed discussions of reviews of submitted papers, accepted or not. What we’re left with as options then, are those public fora easily accessible to both writers and readers, websites (especially “blogs”) being the best example, circa 2013. They’re easy to start, they’re open, they’re instantaneous to publish in, and readers can respond with comments; ideal in all those respects. In light of these things that’s where, at least for now, I’ll discuss problems in tree ring analyses and the manuscript review process. Because, for both, I want the public to understand these issues, and for the latter, there’s simply no place else to do it.

And please don’t quote me the oft-heard response “Yes, but the peer review process works well in the long run”, because that doesn’t address the great inefficiencies in the process, nor the frustration and career effects on individual scientists on whose backs the short run is sacrificed for this supposedly sound “long run”. Nor is the claim particularly correct even aside from those things, given how much bad work does get published, among other issues.

10 thoughts on “Discussing reviews of scientific manuscripts: where?

  1. I couldn’t agree more. I just received a pretty terrible review, in terms of quality, and had the paper rejected. When I countered this to the editor, his reply was that this particular reviewer was “long-term and experienced reviewer for [journal], [and] took a strong position” that he couldn’t ignore. The problem is that it was a position not based on the evidence presented.

    It seems that many who complain are just written off – “tough – we ALL get rejected, so quit whining” rather than trying to improve the process (or at least start discussing about how to improve the process).

    • Definitely sorry to hear about it Alex, but glad to hear you said something about it. I have absolutely zero sympathy for the “tough cookies, stop whining” response, which is nothing more than a cop-out and acquiescence to the status quo IMO. Humans, I have learned over time, are very good at that, and very good at denying that that is in fact what they are doing.

  2. …for example Climate of the Past, where the reviewers are not anonymous and the full exchange between authors and reviewers during the review process is published along with the paper itself… But even it only applies to papers that actually get published…

    I’m pretty sure that’s not the case. At CPD there is an Open Discussion page which lists dozens of papers which have been submitted and anyone is able to post a comment/review on them. None of these papers have yet been published but you can see all the review correspondence at this stage. I think there is even an archive for rejected papers, along with all the open review comments and replies. There’s some you can see in their Most Commented Papers list that are marked ‘Publication in CP not foreseen’, but all the interactive discussion is still available for these.

    As I understand it though, there is an issue that the open discussion represents only the first round of review. After that the editor will put the paper out for anonymous review, basically the same as in standard journals.

    • OK, very good, thanks for the correction Paul. I should have checked on it before stating what I did–was going by recollection, which was wrong. Climate of the Past is even further ahead of the curve than I realized and that’s a good thing indeed, although not fully where we need to be either, as you note at the end.

  3. One of the problems in the days of yore, when I reviewed and was reviewed in a number of international journals, is that only one editor ever took the time to explain what a reviewer’s job was – and what is was not. He spelt out what he expected to see in a review, what was acceptable and constructive and what was not. He did so publically in the front pages of the journal every year or so. Reviwers had to agree to his terms or decline to review. The editor checked regularly that his terms and conditions were adhered to. This ensured that all submitters and all reviewers were singing from the same song sheet. Regrettably his successor stopped doing so.

    Do any journals do this today? If publishers and scientific organisations took the time to do so it may help reduce some of the present angst. I am well aware that far too many reviewers seize their opportunities to blow-off perceived competitors be it for presitge/standing, grants and/or future positions.

    • Excellent story K.A.–that would help things enormously if applied science-wide. It raises the point that there’s just far too much personal freedom to concoct any and all possible objections to a paper. Many people do whatever they can get away with, until forced not to. Same old story with humans.

      In answer to your question, I don’t know of any, but would like to know as well. But until anonymity in the review process is done away with completely, I don’t see any substantial reform to the system overall. Give people a chance to duck full responsibility for what they say, and you can bet there will be a whole number who will.

  4. Jim:

    I picked up the trail of your dendro posts from the Warren Pearce article. I am a total layman and can’t begin to do the math involved, but appreciate the opportunity to read this type of material and try to wrap my head around some of these issues. Your explanation of the conceptual problems involved is among the clearest I have ever seen. Please don’t give up.

    On the issue regarding the article for PNAS, given the politically charged nature of the discussion which surrounds the climatological record, frankly I think I would have been more surprised if the paper had been accepted. Essentially, if I have understood the issues correctly, what you have posited requires that much of what has been written on the subject essentially to be dropped in the dust bin. That position will not easily see the light of day in a refereed journal, where it is almost certain that the referees in question will have invested so much in the ICS/RCS methodologies and attendant outcomes.

    Patience, thick skin and a willingness to weather rejection is going to be required if success is ultimately to grace your efforts. Best of luck!

    • Thanks for the good word Ian. I am very glad to hear that my explanations were helpful to you, because that’s the whole point here.

      Don’t worry about the math/stats–they merely provide the quantification of the verbal concepts described.

Have at it

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s