I’ve had my doubts about why I’ve bothered to spend time here discussing the serious problems in dendroclimatology, including getting into the specifics of the review of my PNAS manuscript. Why should I bother? I do wish at this point that I’d re-submitted it elsewhere sooner, and time spent writing articles here could have gone toward that end, but that’s not the main point here. The main point is this: there’s no convenient place for scientists to easily and openly discuss problems in science, and especially, problems in the review process in science. This is a big problem in my view.
I’ll focus on the latter issue. If you talk to seemingly almost anybody in science, you’ll hear a story, often more than one, about a bad (often atrocious) review they experienced, wherein reviewers rejected a manuscript for illegitimate or very questionable reasons. Those reasons vary, but often the reviewers just simply didn’t understand the paper in question, because they either didn’t spend the time to read it carefully, or didn’t really understand the topic area as well as they needed to. Sometimes the reason given for rejection is just frankly a convenient wrapper, such as “author(s) do/does not provide an adequate defense for assertions” or “authors’ tone reveals an underlying bias and is not acceptable”, or similar vague, general and/or unsupportable or irrelevant statements.
That’s a big problem just in itself of course. But what’s just as interesting to me is what happens next in the process. Interesting, as in, symptomatic of a deeper problem. I’m guessing that in at least 99.99% of all cases, authors of rejected papers simply move on to another journal and re-submit it there. What else can you do, you want to get it published somewhere, right? The problem is that this course of action leaves entirely unaddressed the quality and efficiency of the peer review process; it just sweeps those problems under the rug. For whatever reason, there has never really been an outlet for discussing reviews of manuscripts. Such discussions are universally “underground”–the stories you hear when you talk to individuals at any venue where the self-defined rules of “proper” scientific discourse cannot be enforced and people are thus free to speak their minds openly. Well that situation rubs me the wrong way; I don’t like the thought of people having to discuss things underground that ought to be discussed above-ground, nor the idea of things being swept under rugs and ignored. At all.
Like many things in life, it’s hard to know why this situation exists, which likely entails knowing how it originated. One issue here is the potential for anonymous retribution, and this one is big indeed. If you openly criticize somebody by exposing the quality of their review, they could retaliate in any number of ways in the future (future manuscripts (including the same one when you re-submit elsewhere), grants, and job applications to name three big ones). This problem is severely exacerbated by the one-way anonymity at most journals: you as a manuscript author have no idea who the reviewer is you are publicly criticizing, but they most certainly know who you are. That is a BAD situation to be in! Two-way anonymity is better, and some journals have that, but it’s not an ideal remedy either, because the authors can often be inferred based on the topic, writing style and reference list, whereas the reviewers can remain perfectly anonymous.
There are cases of good open discussions in the scientific literature, for example Climate of the Past, where the reviewers are not anonymous and the full exchange between authors and reviewers during the review process is published along with the paper itself (as a “Discussion” paper). So, this is a big improvement over the typical situation, especially the identification of all involved. But even it only applies to papers that actually get published, so it’s still a limited solution, [NOTE: No, I’m wrong on that, see the comment with links by Paul S in the comments section] and I know of no outlet devoted solely to detailed discussions of reviews of submitted papers, accepted or not. What we’re left with as options then, are those public fora easily accessible to both writers and readers, websites (especially “blogs”) being the best example, circa 2013. They’re easy to start, they’re open, they’re instantaneous to publish in, and readers can respond with comments; ideal in all those respects. In light of these things that’s where, at least for now, I’ll discuss problems in tree ring analyses and the manuscript review process. Because, for both, I want the public to understand these issues, and for the latter, there’s simply no place else to do it.
And please don’t quote me the oft-heard response “Yes, but the peer review process works well in the long run”, because that doesn’t address the great inefficiencies in the process, nor the frustration and career effects on individual scientists on whose backs the short run is sacrificed for this supposedly sound “long run”. Nor is the claim particularly correct even aside from those things, given how much bad work does get published, among other issues.