Our new consensus study

Polls indicate that a substantial percentage of the American public has doubts about the idea of evolution by natural selection. But just how strongly do working scientists believe in this idea? Is there a consensus view and if so, does it differ from the public’s?

One way to address this question would be to actually present the various arguments and evidence that have been offered on the matter. However, these are somewhat complex and nuanced, so this would require quite a lot of training and expertise, which we don’t really have and which would take a long time to acquire. This is a distinct disadvantage, given that time is of the essence, since our primary goal is to influence policy makers to do something before many more Really Bad Things (RBTs) occur. Indeed, many RBTs are already occurring, and the consensus view is that these are completely overwhelming all other RBTs known to arise from all other causes, or at the very least they surely will soon. Careful science takes lots of time, and that’s a luxury we just don’t have I’m afraid. The consensus viewpoint from a wide variety of sources, especially on the internet, is that this is an emergency situation we’re talking about.

A much better approach would be to search the scientific literature with a search tool like the Web of Science, and then have a group of people look for key indicators of scientists’ positions in the papers returned. However, this approach generates quite a huge amount of information, the sheer volume of which must be pared down for sake of tractability. Scientific papers tend toward the long and aggravatingly complex, where various substantive arguments are (sometimes) made and things like that. Some of them just go needlessly on and on frankly, full of graphs and numbers and whatnot. Who’s got time for it?

Abstracts however, and better yet, titles, are much shorter and easier to handle. If one has a key search phrase in mind, one can skim through an abstract quickly and determine exactly what those authors think about the issue, and why, as long as one is willing to make the effort to read between the lines. Moreover, even most scientists don’t read entire papers anyway, except for those few who are unconcerned about affecting policy makers, and who are often frankly quite obsessive as to scientific arguments, “methodologies” and similar pedantry. This is a serious problem, but not one we can address here, although we consider our work a first step in the right direction.

So, onto the results. Our search resulted in 487,629 papers that mentioned “evolution” or “natural selection” in the abstract. However 451,412 of those could not definitively be placed into one of our seven position-defining categories*, no matter how hard we tried with our group of 20 reviewers. [The consensus view among us is that these reviewers are completely independent and objective; their common participation at our web site devoted to presenting pro-selection arguments, but nothing to the contrary, is just not relevant in this case.**]

Of the remaining 36,217 papers, 35,167 (97.1%) supported the consensus position that over half of the observed evolution over the twentieth century is due to natural selection. The fact that only 126 of these 35,167 papers were actually focused on critically evaluating the topic at hand, i.e. the different possible mechanistic explanations of observed evolutionary change and/or speciation (e.g. random drift, founder and other stochastic events, mutation rate variation, instantaneous genetic barriers, etc), is an irrelevant point, a complete red herring. We can reasonably assume that in at least the majority of 50% of the time, none of these 35,167 authors would indicate agreement with a position that they themselves had not carefully investigated, without having more than half of a predominantly pretty good reason for so doing***. It’s just not really half as difficult as people make it out to be when you boil it down. As we have now done. For you.

In conclusion, there is very clearly a very strong consensus as to the influence of natural selection on evolution during the twentieth century and this consensus has been increasing as the evidence increases. It is important that policy makers realize this and take action. Please pick this up and disseminate it widely so everybody knows about it; everyone else is, so you will be part of the consensus effort if you do. Thank you.

Notes:
* These categories define the level of endorsement of evolution by natural selection. For example, Category 1 entails what we call “explicit endorsement with quantification”. An example of this level of support would be given by the statement : “The evolution during the 20th century is caused mainly by natural selection, especially since the late 1980s”.

**Our methods involved having each abstract read by two people. The consensus view among us was that these people were independent and highly unbiased. Papers on which no consensus could be reached by these two were decided by the consensus opinion of the results of ten coin flips.

*** This basic concept was first elucidated by Hall of Fame baseball star Yogi Berra, who at some point or other is said to have first discovered that “at least 50% of baseball is half mental”. Even Yogi himself did not at that time realize that this concept describes a fundamental property of the universe.

33 thoughts on “Our new consensus study

  1. Well this is jutst perfect! Part of me is entertained and amused by this post, part of me is ashamed for having spent the time reading the blog back-and-forth on the paper being skewered. It has gotten to the point where I would almost rather read about the proper methods for splicing together data from different types of proxies!

  2. This was great, but Good Lord man, are you off the reservation? If so go private to protect your career. You are veering into fringe/lunatic territory, where to even question the “consensus” view will ruin an honest man.

    • Just to be clear, I am not questioning that there is a strong consensus among climate scientists w.r.t. the effect of humans on the climate–both definitely exist (a consensus and a strong effect, and moreover, the consensus derives largely from the evidence, IMO). Nor do I have anything against the authors. Those are not the issues here. What is very much at issue for me, is the methodology they used, and the fact that it got published in ERL and had such popularity without much critical commentary from scientists.

      I’m not going to “go private”, i.e. anonymous, just so I can say what’s really on my mind–I decided that a long time ago. I don’t want to live that way and can’t imagine why anyone would. It seems to backfire with about everyone who tries it, as far as I can tell. I just call ’em as I see ’em and let the chips fall as they may, or as Bob Dylan once said “I try my hardest to be just like I am”.

  3. What is very much at issue for me, is the methodology they used, and the fact that it got published in ERL and had such popularity without much critical commentary from scientists.

    I agree.

    You’re a scientist, Jim.

    Show the way.

    • I’m trying Willard. I try to steer clear of this kind of thing and focus just on the science itself, but the authors of this thing and their supporters have been so outrageously over the top on it that I had to start saying something. This whole incident is very revealing.

      • Eg it has revealed how pathetic and ignorant of science all supporters of Cook et al actually are.

        not that there were any doubt about that

        if that’s what can pass as climate change science one can only look in the despair at its abysmal lightness

      • Important to remember that this is climate change communication science, not climate science itself. If one wants to investigate this topic, then get a decent sample of climate scientists who specialize in attribution-type studies, and conduct very well structured interviews with them. Even then, you’re just getting their opinions on things, it’s not like you’re getting a multi-model ensemble of results from common input data sets or something like that. There are serious limits to this kind of thing.

    • Jim,

      I’m not sure what this incident reveals exactly. You must be new here. What does surprise you?

      Don’t get me wrong. I like a good parody. I thought you had something more constructive in mind when you ask scientists to provide a critical commentary.

      What would be your main point of contention, besides the fact that you find the whole exercise ridiculous?

      • “What would be your main point of contention?”

        Principally, (1) bad methods and results which do not flow logically therefrom and (2) stated goals of influencing the public, and policy makers, even before they had results in, i.e., they pretty clearly had a foregone conclusion in mind and set out to find a way to justify it. Richard Tol also raised a number of statistical points, which are more minor IMO, but which he probably possibly had to focus on (knowing how hyper-sensitive some journal editors are), in order to get his article published. And some other things, not going to get into those now.

  4. Thanks, Jim.

    Since you’re not new here, then I’m at loss to explain how all this surprises you. This episode of ClimateBall ™ more or less follows the same structure as the previous ones. I’m not sure if it’s up to the pilot, as I’ve not contemplated the idea that ClimateBall ™ episodes had a pilot.

    I think that your (2) suffers from the very basic observation than just about any study with policy implications invalidates it. See for instance:

    The cholesterol factor is of minor importance as a risk factor in CVD. Of far more importance are smoking, hypertension, obesity, diabetes, insufficient physical activity, and stress.

    http://chriskresser.com/cholesterol-doesnt-cause-heart-disease

    The general claim that a study is unethical because it has policy klout falters on empirical grounds. Perhaps I misunderstand your (2), so perhaps some clarification is needed.


    In any case, I think your (1) is more important. What would be good methods and results that would flow logically therefrom according to you? In your response, please think about the problem of improving scientific crowdsourcing like Cook & al tried to do.

    I’ll comment on Tol later, as I believe you allow only one link per comment.

    • It surprises me by its sheer chutzpah, the brazen-ness of it, relative to where it was published. I see two possible explanations here: (1) They really don’t understand why their study’s got problems, or (2) They do understand but are basically saying, “yeah, well we know how this political game is played, we’re going to play it, and then defend it like hell against anyone who criticizes it” (also known as “act first, deal with questions later”, aka “force the issue”). I don’t know which of the two it is.

      I think I agree with you that my (1) is the more important of the two points, but I think their choice of methods were likely motivated to a large degree by point (2).

      As for your question, the whole idea of “scientific crowdsourcing” on something like this is just an awful approach. The whole idea of trying to quantify a consensus in the first place is questionable. Note that there have already been much better ways used to go about determining what scientists believe, e.g. Pitelka et al (2001?), Zickfelds etal (2010), just off the top of my head.

      [note: WP held your comment in the queue instead of doing the typical auto-approval. That’s the 2nd time I’ve seen that, and I have no idea why. Did you originally try to include >1 links?]

    • Jim,

      I have little time, since I’m busy with rereading Popper at Bart’s. But I promised a link for Tol’s comment. Here it is:

      Discussing mistakes in influential, published papers on important topics is not “bitching”; it’s called criticism, and it’s an important part of science.

      http://andrewgelman.com/2014/05/27/whole-fleet-gremlins-looking-carefully-richard-tols-twice-corrected-paper-economic-effects-climate-change/#comment-167776

      I put this quote to make two points.

      First, while we should distinguish bitching and criticism, we should also distinguish stylish snark and tasteless bitching. At AT’s, Steve Bloom basically said that you should be ashamed of your parody. I disagreed with him, and reproved his own shaming stance. While both your parody and Steve’s comment arguably qualify as bitching, I believe that your editorial has style and took time to write. It deserves to stand, not Steve Bloom’s.

      Second, notice the date. It took 5 years to notice that Gremlins have been messing with Richard’s data. Five years is a lots in scientific time. Things like these take time, even when comes the time to pay due diligence to the most cited scholar in Stern’s report. Imagine now a paper that establishes what few climate scientists dispute. All you have is hundreds of bickering editorials in contrarian outlets. Why would anyone care? There’s nothing to gain, and lots of time to lose.

      See it from your own perspective. I know this is not your specialty, but you seem to trust your reading of C13 enough to question if not dismiss the whole enterprise. On what grounds? You seem to have read Richard’s comment. You recall similar research. Perhaps you read a few other blog posts. But I think that’s about it. If you don’t care about working on a more constructive criticism, why would anyone?

      My own working hypothesis is that we see so much bitching because there’s little incentive for something else. Nothing to gain. Too much to lose. Those who had some skin in the game have spoken. The caravan goes on.

      I hope this answers your “Y U NO SAY SOMETHING, SCIENTISTS?”

      I’ll return later on your criticisms if you please, but I have a feeling you don’t bother much.

      Bye,

      w

      • I’m not sure who called Tol’s criticisms “bitching”, but I agree with his response that you quote there. [Edit–my mistake, Tol called Gelman’s cricitcsms bitching (apparently); I agree with Gelman, and Tol shouldn’t have said that.]

        I agree with your point about it being largely a waste of time for scientists to engage on blogs. It largely is–there’s not much return on it, and especially if people don’t reply to what you’ve said. On the other hand, thank god for people like Isaac Held for example, just to pick one very good one, who do a real service by blogging on the nitty gritty details of the science, from which we can all learn.

        See it from your own perspective. I know this is not your specialty, but you seem to trust your reading of C13 enough to question if not dismiss the whole enterprise. On what grounds? You seem to have read Richard’s comment. You recall similar research. Perhaps you read a few other blog posts. But I think that’s about it. If you don’t care about working on a more constructive criticism, why would anyone?

        Willard, I don’t thin it needs to be anybody’s specialty–the flaws in the paper are immediately obvious to anyone with a sense of how inference proceeds. The grounds are as follows, in no particular order: (1) They had blog members evaluate the abstracts of ~12,000 papers. This blog has a well known history of presenting only arguments that favor the “CO2 is a bad thing” viewpoint. It is not a site that evaluates the soundness of science methods–their members are not trained to do so, (2) An abstract is a synopsis conveying the bare minimum results–it’s not designed to explain authors’ full set of arguments (and indeed, even full papers can fail to do so, due to page/word limits), and (3) They appear to have used many hundreds to thousands of abstracts that had nothing whatsoever to do with the attribution of the current warming to particular potential causes, i.e. the principal thing they’re trying to investigate . They ended up with ~64 abstracts that explicitly (according to them) argued for their claim about what’s causing >50% of the observed 20th C warming.

        Furthermore, one could easily take issue with their underlying premise, which is “if policy makers only understood how strong the consensus is, they would get serious and take some action to curb emissions”. Well, maybe they would and maybe they wouldn’t. Policy makers are an unpredictable bunch, not unlike weather vanes I believe Mark Twain (or perhaps Yogi Berra) might have said. They might for example, not much like it when they find out the papers’ methods weren’t sound and the results thereby exaggerated.

        Here’s one really important point. Most of us are busy and there’s a HUGE mountain of scientific literature out there, especially in climate change. You have to learn how to very quickly sus up the likely validity of papers by looking at a few key methodological details, or you will be forever drawn into the weeds and waste you time. This is an enormously important skill. If a paper doesn’t make it past my first set of filters, that’s it, I’m not going to waste any time on it by going further. Moreover, I keep mental note of the authors, especially the lead such, and the next paper I see by them, I’m going to go into Bayesian mode, if you know what I mean.

      • btw, not sure who “AT” is, but it’s always nice to see that at least somebody appreciates what one has to say. Steve Bloom has a good head on his shoulders it seems to me, but he is solidly in the camp of those who want to control the public messaging, while I most definitely oppose that, as it attempts to stifle expression of the full degree of uncertainty on the science.

    • Thank you for your response, Jim.

      I may not have the time to comment on your argument against C13 for a while in a way to do it justice. I’m still rereading Popper with a guy at Bart’s, and yesterday I got distracted with the Lomborgian “but the poor!” at Judy’s. And tomorrow I’ll be AFK for a while. (No, I may not go in a forest, sorry.) I’m already glad that you clarified your position.

      First, I want to clarify this:

      I agree with your point about it being largely a waste of time for scientists to engage on blogs.

      My point is that there must be a reason why science blogs have evolved the way it did, with its daily dose of snarky uppercuts and smarmy jabs. I surmise that there are less incentive for rational criticism than playing ClimateBall ™. ClimateBall ™ itself provides a disincentive. There’s also the fact that bitching is just easier and more fun.

      You can find Bloom’s comment on this thread at AT’s:

      Defending the consensus, again!


      My second part will follow. I need to put another link.

    • That you implement Markdown is very cool, Jim! Here’s part 2.

      My objective, here, with Richard a year ago and elsewhere, is to experiment with promoting constructive criticism. That does not hinder ClimateBall ™ at all: it is too much fun an ethos to let go. Reminding ourselves that we all abide by rational enquiry may help us appreciate ClimateBall ™ even more.

      Richard started with saying to Dana “C13 is a load of nonsense,” and ending up writing a formal comment. Me and others insisted that he backs up his dismissiveness. Richard wrote a formal comment. I’m glad he did. This is why I spent more time reading his previous versions than I dare declare. (My name appears in the acknowledgements.)

      That does not mean I think he’s right. For starters, there are problems with the main result of his last version:

      The Tol comment method predicts that it changes from 96.73% to 87.44% (versus observed 97.06%). Working backwards, Tol’s correction requires that the pre-reconciliation consensus should be >100% (as pointed out by Anonymous). I’m sure that Professor Tol will realise and retract his paper or, at the very least, submit a quick correction.

      http://rabett.blogspot.ca/2014/06/in-spirit-of-deep-climate-anonymous-as.html

      In return, I agree that C13 is far from perfect. I myself offered a few at Bart’s, at AT’s, and perhaps elsewhere. I would have to find back everything, as I deleted my collection of bookmarks. There was more than a hundred bookmarks on that collection. If you don’t believe me, count all the posts at Lucia’s about C13, including the ones where she suggested ways to hack the survey.


      Anyway. Richard’s comment was way more constructive than he claimed. OTOH, Richard also spent way more time than I think he expected when he said to Dana that C13 was “a load of nonsense”. I too spent lots of time on this. Two months of research, more or less.

      So on that basis, here’s what I could answer to your criticisms:

      1. That ABSTRACT raters are not trained to judge the merits of an argument about AGW, besides being ad hom, may not be quite relevant to question the ability that requires the task of classifying an ABSTRACT. Background knowledge may suffice to be able to read, understand, and classify the ABSTRACT.

      This, of course, is independent from what you could say about the classification itself. But whatever you might think of the classification, it should apply to any kind of rater, whether it’s a specialist of the field or a 3rd grader. That task was also indirectly tested on the authors themselves. Which leads us to your number #2.

      1. While it is true that an ABSTRACT does abstract away many details about a study, it’s supposed to be a good ABSTRACT. A good ABSTRACT should be able to do its job to convey the main points and the method.

      There is of course the possibility that this is untrue, but that may need to be verified. There is no reason to assume that such a possibility would offer a bias against C13’s results. In fact, it could very well be the other way around: be default, we could expect the raters to rate conservatively.

      Arm waiving this possibility is therefore not enough. This may provide a reason why Richard did what he did.

      1. The study is not about attribution, but about the endorsement of the consensus on AGW as we can read it on the IPCC’s documents. That’s it. There already are studies of what attributions studies say on atribution. While I believe they are quite conclusive, I grant you that they themselves could be improved with a crowdsourcing effort like C13 tries to do. That said, unless you have an argument according to which only attribution theorists should be allowed to endorse the consensus, I think you’re criticizing for now doing what you’d have preferred them to have done.

      Here’s an argument that appeals to ordinary language. Attribution theorists are those who created the studies the consensus is about. You can’t say that they endorse it without reducing what they did.
      I think this shows a problem with your claim that only attribution studies should be taken into account to measure endorsement.

      In the end, what you think is a bug may very well be a feature. I understand that measuring implicit endorsement may not be that convincing. If in the end we don’t, we might as well ditch the concept of endorsement altogether, at least until we write scientific reports like international treaties, where endorsements are all explicit. And even then, do we really know what power the new AUMF will grant the Commander-in-Chief the power to act against an “imminent threat”?

      I urge everyone to search for “60 words” on Radiolab for a story on this. This is way more important than “unprecedented”, “delete” or “endorse”.

      Implicitness is everywhere, including in my comment.

      C13, for all its faults, accomplishes something that should not be dismissed out of hand. Perhaps the moral of the story is that unless you’re willing to risk having your arm caught in the ClimateBall ™ machine, stay away from bitching.

      Enjoy your week,

      w

      • Once again this got held in the queue when it should have automatically appeared–sorry. Yet your previous one went through fine. No idea what’s going on here.

      • Usual plentiful verbiage and silly name dropping from w but little content, as there is zero meaning to discuss C13 without agreeing what “consensus” was actually measured by it. Plus the criticisms of C13 go way beyond Tol’s published comment. Plus it’s rather sad to see the reputation of a scientist impugned because he had the temerity to accept there were errors in a paper of his.

        Don’t you worry Jim, if only w could get his hands on managing your blog as well, you’d be surprised how many comments would disappear for the sake of his twisted ideas regarding constructive criticism and the mysterious Climate Ball.

      • OK, enough of that please. Willard’s as welcome to make his points as anybody is. Let’s just stay focused on the topic, not each other.

    • Seems to work!

      Here’s the second one:

      This is one of the criticisms I alluded to earlier.

      Thanks, Jim. Next time, I’ll make sure to number my paragraph in a way that won’t be misinterpreted by Markdown. (All the “A”s are “1.”, “2.”, etc.)


      I’ll let omnologos reflect on the circularity of his semantic argument. Meanwhile, he might profit from consulting thy Wiki entry on “consensus”.

  5. Willard wrote:
    “Attribution theorists are those who created the studies the consensus is about.”

    Presumably this was supposed to be a throwaway sentence, and I suppose “attribution theorists” is just meant to refer to those who think the recent warming is caused by CO2. But it is important to realize that at the very heart of the consensus paper is an assumption that some well-formulated work has gone into attribution. It hasn’t. I am no climate scientist but have read enough to be competent on the subject. I am absolutely appalled at the amateur effort, spearheaded by the IPCC, that has gone into developing the framework of attribution logic. I have been performing “attribution” (we call it root cause analysis) for over thirty years, and I can confidently attest that pretty much every guideline has been broken by the IPCC. The folks you call “attribution theorists” have no idea about attribution theory, which lies in the field of engineering. Had they asked for help on how to establish the root cause of an emergent phenomenon, they surely could have gotten expert advice from engineers. The interim and final products would not resemble the ARs in any meaningful way, of that I am certain. So in addition to all the methodology problems that Jim skewers, there is also the fact that even the most pertinent papers that the consensus study was supposed to winnow out sum to incoherent evidence of attribution.

      • Mmmm, I’d have to say that if you’re requesting a citation for that, then one could as well be requested (as well as a definition), for your original statement that “Attribution theorists are those who created the studies the consensus is about”. Are you confusing the practice of attributing causes, which many scientists do, with the theory thereof? It looks to me like you are.

        I also think there are some very questionable or wrong statements in your comment. I didn’t have time to list all my objections to this paper (because there are so many), but certainly one of them is the idea of “it’s not an attribution study, it’s an endorsement study”. Endorsement is a big hornet’s nest in this context–the vast majority of scientists in a more broadly defined field are going to endorse whatever the prevailing viewpoint is among the specialists. You’re going to get a high, so-called consensus, but so what, it doesn’t mean anything, given this reality. Scientific knowledge is very hierarchical and specialized, and progress is very much based on non-experts trusting the work of the experts. It has to be–there’s too much to investigate for it to be any other way. And I’m talking about just among scientists there.

        I’d also like to see an explanation of how it is that a sociological study made it’s way into a physical science journal. I’ve not even touched on what this indicates about ERL and its editors, but it ain’t good. If the goal was to inform scientists, well that’s not needed because scientists don’t need to be informed that there’s a general consensus that CO2 affects the climate, although it’s not clear from what I’ve readexactly what point is being investigated. If the study’s instead targeted at the public, which is what they claim…well, then what’s it doing in a physical science journal?

        And one last point: it’s not ad-hominem to question the expertise of somebody doing a study. It’s sort of the first question you should ask in fact.

    • Jim,

      Everyone’s free to ask for a citation. But since you ask, here it is:

      If one wants to investigate this topic, then get a decent sample of climate scientists who specialize in attribution-type studies, and conduct very well structured interviews with them.

      Our new consensus study

      My remark follows from whatever is presupposed by that. I don’t think my confusion between “practice of attributing causes, which many scientists do, with the theory thereof,” if any, is relevant to make a point that follows from your own argument.


      I also think there are some very questionable or wrong statements in your comment.

      I’m glad you do. When you’re ready to voice your thoughts on these, don’t hesitate. I’m all ears.


      Endorsement is a big hornet’s nest in this context–the vast majority of scientists in a more broadly defined field are going to endorse whatever the prevailing viewpoint is among the specialists. You’re going to get a high, so-called consensus, but so what, it doesn’t mean anything, given this reality.

      I agree that doing a consensus does not mean much. I’d even say that it means what it means, and nothing else. And yet that’s what they did. Andy Skuce tried to explain why:

      http://planet3.org/2014/06/26/consensus-criticism-communication/

      You might wish to discuss that point with him.

      I personally find “meh” as easy as it is uninformative.


      I’d also like to see an explanation of how it is that a sociological study made it’s way into a physical science journal.

      You might need to ask, then. I can’t answer that question for the journal in question. Raising it in one’s blog can either be academic, or rhetorical, say as a way to indulge into counterfactual thinking.


      And one last point: it’s not ad-hominem to question the expertise of somebody doing a study. It’s sort of the first question you should ask in fact.

      I disagree. There’s a difference though between questioning someone’s competence after checking that person’s work, and undermining that person’s authority by looking at the area of expertise of that person simpliciter. Unless you can separate a person from that person’s expertise (I assume it runs deep into a researcher’s persona), both are ad hom. The first may be valid, the second seldom is, if ever.

      Until you present some constructive criticism, you run the risk to raise the second kind of argument.

      • You’re not getting it Willard, and are reverting to your habit of arguing for the sake of arguing, and playing word games. I don’t have time for it; it’s not high school debate class here and my patience for that kind of thing is very very short.

        One can certainly criticize a piece of work for it’s flaws without having to meet your (or any imposed) standard, whatever it is, of “constructive criticism”, and neither Tol nor I nor anyone else has to satisfy your criteria of what counts. You’re just repeating a tired jingoism among academics with that claim. And at any rate, I did in fact offer one alternative way with the quote you copy there, and was in fact the method used in a number of previous studies of this type, by e.g. Pitelka et al, 2001, Zickfeld etal 2010 and a couple others.

Have at it