Is Popper responsible for this mess?

OK, admittedly this is a bit of a weird post, but otherwise I’d have to be actually working.

It’s just a question post really, because admittedly I’ve read very little of Karl Popper’s writings, and whatever little that was, it was a long time ago. I just know what everybody in science “knows”: he’s the “falsification” guy. That is, he reportedly believes that scientific advancement comes mainly via testing hypotheses (ideas, concepts, theories, call ’em whatever you like as far as I’m concerned) and then assessing whether the hypothesis withstood the test successfully or not. If it didn’t, chuck it and come up with another one; if it did, test it some more and scale your confidence in it with the number (and/or stringency) of the tests it’s passed.

Hmm, well OK I guess, but it leaves me with this image in my mind of some authority figure standing over me saying “Your idea has been falsified by group X doing unequivocal test Y. Your idea fails. Now get out of here.”

Not to go all Bayesian Bandwagon on the issue, since I have serious questions about that viewpoint also, but if you’re addressing a complex question and you carefully and repeatedly add a little bit of good evidence at a time, over time, thereby eventually narrowing down the list of likeliest explanations for your observations, then you don’t really need to worry about “falsifying” anything really, do you? I mean, lay a solid foundation, then add floor one, then two, etc…. and there you go. I get the feeling Popper thinks science is a bunch of wanna-be sand castle architects running amok on the beach trying to outdo each other but without much of a clue really, but then WHOA, here comes the sand castle judge and he’s going to wreck all but one. But then maybe it is, at least in some fields. Jimi Hendrix could have written a song about it.

I think my main question really is this: did the obsession with hypothesis testing–and all the problems arising therefrom–come from following Popper’s ideas, or did Popper just describe what the hypothesis testing fanatics were already doing? Chicken and egg question really.

If this post has been unsatisfactory to you, I am willing to tell Rodney Dangerfield jokes or discuss baseball. Thanks for your attention either way.


14 thoughts on “Is Popper responsible for this mess?

  1. I suspect that the horror that is “null hypothesis significance testing” would have deeply offended Sir Karl’s logical sensibilities. There was a lot more to his view of science than FALSIFICATION. He has lots of interesting things to say about scientific literature, objectivity, the evolutionary nature of knowledge and lots more besides. His take on falsification is far more nuanced than the caricature it has become.

    He’s worth reading in his own words. Logic of Scientific Discovery is the classic. It’s pretty good – and if you get a modern version has amusingly exasperated footnotes about how he was serially misunderstood – but Conjectures and Refutations and his autobiography, Unended Quest are less heavy going.

    • Excellent, thanks John. I guess it’s really just the people who spout the Reader’s Digest version of Popper that I have an issue with. And mainly I wasn’t being too serious to begin with.

  2. “…science is a bunch of wanna-be sand castle architects running amok on the beach trying to outdo each other but without much of a clue really…”

    Well, yeah, it often seems like that. A lot of the stuff that gets published in journals does seem like sand castles because the evidence is so widely dispersed that no one can really describe it. Typically, evidence about a hypothesis consists of supporting evidence, plus evidence that refutes other explanations. But how do you know that ALL other plausible explanations have been refuted? Academics, writ large, tend to spend a lot less time on refutation than on confirmation (and there are all sorts of laments about that, I realize). But if you actually draw up a logic diagram for a phenomenon showing all plausible explanations, you will quickly realize that most of the work is in refutation. This hews very close to what I do for a living, basically building objective, evidence-based logic trees around what scientists call “attribution.” My impression is that academics would rather not be bothered with the much higher threshold of evidence – basically a complete set – that is routinely required of us engineers.

    • I’ve been thinking about some of your points a lot lately Matt.

      The first thing I would say is that academics are all over the map in terms of their rigor; some are very good and thorough and others much less so, which makes it difficult both to generalize about academics as a whole, and also to ascribe causes for the existence of bad scientific practice. Added to that is the fact that some disciplines the questions asked are more difficult to answer, due to the nature of the subject matter and the historical, developmental stage of the discipline. For example, I think I could make a decent case that although the molecular geneticists/evolutionists have extremely strong evidence for their claims, relative to say ecology or climate science, the nature of at least some of the questions they ask is much easier, because the laws governing the patterns of DNA transmission are far simpler and more tightly constrained than anything in the those other fields. [Gene expression, genotype to phenotype relationships (and similar topics) however–that’s another story altogether, much more difficult].

      I used to see people make disparaging comments toward engineers, especially readers at RealClimate, which I thought was odd and unwarranted. One thing about “practical scientists” (by which I mean people like engineers, doctors, farmers, or trouble shooters of any kind really–people who try to diagnose and solve problems, or construct things that have to function as intended) is that there’s a tightly coupled relationship between the development of concepts, and immediate tests thereof. You don’t get away with building a bridge and having it fail, or administering the wrong drug and having it kill your patient. In academia unfortunately, this coupling can be very loose to essentially non-existent. That ain’t rigor.

      Relatedly, in parts of academia there’s often little to no coordinated, strategic effort in addressing particular questions. It’s sort of every man–or research group–for him/themselves. That’s an exceedingly inefficient way to proceed, for a number of reasons. On some topics I often see nothing more than an attitude that’s something akin to “Well, I think we should go about it my/our way, not your way, and that’s what I/we are going to do”. Such fields are often an analytical mess, because they never took the time to come up with agreed-upon methods that are sound and strategic, before applying them to specific situations or questions. They never laid a sound foundation.

      Your point about evidence for confirmation taking precedence over that for refutation is a good one, and is indeed a very big problem, and one that Popper discusses in this brief essay on what drove him towards his falsification ideas. Ronan Connolly provided that reference via Twitter.

    • “In parts of academia there’s often little to no coordinated, strategic effort in addressing particular questions. It’s sort of every man–or research group–for him/themselves.”

      That captures my thoughts much more succinctly than I expressed them. I agree that engineers usually enjoy certain luxuries in terms of being able to perform I/O testing, while climate scientists, just as an example, can only wish that were possible. One analogy that relates to the varous disciplines is that if you are writing papers in theoretical physics, you are basically an ant carrying a grain of sand to the top of a pile of sand grains, expected to find the exact spot where your grain fits. At the other end, academics such as sociologists and psychologists are building something akin to a web. Your paper need only have a single, weak connection to anything else anyone has done, and is likely to vector off in some random direction. Over time in a field like physics, a few grains of sand may slump off the pile as bold new ideas arrive, but the pile is largely immutable. The webs, though, are regularly ripped asunder. as we see with all the “revolutions” in psychology that tend to completely dismiss the previous paradigm. Unlike some others, I do not see the folks building piles of sand as inherently more scientific than those building webs, the structures have a lot more to do with the questions being asked and the tools available to answer them. But even with that in mind, it seems way too many academics hoping to add a new thread to a web have no idea (concern?) about what the rest of the web looks like, and where the new threads are needed. Engineers have slogged through the work on formalizing strategies for objective attribution analysis, and IMO there is plenty that academics could and should borrow, but there are some caveats. You have to be able to precisely frame the question before engineers can help answer it, and if you put engineers in charge, innovative approaches would be shunned in favor of the tried-and-true. So in the case of an organization such as the IPCC (which IMO really struggles with attribution strategy), were it run by engineers, we would have seen less progress in our general understanding of climate, but we would have a better understanding of the risk associated with AGW.

  3. Hi Jim,
    It sounds like you are talking about the empirical side of things – amassing observations that seem to add to your understanding of a system. You can build a stone wall from a collection of rocks without much theory, but walls are of limited use (and I say this as an empiricist with a deep distrust of theory). Eventually, though, you may end up with a pile of observations that you hope have a more general implication than the particular system you are working with.

    When I had to teach this, I always recycled Henri Poincaré’s “Science is built of facts the way a house is built of bricks, but an accumulation of facts is no more science than a pile of bricks is a house”. That always seemed to get across to 1st year students – or at least that amorphous mass of 400+ stayed relatively quiet (crowd control is the first law of 1st year teaching) and I could move on to how a scientific hypothesis was different from an opinion.

    My understanding of Popper was that if you wanted to build a house out of your accumulated bricks, then you needed a prediction as to what the edifice would look like and a test to see if it worked. If it fell down during construction or need constant ad hoc patches to keep it up, then you needed a different design or better bricks or whatever. I also thought that Popper was essentially correct in claiming that you can never be sure of what is true (is this really the best house I could build with these bricks?), only of what is false. However, all my knowledge of Popper is secondhand, so I should probably follow John Kennedy’s advice and read “Logic of Scientific Discovery”.

    • That looks like a good read at first glance–thanks Bob. Lately I’ve been pondering just exactly what type of “memory” muscle memory constitutes, for whatever that’s worth.

  4. Jim,

    Never really explored the connection between Popper and the beginnings of frequentism, but I stumbled upon this post that mentions both Fisher and Popper:

    From what I can read, it seems that Fisher was after solving induction, while Popper rejects it.

    There’s also this interesting paper by Mayo & Cox, who claim that Popper and Fisher are mostly in violent agreement:

    They state, though, that falsificationism may have needed a better formal apparatus. Something like frequentism. But then that’s to be expected from Mayo.

    Popper is worth the read. Start from Realism and the Aim of Science, which is more mature than his youth stuff. But of course in the end holism wins.

    I might be biased, though.



Have at it

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s