Severe analytical problems in dendroclimatology, part four

In part two, I discussed problems with the second of the three problem issues being addressed, problems introduced in the “detrending” step, which is supposed to remove the long-term tree age/size effect (noise) and retain the climate signal.  I’ll continue that here, and begin to bring the hammer down on why estimates of long-term climatic trends must frequently be wrong, at least when ring size is the analyzed variable (as it most frequently is in tree ring studies).  I include some graphs this time to help bring the take-home message, well, home.

Update, 12-19-12:  I replaced the four graphs with those that give a correct representation of the differences between the two types of RCS splines (stiff and flexible) tested.

To summarize briefly, the essential problem is that changes in tree size and long-term (“low frequency”) climate changes (~ century scale and longer) both affect ring response and can and do occur concurrently.  Therefore, there is no obvious way to determine how much of each year’s ring response is due to tree size vs climate–a classic example of statistical confounding.  The RCS method was developed to address this issue, by estimating the “expected” (~ mean) ring response for any given tree age (but size is actually the better predictor and my analyses use it, rather than age).  This expected response is assumed to apply to every tree, and the deviations of each tree ring from it are therefore assumed to represent the effects of climate. For this concept to work however, there must be a good mixture of tree ages in the sample, so that (hopefully) each calendar year is sampled by rings from many different tree sizes (and conversely, each tree size occurs across a large part of the range of climate states experienced over time).  I mentioned before that the method also requires that trees have as similar responses to climate as possible, and that they also experience similar non-climatic environments (e.g. soils, topography, competition etc.).

This seems at first glance like a reasonable solution to a potentially thorny problem.  There is one small hitch though: it doesn’t work.  A second small hitch is that there are no other known solutions to the problem.  Only in certain highly optimal situations, completely unrealistic for most real tree populations, does it approximately remove the tree size effect and return an adequate long term trend estimate.  Otherwise it will fail badly, usually with a definite directional bias.  Big time problem.  As in, leading to an uncertainty that is fatal to confidence in resulting chronologies and climate reconstructions.  I’m not exaggerating; keep reading.

So why do people use it? The first answer is that a lot of times they actually don’t, they still use the older ICS method instead (described briefly in part two), with its known severe weaknesses in recovering long term trends.  When they do use it, it’s presumably because it’s thought to be better than the ICS method, which it is, and because there are no other methods available (other than variants of RCS which have the same set of problems).

How well-known is this problem and has it been described before?  There is some limited understanding of the method’s weaknesses in the literature, but a full understanding requires a systematic evaluation via simulation, which nobody appears ever to have done, until now.  Why they haven’t done so, I have no idea, just as I have no idea why the issues raised by Loehle (2009) haven’t been recognized and embraced, or why people still estimate long-term trends using methods (ICS) known to be invalid for the purpose, without any attempt to quantify, even roughly, how much error might be introduced thereby.

One of the beauties of simulation analysis, among others, is that you can definitively test the limits of validity/accuracy of various analytical methods under various conditions.  This for example is the basis for “pseudoproxy” tree ring analyses designed to determine which large-scale reconstruction methods are best under which conditions.  You can test when methods do and do not fail, and how badly they fail when they do.  So why hasn’t this same approach been applied to other serious issues of concern in the field? Again, I don’t know, but this in fact is exactly what I have done and was the basis of my recent submission to PNAS on this whole set of issues.  I built a tree growth model, one that includes all the essential characteristics needed to evaluate the problem, such as how rings respond to climate and tree size, and how slightly different versions of the ICS and RCS methods affect the results.  For this purpose, detailed physiological mechanisms of tree growth are not required.  One simply defines various relationships between climate, tree size, and ring response, sets growth conditions and grows trees accordingly, and then evaluates whether commonly used detrending methods accurately return the known, long-term climate trend.  [I didn’t have to build the tree growth model but there were some real advantages to doing so, a very big one being that I have something I know inside and out, another being that I learned a lot in the process.]

Relative to evaluating the limits of accuracy, the most stringent test is one in which all potentially influential variables are made to have no influence.  Specifically, in this case I set conditions such that (1) there is a very good sample of trees/cores (50 trees with two cores each, whereas a typical tree ring site is often sampled by 10 to 20 trees, two cores each), (2) the start dates (first ring) of the trees are perfectly spaced throughout the 500 years of the chronology, that is, every ten years, such that the age mixture of the site is maximal, given the number of years and number of trees, (3) there is no variation in the inherent growth rate between trees, corresponding to no site or genetic differences between trees (4) there is no biological effect on ring response (ring area, or basal area increment (BAI)), meaning, the climatic variable fully determines the variation in ring response, (5) the effect of climate on ring response is perfectly linear, with no noise (i.e. y = a + bx, where y = ring area, x = climate variable (e.g. T), and no other factors influence y), and (6) a minimum sample size of six rings in any year, up to a maximum of 100.**

I then impose a steady linear increase, and then a decrease, of arbitrary magnitude, on the climate variable (e.g. T) over the full 500 years, but with a substantial amount of year to year variation, grow the trees, and then apply a number of different ICS and RCS detrendings.  Each of these has slightly different parameters, following the conventions used in the literature and/or in the two most highly used software packages, ARSTAN and dplR.

So…under this perfectly optimal set of conditions for the RCS method, which together constitute a wildly unrealistic best-case scenario under which it must not fail if it is to have any general validity whatsoever, we get the following result…

Graph1_2Figure 1.  RCS-based estimates under increasing and decreasing climatic trends.  True climatic trend is the straight black line.  Two RCS-based trend estimates are given by the straight blue and gray lines, as derived from the annual values (thinner blue and gray).  The two RCS methods differ only in the flexibility of the spline-based fit used in constructing the regional curve; the nearly perfectly overlay of the blue and gray lines shows their similarity.  The year to year variation of the climatic variable is similar to that of the RCS estimates, but is omitted to make the graph less cluttered.  The RCS detrending was done using Andy Bunn’s dplR package in R.

Not good.  There is a very clear under-estimation of the total trend magnitude in both directions, slightly greater for negative trends.

Suppose I impose the more realistic condition that all trees must have their first measured ring within the first 1/3 of the chronology length (first 167 years):

Graph2_2Figure 2.  As in Figure 1 except that all trees originate in the first 167 years (and hence, are somewhat more even aged).

This returns a slightly worse result, especially when trends are positive.

Suppose I restore that condition and instead relax the restriction that all trees grow at the same inherent growth rate, such that some trees may grow up to 33% faster or slower than the average growth rate (at a maximum; the average deviation from the mean will be about half that, of 16.7%):

Graph3_2Figure 3.  As in Figure 1, except that inherent growth rates of individual trees can vary by 33% above, or below, the mean value.  [The magnitude of variation from the mean in each tree is random, so, in contrast with the other graphs, this results will change more from run to run.  This is just one instance.]

This condition gives a better result under positive trends, but a little worse one under  negative trends, thereby imparting a definite bias towards increasing trend.

Suppose I restore that condition and instead impose a definite (but mild) biological trend, mimicking the very common forestry observation that trees typically display unimodal growth in ring area (BAI), with an optimum at mid-life medium diameters:

Graph4_2Figure 4.  As in Figure 1 except with a mildly unimodal growth response imposed.

Not much worse, but certainly no better, similar to the second condition in Fig 2.

Clearly, in all of these cases, the magnitude of the estimated long-term trend variation is muted, usually greatly, relative to the actual value.  These are not exceptions to rules; except for Figure 3 all of these results are extremely stable, even though the inter-annual climate variation (again, not shown for graph clarity), is fairly large and stochastic (random).  That random variation has little impact however, when multi-century trends are the focus, as they are here.

**Note: There is one important point relative to steps (4) and (5) described above, which is that the linear climate effect always acts on ring area, not ring width.  The reason for this is that ring width has a purely geometric noise component to it.  That is, the same amount of wood added each year to a growing tree bole will necessarily produce a gradually declining ring width, due strictly to geometric considerations, which biases the estimate of the total wood produced, and hence the predicted climatic variable. Such variation has nothing to do with either the climate, nor with tree physiological “decisions” (e.g. changing carbon allocation), and so it must be removed; the use of ring area (= BAI) accomplishes this automatically.   However, since almost all existing studies analyze ring widths, and not ring areas, I convert the areas to widths once the tree is grown, before doing the various tests.

Still more to come.

14 thoughts on “Severe analytical problems in dendroclimatology, part four

  1. Although you attempted admirably to explain what you were doing, some range of ambiguity may lurk for us plodders. So could you let me know where I went off the trail in explaining your work to myself thus:

    X (which is temperature) is an N-length real vector of Gaussian noise (low-pass filtered and?) added to a positive or negative length-N ramp.

    The year-n age of the kth tree is given for n > g_k by n - g_k, where g_k = (k - 1)N / K and K is the number of trees.

    For the first, simplest simulation, the synthetic data are generated in accordance with:

    d^2_{g_k, k} = 0

    d^2_{n,k} = d^2_{n-1, k} + a + bx_n for k>g_k

    y_{n,k} = d_{n,k} - d_{n-1,k}

    to obtain K vectors, of which all but one have missing data at the top, i.e., in the early years.

    You then adjust these data for their respective ages n-g_k in accordance with various flavors of ICS or RCS to obtain K putatively temperature-indicating “index” vectors to which you apply a model linear in n to infer a trend, whence the heavy blue lines above. The black lines, I assume, represent a+bn.

    I have no doubt butchered your approach, but the foregoing exegesis may nonetheless be helpful in assessing the merits of perhaps providing R code so as to dispel ambiguities (and relieve your readers of the burden of writing their own code).

    [Hi Joe. I think you might be making it unnecessarily complex, especially with respect to taking 2nd derivatives. The simplest case (Fig 1) is simply a straight linear production of wood (as ring area) as f(T). The trees start (1st ring) every ten years. There is no effect of tree size on ring area, nor random variation, just climate effects. The black lines represent the linear climate trend, which is a + bx. The R code is long and complex and I definitely can’t release it at this point, for several reasons.–Jim [update: sorry, yes the climate state in year n is a + bn]]
    [The reason the code is so long and complex is because the program does a lot more than I’ve described here. However, if I get time (unlikely however) I can generate something simple for the Fig 1 scenario (only)–Jim]

    • I appreciate the response, but, without actual equations, your description is too ambiguous to replicate, and, if it’s not replicable, it’s not science. Thanks for playing.

    • Three points here:
      (1) This is a blog for a general readership. If I try to describe what I did strictly mathematically, many people will not understand it as quickly, or at all. That’s why I give primarily verbal descriptions; I’m not here to snow people with opaque math, especially with unclear notation.
      (2) Nevertheless I did in fact give the necessary equations and descriptions to evaluate what I did in Fig. 1, with the exception of the two “nyrs” parameters for the dplR-based spline fits in RCS. Those values are: (a) for flexible splines, the default value, nyears = 0.10 * 500 and (b) for stiff splines, nyears = 0.67 * 500. Those nearly bracket the range of possible spline fits. Since one of those two is the default value, and the two methods produce almost identical results, this means that anyone who knows R, could in fact do their own checks on my results. Did you try that?
      (3) I do not appreciate the accusation, nor the attitude that I have time to spend evaluating your mathematical interpretation of things, and if you think I’m playing around here, think again.–Jim

  2. Thanks Jim for this extremely interesting series of posts. I hope that the PNAS submission goes well and that the reviews have not been too “robust”. The questions you have been discussing are exactly the questions I have wondered about for quite some time, but have never really felt competent to have an opinion on, so it is great to see your detailed thoughts on these matters.

    [Thanks for the compliments Jonathan, I do appreciate them and I’m glad the posts are interesting and helpful to at least someone. My PNAS paper was rejected after what I consider to be an outrageous review (the details of which I may get into).–Jim]

    • Sorry to hear about PNAS. I’m afraid that outrageous reviews happen in every field, especially when you are questioning the adequacy of the mainstream approach to a problem.

      Some years ago I had a one page proof that a sub-field in my area of NMR quantum computing (which was creating significant excitement and being feted by people and journals who should have known better) was fatally flawed and beyond all hope of rescue. Getting that paper published was an “interesting” experience, but I managed in the end. Sadly my paper hardly ever gets cited because the sub-field is now dead. Being inconveniently right is rarely the route to fame and glory I’m afraid.

    • Thanks for your perspective and sharing your experience Jonathan; “interesting” is a good choice of words. I agree that these things happen in every field, no question, and there is always resistance whenever existing problems are pointed out, sometimes ferocious resistance. There are some circumstances surrounding this one however that up the ante a fair bit, which I may very well get into later.

  3. Jim –
    I second all of Jonathan Jones’ comments.

    I have a further question. You mention converting basal area increment into ring widths using geometry. Looking at e.g. this picture , it’s apparent that trees do not grow uniformly in all directions; certain directions seem to be favored for growth, and cross-sections are not perfect circles. With two cores per tree, one is sampling the growth at four radial angles. I’m wondering whether you’ve investigated the effects of sampling the varying ring width, on the temperature reconstruction.

    • Thanks Harold. Yes, tree boles are often not perfectly circular, often tending toward an ellipse, especially so near treeline where wind and consequent reaction wood are in action. However, that effect is likely to be very small relative to the issues I’m addressing. One could instead model the bole area as an ellipse, assuming the two cores were taken at right angles to each other (which is always my practice in the field but with data obtained from the ITRDB, that information’s not provided). Regardless, based on my analysis of a number of other, potentially important factors, it likely makes no difference to the basic results of inability to return the correct long term trend.

    • Thanks for the reply, Jim. As an outsider, one sees a potential problem and wonders whether it has been thought through, or assumed away. Good to hear that this is not a significant error source.

    • Your question’s actually real important because it points up the fact that almost no immediately obvious source of error is responsible for producing these consistent mis-estimates, which in turn strongly indicates that there is something fundamentally wrong with the mathematical assumptions of the RCS method.

    • Based on my (minimal) experience on Almagre (cf the Almagre Adventure and in particular Tree #31. Probably directly google-able 😉 )… BCP’s at least can’t be modeled as ellipses. And samples within a few inches of one another produced a 300% data variation. This for a tree in the ITRDB.

      My conclusion: on top of everything else, there’s simply huge uncertainty.

  4. > Being inconveniently right is rarely the route to fame and glory I’m afraid.

    I’m afraid the route of fame and glory, like any business venture, nowadays requires op-eds, press releases, and many other forms of social engineering. Even then, the route comes with a gamble, since our results need to serve someone else.

    I doubt fame and glory is worth our only life on Earth, but others may disagree about that.

Have at it

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s