Severe analytical problems in dendroclimatology, part seven

This is a long post with many graphical results, and is the most important one in this series to date.  The gist of the types, and magnitudes, of the problems introduced by the detrending step are presented here.  Given the general lack of interest this may also have to be the final installment, even though I probably have as much or more to say as I already have.  It’s just not worth the time when so few are interested in what I consider to be an extremely important and yet overlooked issue,  and still others simply want to aggravate.

In part four of this series I discussed the kinds of trend estimation errors that RCS would produce when standard practices are followed. In this post I’m going to elaborate on that, and also on the topic raised in parts five and six regarding the importance of the ring response variable used, and then discuss a couple of other important points. These evaluations require using the full model, not the abbreviated one linked to in part five, because the former is capable of testing far more complex situations.  It also performs calibration and prediction of the climate state itself, from the tree ring chronology. The difference between the two doesn’t matter there, but it can in more complex situations and so it’s generally best to present the actual reconstruction estimates as things get more complicated.

In part four, I standardized the chronologies to the climate variable so I could easily graph them together on the same scale. I used the ratio of the means of the two variables to do that, but there are other ways to do it, and these could potentially make a difference in the results. One way would be by using the difference in means; another (and very common) way would be to compute Z scores for each, which automatically puts them on the same scale.

When ring width is the response variable (and hence the predictor of the climatic state), with no biological trend in the rings (but with a geometric effect) here’s what results, using the same “best case” conditions described in part four, and standardization by the ratio of the means, as used there:

Graph1a If I instead standardize using the difference of the means, I get:

Graph1b

If standardization is instead by Z scores (subtraction of mean followed by division by the standard deviation) we get this:

Graph1cThus, the method used to relate the chronologies to the climate variable has almost zero effect on the results. In part four I showed results when three other variations from ideal conditions were instituted, but without really drastic deviations (such as for example, nonlinear relationships between climate and ring response).

I’ll address the important issue raised in parts five and six: what happens if we remove the purely geometric effect induced by the use of ring widths as the response variable, by instead using ring areas? Does that solve the problem? Here’s a repeat of the first three graphs above, using ring widths areas as the response variable:

Graph7aGraph7bGraph7c

Once again the three standardizations give nearly identical results. These results are clearly better than those obtained using ring widths, especially when climatic trends are negative, so this is definite progress: eliminating the purely geometric effect does indeed provide a clearly better estimate of the long term trend, at least under these conditions. It’s still not perfect however, especially for negative trends, where a mis-estimate of almost one unit over the 460 year time-span still exists, and remember, this is the best case condition being evaluated here, so results are unlikely to get better as we deviate from them.

I’ll  do that by first changing the age structure, by forcing all trees to begin their lives before the 2/3 point of the full timespan spanned by the sampled trees, that is, before the 334th year, thereby shifting the sample some towards the more even aged end of the spectrum.  Using standardization by ratio of the means, this gives:
Graph8aA definite decline in accuracy is seen, relative to the previous graph.  Now I instead institute growth differences between trees, up to +/- 33% of the mean (but with optimal age structure restored).  This results in:

Graph9aIf I combine these last two conditions together, it gives:

Graph10aNow imposing a mildly unimodal biological growth trend, such that trees’ growth rates increase rapidly  gradually from year zero, plateau at a high point in the middle decades, and then gradually decline afterward (but with the age structure and inter-tree growth rate variation restored to their “best case” states), we get:

Graph11aNow combining deviations from the ideal state in all three of these variables gives:

Graph12aThis result could reasonably be described as complete failure–there is clearly almost no ability to predict the actual climatic trend of the past.

A couple of important points here.  First, none of these three non-ideal conditions just added are particularly extreme: the unimodality of the biological effect is very broad and shallow-sloped, and forcing trees to originate in the first 2/3 of the analysis period is a condition far exceeded in many ITRDB tree ring data sets that are much more even aged than this.  The inter-tree growth rate variation is difficult to place in context, but certainly trees growing a maximum of 33% above, or below, the mean growth rate is not unreasonable (and since 33% represents the maximum value, as defined by a uniform distribution, the average departure of the trees from the mean growth rate will be only half that, or +/- 16.7%).  As before, there is no difference if one of the two other ways of standardizing the chronologies to the climate is instead used (results not shown).  The point is that none of these conditions can be considered extreme or outside the realm of reasonable expectation for actual data.  Indeed, they are on the mild side, conservative.

The second point is that this last graph has no geometric component to it–it uses ring areas as the dependent variable.  But that of course is not standard practice, which is to use ring widths.  Here’s what standard practice would result in:

Graph13Now we see that the negative trends are actually estimated so poorly that even the direction is wrong–the method actually predicts a positive trend even though the real trend is very strongly negative.

There are also at least two other very important things to note here, when ring widths are used.  The first is that there is a universal tendency for (1) trend magnitudes to be under-estimated, and (2) the negative trends to be mis-estimated much more severely than are positive trends.  The implications of this are fairly serious.  Clearly, if these tested conditions held over some large area, and there had been a negative climatic trend therein, the RCS method would give a completely misleading trend estimate.  But more generally, the method fails to capture the full magnitude of long term variation present regardless of the trend direction.  This is just the latest in a series of such findings, using highly varied analytical approaches, that come to this same conclusion: tree ring methods badly under-estimate the true amount of variation in past climates at medium to long time scales.  Although the RCS method is certainly an improvement upon the ICS method in this regard under good analytical conditions, it is not much better, if any, when conditions are worse.

The second point is that RCS introduces error to the high frequency and medium frequency variation estimates as well.  The following two graphs show the course of the actual climatic state variable, and then the reconstructed climate estimate, both taken from the previous graph but shown separately for clarity:

Graph14Graph15Notice the high frequency (year to year) variation over time in the first graph, the true climatic state.  It’s constant over time, for both trend directions.  This is as expected, because this variation is defined in the simulations to be so, i.e. drawn from a distribution with a constant standard deviation.  Now look at the variance in the second graph by comparison.  Clearly, the variance itself trends (technically referred to as being “heteroscedastic”, or non-constant), and in a direction that is opposite to that the climate trend itself–when the climate state value is relatively high, the variance is relatively low, and vice-versa.  Thus, existing methods also distort the estimates of year to year (and/or decade to decade) variation, along with the trend.  When ring areas are instead used, this problem is generally reduced, but not eliminated.  As an example, the following result obtains when there is no inter-tree growth variation or biological growth trend, but where all trees originate in the first one-third of the time period:

Graph16Thus, in essence, what the RCS detrending method is doing, especially when ring widths (common practice) are used, is shifting some of the variation that should actually be ascribed to long time scales, into shorter time scales, thereby introducing error into both time scales. That is, estimated trends are diminished relative to actual, while estimated high frequency variations are exaggerated, relative to actual.

What?  By this finding we have taken an already very serious analytical flaw and made it even worse, thereby topping out on the failure meter.  If you can’t use ring widths to estimate long term trends, nor high frequency variations, then what can you use them for?  Fortunately, the high frequency variations can be estimated in other ways that do not distort them badly, and so be salvaged, and so this is what should be done whenever year to year variations are the focus.  Year to year variations are certainly interesting and important, but they tell us nothing about longer term trends of course.

One last, but important, point on the errors in high frequency variations.  All of the results presented to date have been defined by a strictly linear, deterministic relationship between climate and ring response.  The high frequency variations in all cases, even when the estimated magnitude of that variation is distorted by the RCS procedure, as just shown, still always track reality at that scale, directionally: the true values, and the corresponding estimates, go up and down in synchrony.  This is important because it shows that the rings are in fact responding to the climate, and that the RCS algorithm is capable of recognizing this.  Therefore, long term trend mis-estimates do not require a departure from linearity in the relationship between climate and ring response, as per the issues raised by Loehle (2009; described in parts one and two of this series), and can therefore be due to other (analytical)  causes.  This fact has important potential uses when analyzing and diagnosing trends estimated from real data.  Hopefully, more on that point in the future.

About these ads

18 thoughts on “Severe analytical problems in dendroclimatology, part seven

  1. Thought you might like to know that Judith Curry @ Climate Etc has added you to her blog roll today.

    • I may at some point post a statement to the ITRDB forum, like Rob Wilson did recently regarding he and others’ letter to Nature Geoscience. In the end, a blog just might not be the right forum for this stuff. It’s the tree ring community that has to pay attention to it, and they’re not.

  2. Communities have a way of closing in on themselves like the IPCC and our OZ ABC coterie Criticism is coming from the blogs, these days, Climate Audit, WUWT, Chiefio, Climate Etc… which seem to be provoking response even if viewed as nuisance noise. Have U considered doing a post on one of these to by pass gate keeping.?

  3. Jim, I find this blog very compelling. Unfortunately for you, I am not a tree-ring expert (though when I drop them for firewood, I do like to count the rings to see how long the beauty lived before the beetles got em). Your writing style is very clear so that i think that I understand about 50-60% of what you are trying to present. But you are right, If I am interpreting what you are saying correctly, then yes, it is very important for the tree ring community to take notice of this. So probably you will have to present it in another form. Hopefully you will keep this blog going if only for the “occasional whatnot”. If not then know that I have enjoyed your venture immensly.

    Cheers – DeNihilist

    • Many thanks for the kind words DeNihilist. I’m glad that the posts have been helpful to at least some people. I will continue working to get it published, and it’s possible I may keep posting on the topic here, not sure yet. It’s much more enjoyable to post on other topics though.

  4. Jim,

    Just discovered your blog from JC’s blogroll. I’m saddened to hear that you may end this series. Sadly, I’ve not read most of it – I’ve had a brutal few weeks – but appreciate you putting it here. When I catch up on my rest a bit I’ll read it end to end. I’m not a tree specialist. I’m a geologist who’s always had alot of curiosity about dendro.

    Of course, you have to balance how much time you have for this stuff and, of course, as a consumer who benefits from your (unpaid!) work, I encourage you to forge ahead on your somewhat (hopefully less) thankless tasks. It takes time for ideas – new or old – to gain (or regain) currency.

    • Thanks for the encouraging words jim. Time and energy permitting, I will continue to force the issue on this, the only questions are in what fora and at what rate. It’s important to remember here that I have already tried to present this via the “proper” route, by submitting an exceedingly detailed and thorough manuscript to PNAS, only to have it completely brushed off. I can still ramp up the seriousness of this a couple more notches at least, including the details of that review, and if I decide to, I will.

  5. Jim,

    You left out an important point! Running simulations can just be enjoyable and engaging. One can get drawn into that artificial world, systematically exploring it. Perhaps that is because it doesn’t take too much complexity in the underlying models to provide us with twists and surprises–a good thing. And of course simulation can help clarify statistical aspects of the problem under study.

    In my view simulation gives you perspective and insight analogous to how bootstrapping expanded the world of statistics. Not only is a good tool to sharpen focus on the issues you have laid out for now but it also suggests a framework in which ‘how what comes next’ might be taken on. All this is a wordy way to say IMO you’ve made a good investment with you approach to date–hang tight.

    It is also statistics that brings me to an off-thought here: down the road, might there be any merit to simulating forests and aggregates of forests–basic working with composite or spatially varying pdfs–and then looking at the effects of sampling? Maybe life might get interesting at several scales when you add spatial variability or a even just lumpiness into the mix. If existing observation supported/suggested/was consistent with the concept, if sure could be a hoot.

    A nice cohesive piece of work is discussed here and the tone of your blog hits the mark. Thanks for bringing it out.

    Best wishes for the season

    mwgrant

    • Thanks for the good words, I appreciate them. The model can definitely simulate multiple sampling sites having different characteristics, easily. The errors introduced by sampling issues will only add to those I’ve documented here however–they cannot ameliorate them. Reconstruction results can only get worse, not better.

    • Hi Jim,

      Yes, things would get worse from the perspective of the entire population. I was not looking to find a way to ameliorate problems–averaging them away in some sense. Part of the idea was more in the spirit of can sample biases and other sampling factors artificially make a silk purse out of a sow’s ear–an even worse situation–to an investigator. What is a cleaner way to say…what are the potential effects of sample design and field realities? (Wish I said it that way.)

      Thanks for taking the time to answer. Sorry I wasn’t more clear. Try to kept at–if it is worth doing and you can sustain it, then plug at it. Somebody will do it sometime.

    • Thanks for clarifying. One situation where an approximately correct reconstruction result would arise and falsely give the idea that existing methods are not faulty, is whenever there has in fact been no, or very little, long term trend across the area of interest, but that is a different situation from what you are asking about. If there is in fact a trend, there could be some small chance that offsetting errors at different sites would produce a roughly correct reconstruction estimate for the entire area, yes.

  6. Jim thanks for the hard work of putting this together and the interesting posts, and I encourage you to continue working on this. My only advise is don’t get too discouraged by the apparent lack of interest. Silence can either mean being ignored, or it can mean your work is being quietly munched on.

    (Isaac Held’s technical blog is like that too… I’m sure for every comment there’s a hundred reads.)

    • Thanks Carrick, thoughts much appreciated. I’m confident of what I’ve done and will therefore continue to press the issue–just a question of how.

  7. Pingback: Friday links: do ecologists just reinvent the wheel, NSF grant stats, and more | Dynamic Ecology

  8. One of Carrick’s quiet munchers here. I measured an awful lot of tree rings as an undergrad and am slowly but surely working on an analysis & paper while doing a totally unrelated project for grad school. I’m not trying to recreate climate with my tree rings, but to understand how climate influenced growth. A lot of the standard standardizations seemed to remove the variation I was interested in (or turn ARSTAN catatonic when fed my series). This series is helping me understand some of the techniques better and hopefully will improve my own work.

    • Thanks susannah. I haven’t gotten into the several issues in calibration yet, which should or may be more helpful to you. I will, just not sure when. Not sure why ARSTAN might have choked on your data.

  9. It is not a lack of interest; rather, a lack of awareness. I encourage you to continue your efforts.

    Awareness will grow

Comments are closed.