An article appearing in JAMA Neurology links exposure to certain environmental toxins, like pesticides to Amyotrophic Lateral Sclerosis (ALS or Lou Gehrig’s disease). While I could spend these 150 seconds talking about whether or not we should run home and clean all the Round-Up out of our garage, I’d like to take this chance to talk about 3 methodologic issues a study like this brings to the fore. For the video version of this post, click here.

But first, the details:

Researchers from the University of Michigan performed a case-control study of 156 individuals with ALS and 128 controls. They administered a survey, asking about all sorts of environmental exposure factors, and, importantly, they drew some blood to directly measure 122 environmental pollutants. The bottom line was that there did seem to be an association between some pollutants (like pentachlorobenzene – a now-banned pesticide) and ALS.

So – on to the three issues.

Number 1 – multiple comparisons. As I mentioned, the authors looked at over 100 pollutants in the blood of the participants. Given no effect of the pollutants, chance alone would leave you with several apparently statistically significant relationships.  In fact, a robust demonstration of the multiple comparisons problem is that lead exposure, in this study, was quite protective against ALS. This is not biologically plausible, but reflects that multiple comparisons can cut both ways – it can make measured factors seem to be positively, or negatively associated with the disease. Indeed, several pollutants seemed to protect against ALS.

The authors say they account for multiple comparisons, but I’m not sure this is true. In their statistics section, they write that they used a Bonferroni correction to lower the threshold p-value (from the standard 0.05 to 0.0018 to account for all the comparisons). But they never actually do this.  Rather, they report the odds ratios associated with the various pesticides and just don’t report the p-values at all, except in multivariable models where the Bonferroni correction isn’t used.

Number 2 – the perils of self-reported data. The survey exposure data – questions like “do you store pesticides in your garage?” and the measured blood data were hardly correlated at all. This should be read as a warning to anyone who wants to take self-reported exposure data seriously (I’m looking at you, diet studies). When in doubt, find something you can actually measure.

And Number 3 – the lack of variance explained. Studies like this one that look at risk factors for an outcome are building models to predict that outcome. The variables in the model are things like age, race, family history, and the level of pentachlorobenzene in the blood. It’s a simple matter of statistics to tell us how good that model fits – how much of the incidence of ALS can be explained by the model. We almost never get this number, and I suspect its because you can have a highly significant model that only explains, say, 1% of the variance in disease occurrence. It doesn’t make for impressive headlines.

So while we haven’t learned which, if any, organic pollutant causes ALS, hopefully we’ve learned something about the perils of risk factor research.