A new study makes sense, but the analysis is missing some important controls.

It’s football season, but that’s not the only grand American tradition going on right now. No, we also seem to be in the “evaluating old COVID  policies” season – as papers looking at the outcomes of decisions public health officials made in 2020 and 2021 are bursting forth like the Wolverines from Michigan Stadium.

Last week we looked at the impact of masking in public schools, and this week, we’re looking at, appropriately enough, attendance at NFL games. That’s right. It’s time to do some Monday morning quarterbacking.

We’re talking about this study, appearing in JAMA Network Open which looked at the 2020-2021 football season.

The key to the study is the fact that teams did different things over the course of the season with regard to how many people they let in the stadium.  You had teams like the Vikings and the Chargers who never let any fans in the stadium, and teams like the Cowboys who had more than 20,000 fans every single game.

Did it matter? Was restricting attendance a good idea? The authors decided to see if COVID rates went up in the county where the game was held in temporal proximity to game time. They looked at 7-day windows after the game (which are probably too short to see an effect of game-day transmission), as well as 14- and 21-day windows.

But hold on - what does it mean to say that COVID rates went up? This is where the study starts to feel more like a discussion on some stock technical analysis forum than it does like a standard medical paper.

The authors tabulated case rates on a daily basis based on publicly available records. They used some statistical tweaks to account for the expected differences in reporting on weekends and the occasional data dump. They then used several different outlier detection techniques to find days where more cases than expected happened.

The simplest technique used a moving average approach.  Here’s an example using the Buccaneers. The black line shows the 21-day moving average case rate. 

You can see how it climbs from September to January – that’s the alpha variant wave – how quaint – and then starts to descend. The dots reflect individual daily case rates, and, when they are significantly higher than the moving average, they are coded as outliers – yellow in this case.

In other words, the operational definition of “more cases than we’d expect” is based on the assumption that case-rates should rise and fall with some smoothness. I’m not entirely confident in this approach – but I’ll get to that in a minute.

Ok, assuming we believe that those yellow dots are important signals, we can ask a straightforward question. Would games with high attendance have more yellow dots following the game than games with low or no attendance?

The authors find that they do. In both 14- and 21-day windows following attended games, there were about 30-40% higher rates of “outlier days” compared to unattended games.

As a partial negative control, the authors report no relationship using a 7-day window following the game, which feels like what we’d expect epidemiologically given the incubation period of COVID but for the life of me I can’t find the actual numbers in the manuscript or supplement. For games that had more than 20,000 fans, spikes in covid cases doubled, and for games with less than 5000, there was no difference in spike rates providing some sense of dose-response.

This was true for the county where the NFL game was held, as well as contiguous surrounding counties.  I would have loved to see another negative control here looking at remote counties, for example, but no such luck.

At face value, this study provides some data to suggest that big sporting events increase local COVID rates. That, of course, makes sense biologically. You’ve got a bunch of people in close proximity screaming – it’s not exactly surprising. 

But that said, most of these stadiums are outdoors – there’s a fair amount of airflow there.  It’s not an open and shut case.

And this method of flagging outliers got me a bit concerned. One thing I noticed was that there were no outliers in the other direction – no “surprisingly low” case rates.

It’s not that I think massive attendance at NFL Games would reduce COVID rates, but I’m worried that the very choppy daily COVID case data has a risk of generating “outliers” that aren’t truly meaningful.  I mean, look at the graphs – it’s not like the outliers are all one day after another. Why would a game lead to a spike in cases on Tuesday but not Wednesday?  Could this all be noise in the system, and if so, would negative outliers cancel out the positive ones? I spoke to study statistician Justin Kurland about this issue who confirmed to me that there was no analysis of negative outliers, but that very few were detected in any case.

I’m left not entirely sure what to make of the study. Moving forward, if there were a highly infectious disease without a known cure for which no vaccine existed (as was the case for much of the 2020 football season), you’re still not getting me to pile into a stadium.  But I don’t think this study, despite its clever analysis, provides the solid proof that supports my apprehension.

A version of this commentary first appeared on Medscape.com.