Trying to preserve the integrity of the trial is a fool’s errand.

Jason Santosuosso has been in biotech for decades. He has founded and sold companies in that space. So he knew what he was getting into when he volunteered to be part of Pfizer’s phase 3 coronavirus vaccine trial.

“At that point I was like, well, you know, I'm really happy to do this more for my family than for anybody I wanted to basically be able to get into the trial so that I could tell them and encourage people to get it, because I knew a lot of people that were afraid of this vaccine.”

So he enrolled. He went to his local medical center and spent hours in the initial visit doing consent, filling out paperwork, and finally – getting that first shot. He had multiple visits as a study participant. Multiple blood draws, nasal swabs. He had to complete an e-diary about his symptoms. It’s onerous. But he figured it was worth it. Yes, because he was contributing to science but also because there was a sense, according to him, that the placebo group would get priority for the vaccine.

“One of the kids that had written on the study, he had said, well, you know … if you didn't get the vaccine that you'd probably be back in here by January to get it…. If it was earlier, you know, I'd be eligible to actually get vaccinated. And I think that that's what the study people thought.”

For many clinical trials, patients who are randomized to get placebo are given the option to get the study drug when the trial is over. The Pfizer trial, though, is not over. Yes, there is an emergency use authorization but that’s not full FDA approval. And the trial is supposed to last two years according to the protocol. Nevertheless participants feel slighted that, after what they went through to generate the data that led to the EUA, they may have to wait in line while others benefit from the highly effective vaccine first.

Should individuals randomized to placebo in the vaccine trials be first in line for the vaccine after an EUA?

This question has been bubbling through social media, and has made it all the way to the pages of both the New England Journal of Medicine and the New York Times.

The general sense is that, sure it would be great to give the placebo participants vaccine but it will cost us in terms of answering some key scientific questions. 

But is that really the case? I’m actually not so sure. Not because the trial wouldn’t be compromised by giving placebo participants vaccine – but because the trial may be compromised already.

I think the best way to think about this is to imagine a few options – policy choices, call them – and then think through how each would effect a specific unanswered scientific question. I’m going to use the Pfizer trial as an example here, but I think this works for the other vaccine trials as well. So far, the data we have from the Pfizer trial has shown us that the vaccine is highly effective at preventing COVID-19 within a few months of vaccination. We also know that the rate of early side effects is acceptably low.

Here are three big scientific questions we still need answers to.

1.  Are there long-term side effects of the vaccine?

2. Does protection wane over time? Will we need yearly boosters?

3. Does the vaccine prevent asymptomatic infections? In other words, does it only protect you or does it protect people around you as well?

Let’s look at the options Pfizer has on the table and how these questions would be affected.

If you really wanted precise answers to these questions, you’d need to force everyone in the placebo group to not get vaccinated no matter what. This is clearly unethical – all trials allow people to drop out at any time. But yes, if you could somehow do this, you could answer those big three questions.

But let’s stay in the realm of reality.

The policy Pfizer seems to be going with looks like this: Th trial remains blinded – participants don’t know if they got placebo or not. They are followed over time. When vaccines become available at-large for the risk group of the participant, the participant can get vaccinated within the trial.  

For many relatively healthy participants, that may be the early summer or even later. The argument here is that that’s the soonest they could get the vaccine if they hadn’t participated in the trial, so at least they aren’t being harmed by being in the trial. But is our goal really just to make trials not actively harmful to participants? Can’t we do better than that?

Like all ethically-conducted trials, a participant can drop out of the Pfizer trial at any time for any reason. So under this plan, we’d expect a portion of people to drop out of the study whenever they think they could get a vaccine. This is going to happen no matter what – here’s Jason Santosuosso again:

“I think, personally, I would have a moral obligation to drop out, out of rebellion... That leaves 22,000 people vulnerable to getting Covid.”      

Now, those people could agree to continue to provide data to Pfizer, about infections and whatnot, but once placebo patients start getting vaccinated, the benefits of randomization are lost. Not just because you lost sample size or something. It’s really because people will not drop out randomly. My guess is that higher risk people would drop out preferentially.

Moreover, more people are likely to drop out of the placebo group than the vaccinated group, even though this is a blinded trial. Why? We know these vaccines have side effects like fever and myalgias that are pretty common. Participants are trying to use these symptoms to figure out into which group they were randomized.  In fact, Jason Santosuosso is pretty sure he knows which group he was in:

“I'm like 90% certain, I got the vaccine because I had some pretty serious side effects from it.”

As placebo participants, and especially higher risk placebo participants drop out, weird effects will emerge.

 The placebo group remaining will be healthier but may also engage in higher risk activities, you might see the rate of infection going up in the placebo group while the severity of infection goes down.

But how will differential drop-out affect the big three questions?

Well, in terms of long-term side effects, it will be pretty bad. Think of a relatively common bad thing, like a heart attack.

The yearly incidence of MI in the US is 600 per 100,000.  That would translate to 264 out of the 44,000 people in the Pfizer vaccine trial who we might expect to have a heart attack in the next 12 months, but be careful. Trial participants are usually healthier than the general population. The true number is likely to be lower. Still – there will be heart attacks. How do we know if they are due to the vaccine or not? We compare the vaccinated group to the placebo group. But once placebo people get vaccinated, they are no longer good controls. Worse, even restricting the analysis to placebo participants who didn’t go and get a vaccine we are still in trouble because people don’t drop out at random. The placebo group will no longer be a random sample of the study population. They’ll be healthier, I think, meaning less heart attacks, which will make it look like the vaccine causes heart attacks. Not good.

The situation is even worse for rare adverse events. Take Guillain-Barre syndrome. That has an annual incidence of 1 in 100,000 people in the US. Even if the vaccine doubles the rate of GBS, we are not going to see it in a trial of 44,000 people even if we did force people to stay in the trial. We can only pick up stuff like this with vaccine surveillance during the large-scale rollout, which is exactly what is happening.

OK so the status quo doesn’t help us with adverse event detection that much.

What about question 2 – durability of protection?

Look at the cumulative case graph from Pfizer.  If, like 6 months from now, the infection rate in the vaccinated group starts going up, we know that the vaccine is wearing off.

Or do we?

Remember that as time marches forward, people are dropping out of the study to get the vaccine. Now, if you don’t follow-up with those people, you are left with a population of people who now suspect they were vaccinated or who are healthy enough that they don’t really care about getting vaccinated. That could have weird effects like the people staying in the trial engaging in risky behavior, driving infection rates up. Better to follow-up with the people who drop out to get the vaccine elsewhere.  Then you’d see infection rates drop in the placebo group, so your efficacy estimate is off, but at least you can see if they start to increase again at some time point after vaccination, hinting at waning efficacy.

But you know what, you don’t need a randomized trial for that. You can just follow any old group of vaccinated people and we’re about to have tens of millions of them.

What about asymptomatic infections? The way Pfizer is figuring this out is pretty clever.

They look at antibodies to COVID-19 in the blood. Their vaccine only generates antibodies to the spike protein, but true infection generates antibodies to a bunch of other coronavirus proteins as well – so they will know who has been infected (even if they are asymptomatic) over the course of the trial. They almost certainly have enough of this data at this point to tell us if the vaccine prevents asymptomatic infection early – great. But for the same reasons discussed above, due to differential dropout they will not be able to clearly prove that it prevents asymptomatic infections in the long-term.

So what if you just vaccinate everyone right now?  Unblind the trial – tell people what they got – and give vaccine to the people in the placebo arm. How does this affect the big 3 questions? 

Long-term side effects? Still in trouble.

Not enough people in the trial anyway for rare side-effects. Common side effects will occur in both groups. There’s a chance you might be able to use a time-from-vaccination metric to determine if some events are happening in temporal relationship to the vaccine, but this will work best for early side effects and we already have data on that.

Long term protection? Well, under this mechanism in theory everyone is staying in the trial – a major advantage. So, if efficacy wanes, we’d see infections start to tick up in the vaccine group while they remain really low in the originally placebo-but-just-vaccinated group. This will depend a lot on that time differential in vaccination between the placebo and vaccine groups, but there’s a chance we could pick up on this. Also, since we’d lose blinding, people will know if they just got the vaccine or if they were vaccinated a while ago, and this might lead to differences in behaviors between the groups that would mess up this analysis.

Asymptomatic infection rate? Again, our best data would be from the time before the trial was unblinded, which we already have. After the big reveal, we’ll more or less have to compare asymptomatic infection rates with the rate in the general population.

A third idea has been proposed – a blinded crossover study. In this design, everyone in the study gets two more shots.

The vaccine group gets two placebo shots, the placebo group gets two vaccine shots. But they still don’t know which group they were in (of course, they may have a decent guess). The advantage here is that everyone stays in the study, so you minimize that differential dropout that we discussed.

Let’s go through the questions.

For side-effects, we’re still pretty stuck. Once everyone in the trial receives the vaccine, the rate of long-term side-effects is going to be hard to tease out – sure the vaccine group has a few months of extra time, so you could imagine you could see a difference here, but particularly for rare events you’re unlikely to see anything.

For long-term protection, a crossover design could work. Like the policy of just unblinding and vaccinating everyone, if vaccine efficacy fades over time, we’d see the vaccine group (who were vaccinated early) start developing COVID cases faster than the placebo group (who got vaccinated later).  The advantage with blinding is that people’s behaviors will be more similar than in option 2.

Asymptomatic infection? I assume the study would continue to collect blood on everyone, though again you’d basically lose your control asymptomatic infection rate and thus be stuck comparing the whole trial to the general population.

So here’s my point. There are some really critical questions left to be answered about these vaccines. But I’m not sure the randomized trials can answer them.  The horse is out of the barn. The genie is out of the bottle. Now that there is an EUA in place, dropout and worse, differential dropout is imminent. Trial participants will get the vaccine one way or another. And under those circumstances, our ability to make good inferences is quickly lost.

Yes, there are statistical tools to deal with stuff like this – inverse probability weighting and whatnot – but I’m not sure they engender the same confidence in results as a solid randomized trial does.

As such, it strikes me that the best option is to do right by these volunteers. They did their part. They deserve to be vaccinated immediately. I’m ok with an open-label or the blinded cross-over design. But it should happen. The benefit to society keeping them unvaccinated is really not that high given that the trial will imminently be compromised by differential dropout.

To answer the outstanding questions we can use large, population-scale data and historical controls and all those imperfect statistical tools. We can also conduct new trials where people consent knowing what they are getting into. They can agree, during the consent process, not to seek out vaccination elsewhere. They can’t be held to that agreement, of course, but a trial designed with that specific aim could recruit people who want to help answer the outstanding questions.

Trial participants, thank you. Take a bow. And take your shot.

This commentary first appeared on medscape.com.