This week, Johnson & Johnson began distributing millions of doses of its coronavirus vaccine in the United States after receiving an emergency use authorization from the Food and Drug Administration. Critical to getting the green light was a trial that Johnson & Johnson conducted to measure the vaccine’s effectiveness.
Efficacy is a crucial concept in vaccine trials, but it is also complicated. If a vaccine is, say, 95% effective, that doesn’t mean that 5% of the people who get that vaccine will get COVID-19. And just because one vaccine ends up with a higher efficacy estimate than another in trials does not necessarily mean it is superior. This is why.
For statisticians, efficacy is a measure of how much a vaccine reduces the risk of an outcome. For example, Johnson & Johnson looked at how many people who got a vaccine got COVID-19. They then compared it to the number of people who contracted COVID-19 after receiving a placebo.
Sign up for The Morning newsletter from the New York Times
The risk difference can be calculated as a percentage. Zero percent means that vaccinated people are at the same risk as people who received the placebo. One hundred percent means that the vaccine completely eliminated the risk. At the US trial site, Johnson & Johnson found the efficacy to be 72%.
Efficacy depends on the details of a trial, such as where it was conducted. Johnson & Johnson conducted tests at three sites: in the United States, Latin America, and South Africa. The overall effectiveness was less than that of the United States alone. One of the reasons appears to be that the South African trial took place after a new variant swept through that country. Named B.1.351, the variant has mutations that allow it to evade some of the antibodies produced by vaccination. However, the variant did not make the vaccine useless. Far from it: in South Africa, Johnson & Johnson’s efficiency was 64%.
Efficacy can also change when scientists see different results. Johnson & Johnson’s vaccine had an 85% efficacy rate against severe cases of COVID-19, for example. This is important to know, because it means that the vaccine will prevent many hospitalizations and deaths.
When scientists say that a vaccine has an efficacy of, say, 72%, that’s what is known as a point estimate. It is not an accurate prediction for the general public, because trials can only consider a limited number of people; in the case of the Johnson & Johnson trial, about 45,000 volunteers.
The uncertainty around a point estimate can be small or large. Scientists represent this uncertainty by calculating a range of possibilities, which they call a confidence interval. One way to think of a confidence interval is that we can be 95% confident that the efficacy lies somewhere within it. If scientists established confidence intervals for 100 different samples using this method, the efficacy would be within confidence intervals for 95 of them.
Confidence intervals are narrow for trials where many people get sick and there is a big difference between the results in the vaccinated and placebo groups. If few people get sick and the differences are smaller, the confidence intervals can explode.
Last year, the FDA set a target for coronavirus vaccine trials. Each manufacturer would have to demonstrate that a vaccine is at least 50% effective. The confidence interval should reach a value not less than 30%. A vaccine that meets that standard would offer the kind of protection found in flu vaccines and therefore save many lives.
So far, three vaccines, made by Pfizer and BioNTech, Moderna and Johnson & Johnson, have been licensed in the United States after their trials showed that they exceeded the FDA threshold. AstraZeneca and Novavax, which have ongoing trials in the US, have published efficacy results from studies in other countries. Meanwhile, the makers of the Sputnik V vaccine have published results based on their trial in Russia.
For various reasons, it is not possible to make an accurate comparison between these vaccines. One vaccine may have a higher point estimate than another, but their confidence intervals may overlap. That effectively makes your results indistinguishable.
To complicate matters, the vaccines were tested in different groups of people at different stages of the pandemic. Furthermore, its effectiveness was measured in different ways. Johnson & Johnson’s effectiveness was measured 28 days after a single dose, for example, while Moderna’s was measured 14 days after a second dose.
What is clear is that the three vaccines licensed in the United States, made by Johnson & Johnson, Moderna, Pfizer and BioNTech, greatly reduce the risk of contracting COVID-19.
Furthermore, all vaccines appear to be highly effective against more serious outcomes such as hospitalization and death. For example, no one who received the Johnson & Johnson vaccine had to go to the hospital for a COVID-19 infection 28 days or more after receiving an injection. Sixteen people who received the placebo did. This translates into 100% efficacy, with a confidence interval of 74.3% to 100%.
A clinical trial is just the beginning of any vaccine research. Once it enters widespread use, researchers continue its performance. Instead of efficacy, these scientists now measure efficacy: how much the vaccine reduces the risk of a disease in the real world, in millions of people instead of thousands. Early studies on the efficacy of coronavirus vaccines confirm that they provide robust protection.
In the coming months, the researchers will monitor this data to see if it becomes less effective, either because immunity from the vaccine declines or because a new variant emerges. In either case, new vaccines will be created and manufacturers will provide new measures of their efficacy.
This article originally appeared in The New York Times.
© 2021 The New York Times Company