How Flu Vaccine Effectiveness and Efficacy Are Measured

Randomized controlled trials (RCTs)

A randomized controlled trial (or “RCT”) compares outcomes in two groups of volunteers who are randomly assigned to get either a vaccine, any drug, or a placebo (a placebo looks like a vaccine but does not contain any vaccine—often, a shot of saline solution is used). RCTs measure vaccine efficacy. Vaccine efficacy, in this case, refers to the percent reduction in how often flu illness occurs among vaccinated people compared to people given a placebo (i.e., unvaccinated people). RCTs are usually conducted under ideal conditions where vaccine storage and delivery are monitored, and participants are usually in good health or selected for a specific health status. The RCT study design minimizes bias that could lead to invalid study results. Bias is an unintended systematic error in a study, which can include the way researchers select study participants, measure outcomes, or analyze data that can lead to inaccurate results. An RCT, is usually conducted using double-blinded methods, which means neither the study volunteers nor the researchers know whether study participants have been given vaccine or placebo. National regulatory authorities, such as the Food and Drug Administration (FDA) in the United States, require RCTs to be conducted. These studies must show the protective benefits of a new vaccine before the vaccine is licensed for regular use.

Observational studies

Vaccine efficacy, defined above, is determined in RCTs, usually clinical trials. Vaccine effectiveness measures how well a vaccine works in real-world conditions. Differences in real-world conditions compared to the controlled conditions in clinical trials can influence how well a vaccine works. Vaccine effectiveness studies can:

• Be used to determine if people at increased risk of severe flu illness (often excluded from clinical trials/RCTs) respond differently to vaccination.
• Determine if different flu viruses that are circulating and evolving in real-world conditions affect vaccine performance.
• Be conducted to account for how potential differences in vaccine dosing schedules or storage and handling may affect vaccine performance since vaccine dosing schedules or vaccine storage and handling requirements may not be followed as closely in the real world as in clinical trials.

Results from vaccine effectiveness studies are potentially subject to biases that are much less likely to occur in vaccine efficacy studies, like selection bias and confounding, which is why licensure of vaccines depends upon data collected in RCTs.

Once a flu vaccine has been licensed by FDA, recommendations for its "routine" (regular) use are typically made by CDC's Advisory Committee for Immunization Practices (ACIP). For example, ACIP recommends annual flu vaccination for all U.S. residents 6 months and older, with rare exception. So-called "universal vaccine recommendations" introduce ethical challenges in performing RCTs that assign people to a placebo group, which could place them at elevated risk for serious complications from flu. Also, observational studies often are the only option to measure vaccine effectiveness against more severe, less common flu outcomes, such as hospitalization.

What CDC does

CDC has been working with researchers at universities and hospitals since the 2004-2005 flu season to estimate how well flu vaccines work through observational studies using laboratory-confirmed flu as the outcome. CDC's studies are conducted in sites located across the United States and among different age groups to gather more representative data.

Over the past few years, CDC has conducted VE studies using multiple vaccine effectiveness networks. More information on CDC's vaccine effectiveness networks and studies is available at CDC's Influenza Vaccine Effectiveness Networks.