What to know
The measurement of flu vaccine efficacy and effectiveness can be affected by virus and host factors as well as the type of study used. Therefore, vaccine efficacy/effectiveness point estimates vary among published studies.
Background
Different outcomes evaluated in vaccine effectiveness studies
Vaccine effectiveness is a measure of how well flu vaccines work among different groups of people, in different settings, and in different real-world conditions (as opposed to randomized controlled trials (RCTs) or "clinical trials). Public health researchers evaluate the benefits of vaccination against illness of varying severity, from mild-to-moderate illness resulting in a doctor visit to more severe illness resulting in hospitalization, ICU admission, and even death. Different studies are used to measure different outcomes, and estimates of vaccine effectiveness may vary based on the outcome measured (as well as other factors, such as the population studied).
Virus factors
Vaccine effectiveness may vary based on how similar or different the vaccine virus is to circulating flu viruses. The more similar the flu vaccine is to circulating viruses, the more likely it is to offer robust protection against flu illness and complications. However, research has shown that even when the viruses are very different, vaccination can still provide as much as 30% reduction in risk123456. (More information is available at How the flu virus can change: "Drift" and "Shift".)
Host factors
Host factors refer to characteristics of the vaccinated person, including, for example, their age, underlying medical conditions, history of prior influenza virus infection, and prior flu vaccinations. All of these factors can affect how well vaccines work.
Study design factors
RCTs provide the most reliable results because they are less susceptible to selection bias and confounding. However, RCTs may be difficult or unethical to conduct when vaccination is universally recommended in a population or for more severe outcomes that are less common. There are several observational study designs; however, many flu vaccine evaluation programs currently use the test-negative design. In the test-negative design, people who seek care for an acute respiratory illness are enrolled at care settings (such as outpatient clinics, urgent care clinics, emergency departments, or in-patient settings), and information is collected about the patients’ flu vaccination status. All participants in a test-negative design study are tested for flu using a highly specific and sensitive test for influenza virus infection, such as reverse transcription polymerase chain reaction (RT-PCR). The ratio of vaccinated to unvaccinated persons (i.e., the odds of flu vaccination) is compared among patients with and without laboratory-confirmed flu. In this way, a test-negative design study estimates VE by comparing vaccination rates among persons with confirmed flu illness (also called “cases”) against vaccination rates among persons with similar illness who do not have flu (also called “controls”) based on laboratory tests. The test-negative design reduces selection bias due to health care seeking behaviors.
Specificity of the outcome
Factors related to measuring specific versus non-specific outcomes
For both RCTs and observational studies, the specificity of the outcome measured in the study is important. Non-specific outcomes, such as pneumonia hospitalizations or influenza-like illness (ILI), can be caused by influenza virus infections or infections with other viruses and bacteria. Vaccine efficacy/effectiveness estimates for non-specific outcomes are generally lower than estimates made for more specific outcomes, depending on what proportion of the outcome measured is attributable to flu. For example, a study among healthy adults found that inactivated flu vaccine (i.e., a flu shot) was 86% effective against laboratory-confirmed flu, but only 10% effective against all respiratory illnesses in the same population and season7. When assessing vaccine efficacy/effectiveness, the most specific outcome is laboratory-confirmed influenza virus infection by RT-PCR or viral culture.
- Price AM, Flannery B, Talbot HK, et al. Influenza vaccine effectiveness against influenza a(h3n2)-related illness in the United States during the 2021–2022 influenza season. Clinical Infectious Diseases. 2022. doi:10.1093/cid/ciac941
- Tenforde MW, Kondor RJG, Chung JR, et al. Effect of Antigenic Drift on Influenza Vaccine Effectiveness in the United States-2019-2020. Clin Infect Dis. 2021;73(11):e4244-e4250. doi:10.1093/cid/ciaa1884
- Tenforde MW, Talbot HK, Trabue CH, et al. Influenza Vaccine Effectiveness Against Hospitalization in the United States, 2019-2020. J Infect Dis. 2021;224(5):813-820. doi:10.1093/infdis/jiaa800
- Campbell AP, Ogokeh C, Weinberg GA, et al. Effect of Vaccination on Preventing Influenza-Associated Hospitalizations Among Children During a Severe Season Associated With B/Victoria Viruses, 2019-2020. Clin Infect Dis. 2021;73(4):e947-e954. doi:10.1093/cid/ciab060
- Flannery B, Kondor RJG, Chung JR, et al. Spread of Antigenically Drifted Influenza A(H3N2) Viruses and Vaccine Effectiveness in the United States During the 2018-2019 Season. J Infect Dis. 2020;221(1):8-15. doi:10.1093/infdis/jiz543
- Kim SS, Naioti EA, Halasa NB, et al. Vaccine effectiveness against influenza hospitalization and emergency department visits in 2 A(H3N2) dominant influenza seasons among children <18 years old-new vaccine surveillance network 2016-2017 and 2017-2018. J Infect Dis. 2022;226(1):91-96. doi:10.1093/infdis/jiab624