What to know
Results from observational studies are more likely to be affected by various forms of bias than are results from randomized control trial studies (RCTs). Therefore, results from observational studies can be more difficult to interpret. Bias can be reduced through careful study designs and analyses of data collected. Observational studies of flu vaccine effectiveness are subject to at least three forms of bias: confounding, selection bias, and information bias.
Confounding
Confounding is when the effect of vaccination on the risk of the outcome being measured (e.g., flu-related hospitalizations confirmed by testing) is distorted by another factor associated both with vaccination (the exposure) and the outcome. In RCTs, factors associated with exposure and outcomes can be evenly distributed between vaccinated and unvaccinated groups. This is not always true in observational studies. For example, chronic medical conditions can confound the association between flu vaccination and hospitalization with flu in observational studies. Chronic medical conditions increase the risk of flu-related hospitalization, and vaccination is more common among people with chronic medical conditions. Therefore, the presence of a chronic medical condition in a study participant is a potential confounding factor that should be considered in analysis. This is an example of confounding by indication because those at greatest risk for the outcome being measured (i.e., flu-related hospitalization) are targeted for vaccination, and therefore, they are more likely than those without a chronic medical condition to receive a flu vaccine. Not adjusting for confounders can bias the vaccine effectiveness estimate higher or lower than the true estimate. In the example given, the vaccine effectiveness estimate could be biased lower, or towards lower effectiveness.
Selection bias
Selection bias occurs when people with the outcome being measured by the study (e.g., flu virus infection) differ from people who do not have the outcome in a systematic way. In observational studies of flu vaccine effectiveness, people with and without flu may have different likelihoods of being vaccinated, and this can bias the estimate of vaccine effectiveness. For example, people who visit their health care provider in outpatient settings (e.g., clinics and urgent care) may be more likely to be vaccinated than people who do not go to a provider for care as often. If controls are selected from a different population than the cases (e.g., cases are from a health clinic and controls from a community sample) with different health care seeking behaviors, selection bias related to health care seeking (and the likelihood to be vaccinated) may be introduced. The test-negative study design minimizes selection bias related to health care seeking by enrolling patients who seek care for a respiratory illness. This study design is used by many researchers globally, including CDC-funded networks that measure vaccine effectiveness.
Information bias
Information bias occurs if exposures or outcomes are based on different sources of information for people with and without the disease of interest. For example, if researchers obtain information on vaccination for children with flu from vaccination records but ask parents of children without flu if the child was vaccinated, this difference in data collection procedures could bias the results of the study.