Transition from a Dual-Frame (Cell-phone and Landline) to a Single-Frame (Cell-Phone) Sample Design: Impact on Vaccination Coverage Estimates, National Immunization Survey-Child, 2014-2018

Executive Summary

• The Centers for Disease Control and Prevention (CDC) uses the National Immunization Survey-Child (NIS-Child) to monitor national, state, territorial and selected local area vaccination coverage among young children in the U.S. Since 2011, the NIS-Child has been monitoring coverage using a nationally representative sample of children ages 19-35 months through a random digital dial (RDD) dual- (landline and cell-phone) frame survey. Due to a declining number of landline-only households in the U.S., the NIS-Child discontinued landline sampling in 2018 and shifted to a single-frame cell-phone sampling design to increase efficiency. This paper uses NIS-Child data from survey years 2016 and 2017 to assess the impact of the change from a dual-frame to a single-frame survey on vaccination coverage estimates among children 19-35 months old. Beginning with the publication of the 2018 NIS-Child MMWR, CDC also changed its analytic approach and now calculates vaccination coverage estimates by birth year (birth cohort) rather than by survey year. Therefore, the transition from dual-frame to single-frame design is also evaluated by assessing coverage by age 24 months among children born in 2013 and 2014 compared with coverage by age 24 months among children born in 2015 and 2016.
• Nationally, there were no significant differences in vaccination coverage among children 19-35 months old when comparing single-frame estimates with dual-frame estimates in survey year 2016 or 2017. Similarly, there were no significant differences in vaccination coverage by age 24 months when comparing single-frame to dual-frame estimates among children born in 2013-2014 or 2015-2016.
• Across 29 sociodemographic categories, single-frame vaccination coverage estimates for the combined 7-vaccine series were significantly higher than dual-frame estimates for seven of the comparisons and significantly lower than dual-frame estimates for two of the comparisons in survey year 2016. In survey year 2017, single-frame estimates were significantly higher than dual-frame estimates for two of the comparisons and significantly lower than dual-frame estimates for three of the comparisons. In the birth year analysis, single-frame and dual-frame estimates were calculated by race/ethnicity, poverty level, metropolitan statistical area, and health insurance status for ≥1 dose of measles, mumps, and rubella vaccine (MMR), ≥4 doses of diphtheria, tetanus, and acellular pertussis vaccine (DTaP), and for the rotavirus vaccine series. No significant differences by sample frame were seen for any of the sociodemographic factors for any of the vaccines among children born in 2013-2014 or those born in 2015-2016.
• At the state level, the median difference between dual-frame and single-frame design estimates of vaccination coverage was -0.2 in survey year 2016 and 0.0 in survey year 2017 for the combined 7-vaccine series. The majority of the differences in state-level estimates were between -2.0 and 2.0 percentage points for both years. Results were similar when analyzed by birth cohort, with median differences (dual-frame vs single-frame) ranging from 0 to 0.3 across six groups (three vaccines – MMR, DTaP, and rotavirus vaccine, and two birth year groups – 2013-2014 and 2015-2016).
• As the NIS transitioned from a dual- to single-frame design in 2018, it is important to consider that there may be small changes in vaccination coverage estimates with the change in sampling design, and direct comparisons with coverage estimates from previous years should be made with caution.

Introduction

Since 1994, the NIS-Child has been monitoring vaccination coverage for a nationally representative sample of children ages 19-35 months through a RDD telephone survey12. As landline phone use decreased and cell-phone use increased, the NIS-Child transitioned from a single-frame landline sample in 1994 to dual (landline and cell-phone) sample frame in 2011 to ensure that the survey base reflected the U.S. population34. However, due to the declining number of landline-only households in the U.S. in recent years5, the NIS-Child discontinued landline sampling and shifted to single-frame cell-phone RDD sampling in 2018 to increase efficiency. For example, from 2013 to 2017, the proportion of landline-only households for 19-35 month old children in the United States was small and decreased from 3.3% in 2013 to 1.3% in 2016 (Table 1 [1 page, 508]). This paper uses the NIS-Child data from survey years 2016 and 2017 to assess the impact of a change from a dual-frame to a single-frame survey design on state and national vaccination coverage estimates, as well as coverage in select jurisdictions, among children 19-35 months old. Specifically, we examined whether there are differences in vaccination coverage estimates among children age 19-35 months using single-frame cell-phone sample compared with a dual-frame landline and cell-phone sample nationally, by select socio-demographic characteristics, and by state and select local areas using data from 2016 and 2017 NIS-Child surveys. These years were assessed because they are the most recent years of data with the dual-frame design, and provide an opportunity to observe any potential differences between single- and dual-frame coverage estimates. Beginning with the publication of the 2018 NIS-Child MMWR, CDC also changed its analytic approach and now calculates vaccination coverage estimates by birth year (birth cohort) rather than by survey year. Therefore, the transition from dual-frame to single-frame design is also evaluated by assessing coverage by age 24 months among children born in 2013 and 2014 compared with coverage by age 24 months among children born in 2015 and 2016.

Methods

NIS-Child data from survey years 2016 and 2017 were analyzed to assess vaccination coverage among children age 19-35 months using two different sampling designs: (a) single-frame cell-phone design and (b) dual-frame landline and cell-phone design. Dual-frame estimates have previously been published for 2016 and 201767. To produce single-frame estimates, data for 2016 and 2017 were subset to the cell-phone sample and reweighted. Estimates were weighted separately for single- and dual- frame sample designs to adjust for differences in the probability of selection and nonresponse in each sampling frame.

First, sociodemographic characteristics for children under the single- and dual-frame design in 2016 and 2017 were compared.

Next, national vaccination coverage estimates were assessed by sampling design (single- and dual-frame) and survey year for diphtheria, tetanus, and acellular pertussis vaccine (DTaP) (≥3 doses, ≥4 doses); poliovirus vaccine (≥3 doses); measles, mumps, and rubella vaccine (MMR) (≥1 dose); the primary and full series of Haemophilus influenzae type b conjugate vaccine (Hib); hepatitis B vaccine (HepB) (≥3 doses); hepatitis B birth dose; varicella vaccine (≥1 dose); pneumococcal conjugate vaccine (PCV) (≥3 doses, ≥4 doses); hepatitis A vaccine (HepA) (≥1 dose, ≥2 doses); rotavirus vaccine; and the combined 7-vaccine seriesA.* Estimates by sampling design and survey year were also produced for the proportion of children that have received zero vaccinations.

National coverage for the combined 7-vaccine series was assessed by select socio-demographic characteristics (child's age, sex, race/ethnicity, poverty status, health insurance status, and metropolitan statistical area (MSA) status; mother's age, race/ethnicity, and education) for single- and dual-frame designs.

Finally, state and select local area estimates for the combined 7-vaccine series were assessed for single- and dual-frame designs in 2016 and 2017. The distribution of state-level differences for the combined 7-vaccine series using single- and dual-frame designs were graphed.

The differences between dual- and single-frame estimates and their confidence intervals were reported for each national, state, and selected local area. A confidence interval that includes zero indicates that there is no statistically significant difference detected between the dual- and single-frame estimates. Confidence intervals that do not include zero suggest that coverage is higher or lower based on the dual-frame design (if the difference is positive or negative, respectively).

In the publication of the 2018 NIS-Child MMWR and thereafter, NIS-Child data will be analyzed by birth year (birth cohort) rather than by survey year. Vaccination coverage estimates are calculated using Kaplan-Meier techniques to determine the proportion of children from a particular birth year (or combined set of birth years) who received recommended vaccines by age 24 months. To evaluate the impact of this analytic change, this report also includes comparisons of single- and dual-frame estimates among children from birth years 2013-2014 and 2015-2016 for all recommended childhood vaccines. In addition, single-frame and dual-frame estimates comparing the two birth year groups are provided.

Findings

The estimated proportion of children living outside of a metropolitan statistical area (MSA) was slightly lower in survey years 2016 and 2017 under the single-frame design than under the dual-frame design (11% vs. 12%); there were no other sociodemographic distributions that were significantly different by sampling design in 2016 and 2017 (Table 2 [3 pages, 508]).

Dropping the landline sampling frame did not substantially change national vaccination coverage estimates for any of the 16 vaccination measures examined among children 19-35 months in 2016 and 2017. Of the 16 comparisons in each year, none of the differences were statistically significant (Table 3 [2 pages, 508]). Results were similar when the data were examined by birth year rather than survey year. There were no differences between single-frame and dual-frame national coverage estimates for any of the recommended childhood vaccines for children born in 2013-2014 or for children born in 2015-2016 (Table 4 [508]). Coverage among children born in 2013-2014 was compared to coverage among children born in 2015-2016 separately by sampling frame (Table 5 [508]). Results varied only slightly, with single-frame estimates showing significant differences for poliovirus, the HepB birth dose, 2+ doses of HepA by 35 months, and 2+ doses of influenza, while dual-frame estimates showed significant differences for the HepB birth dose, 1+ dose of HepA by 24 months of age, 2+ doses of HepA by 35 months of age, and 2+ doses of influenza.

Across the 29 socio-demographic factors examined, nine statistically-significant differences in the vaccination coverage rate for the combined 7-vaccine series were observed between the dual-frame and single-frame designs in 2016, ranging from -4.7 to 1.2 percentage points, with the largest estimated difference being among those with mothers age less than 20 years; five statistically-significant differences between the dual-frame and single-frame designs were observed in 2017, ranging from -6.4 to 1.9 percentage points, with the largest difference being among uninsured children (Table 6 [3 pages, 508]). The single-frame estimate was significantly higher than the dual-frame estimate in 2016 for non-Hispanic white children, children whose families were at or above the poverty line, children with Medicaid insurance, children with non-Hispanic white mothers, children whose mothers had some college education but were not four-year college graduates, and children whose mothers were less than 30 years old; the single-frame estimate was significantly lower than the dual-frame estimate in 2016 for children with only private insurance and children whose mothers were four-year college graduates. However, these differences were not observed in 2017. In 2017, the single-frame estimate was significantly lower than the dual-frame estimate for children 19-23 months old and for uninsured children; the single-frame estimate was significantly higher than the dual-frame estimate for children 30-35 months old, children with Hispanic mothers, and children whose mothers had some college education but were not four-year college graduates. In the birth year analysis, no significant differences in coverage with ≥1 dose of MMR, ≥4 doses of DTaP, or the rotavirus vaccine series were found by sample frame among children born in 2013-2014 or those born in 2015-2016, when stratified by race/ethnicity, poverty level, metropolitan statistical area, and health insurance status (data not shown).

At the state and local area level, the median difference between dual-frame and single-frame design estimates of vaccination coverage was -0.2 in 2016 and 0.0 in 2017 for the combined 7-vaccine series (Figure 1 [1 page, 508]). The majority of the differences were between -2.0 and 2.0 percentage points for both years. A scatterplot of state and local area level coverage estimates for the combined 7-vaccine series in 2017 shows a high correlation between dual- and single-frame estimates (r=0.99) (Figure 2 [1 page, 508]). Results were similar when analyzed by birth cohort (data not shown), with median differences (dual-frame vs single-frame) ranging from 0.0 to 0.3 for the six groups (three vaccines – MMR, DTaP, and rotavirus vaccine, and two birth year groups – 2013-2014 and 2015-2016).

Few differences between dual- and single-frame estimates for the combined 7-vaccine series among states and local areas were statistically significant. Of the 61 state and local-area estimates in each year, ten had statistically significant differences between survey designs in 2016, with half of the differences being positive and half being negative (range -5.3 to 6.7). In 2017, three had statistically significant differences between survey designs, with one being a positive difference and two being negative differences (range -2.6 to 2.0) (Tables 7 [5 pages, 508] and 8 [5 pages, 508]).

Conclusions

Vaccination coverage estimates using the single-frame design were similar to estimates from the dual-frame design nationally and by state and select local areas. Nationally, no statistically significant differences in estimates between sample designs were observed for any of the 16 vaccination measures in survey years 2016 or 2017 or in birth years 2013-2014 or 2015-2016. Significant differences were observed for ten state and local areas in survey year 2016 and three state and local areas in survey year 2017, but the median differences were small, with the single-frame estimate only 0.2 percentage points higher at the median in survey year 2016 and no difference observed at the median in survey year 2017.

Statistically-significant differences in the combined 7-vaccine series coverage estimate between the dual-frame and single-frame designs were found for several characteristics, but the characteristics showing differences were not consistent between survey years 2016 and 2017. Across the two survey years, the differences ranged from -6.4 to 1.9 percentage points, with most of the differences within two percentage points.

Reasons for these differences are not clear. In general, single-frame estimates differ from dual-frame estimates in two ways. First, they exclude children from landline-only households and children in households with both landline and cell telephones (dual users) reached by the survey through a landline phone. Second, the subgroups of respondents included in both estimates (households with only cell phones and dual-user households reached by cell phone) are weighted differently in single-frame and dual-frame estimates. Thus, single-frame estimates may differ from dual-frame estimates because: 1) vaccination coverage for children in landline-only households may differ from coverage in households with cell phones; 2) vaccination coverage in dual user households that respond to the survey by landline phone may differ from coverage in dual user households that respond by cell phone; and 3) different levels of bias for single-frame and dual-frame estimates may remain after weighting adjustments that aim to make the samples representative of the child population. However, there is some evidence that NIS-Child coverage estimates from dual users are similar, regardless of whether they are interviewed by landline or cell phone8.

Because landline-only households constituted only a small percentage of the NIS-Child dual-frame sample (e.g., 0.4% in 2017), the potential for differences in single-frame and dual-frame estimates is limited. With half-width confidence intervals of differences between plus or minus 1-2 percentage points (Table 9 [2 pages, 508]), statistically significant differences may not be programmatically important. Moreover, some of the statistically significant differences could be due to chance because of multiple comparisons9.

Other limitations should also be noted. Due to low sample size among specific sub-groups, we were unable to determine characteristics that may help explain differences in coverage estimates between the dual-frame and single-frame designs in these groups. Bias may also remain after weighting adjustments for incomplete data from the sample frame and non-response.

As the NIS transitioned from a dual- to single-frame design in 2018, it is important to consider that there may be small changes in vaccination coverage estimates with the change in sampling design. As a result, assessing trends in coverage or comparing coverage estimates from 2018 data with data from prior years should be done with caution. However, any differences in vaccination coverage between single- and dual-frame designs are expected to be small and decrease even further as landline-only use continues to decline in the U.S.

Authors

Holly A. Hill, James A. Singleton, Laurie D. Elam-Evans, Kimberly Nguyen, Sandy Pingali, Tanja Walker, David Yankey, Zhen Zhao, National Center for Immunization and Respiratory Diseases, Centers for Disease Control and Prevention

Christopher Johnson, Benjamin Skalland, Kirk Wolter, NORC at the University of Chicago

Benjamin Fredua, Leidos Health, Inc.

Acknowledgments:

Xian Tao and Kathleen Santos from NORC at the University of Chicago contributed to data analysis and to drafting earlier versions of the report.