Purpose
- The Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) has produced more than 200 reports from question evaluation projects.
- Learn more about the in-depth research, reporting, and analysis from selected CCQDER projects.
Telemedicine availability during COVID-19
Background: To provide the public with timely data about health-related pandemic effects, the National Center for Health Statistics (NCHS) expanded the mission of its Research and Development Survey (RANDS) program. RANDS—the center's methodological research survey—provided experimental estimates of a limited number of health-related outcomes during the pandemic.
NCHS's Division of Research and Methodology consulted both internal and external stakeholders when developing the RANDS during COVID-19 questionnaire. Through this process, RANDS during COVID-19 worked to provide timely information about telemedicine availability and use before and during the pandemic.
Purpose: While researchers tested the initial draft question about current telemedicine availability, the text of the final question was not tested before use. The question's language had been changed during the Office of Management and Budget clearance process. To get information about how respondents interpreted the final question language, CCQDER added an open-ended probe to the first round of the RANDS during COVID-19 questionnaire.
Findings: Analysis indicated that older respondents and respondents with lower levels of educational attainment appeared to have higher levels of potential measurement error. In addition, there may have been cultural differences in how the telemedicine availability question was understood that needed to be addressed to produce more consistent and valid survey data.
Citation: Scanlon, P. Mixed method evaluation of the RANDS during COVID-19 telemedicine availability question: Results from the first two rounds of RANDS during COVID-19. 2022.
Study year: 2020
COVID-19 survey questions
Project: Cognitive testing evaluation of survey questions on COVID-19
Purpose: This cognitive interview evaluation supported federal surveys that had added (or intended to add) questions related to COVID-19. The study looked at the validity of the COVID-19 questions. This helped survey data analysts understand what constructs the questions captured based on observed patterns of interpretation by respondents. The study also explored the question-response process to identify possible sources of response error.
Findings: The COVID-19 pandemic created a contextual backdrop that influenced respondents' interpretations of the questions, sometimes in unexpected ways. The evolving pandemic timeline created challenges for answering some of the questions because respondents had to decide which phase of their experience in the pandemic to consider. In addition, while some new pandemic-related terms were becoming more common, shared understanding of their meaning was not widespread.
Citation: Willson S. Cognitive testing evaluation of survey questions on COVID-19. 2021.
Study year: 2020
Opioid-related survey questions
Project: 2019 evaluation of opioid-related questions for federal household surveys
Purpose: This project examined the performance of opioid use, impairment, misuse, and disorder questions intended for population-based federal surveys. The project included a large, nationwide cognitive interviewing study (in English and Spanish) and a companion pile sorting activity. The interviews focused on the ways in which respondents interpreted the various questions.
Findings: Interviewers were able to determine false-negative and false-positive responses as well as reasons for that error. Respondents interpreted questions and formed responses to the various opioid survey questions in varied, inconsistent ways. Three different aspects of respondents' understanding of opioids informed how respondents formulated answers—
- Their general awareness of opioids
- The connotations they associated with opioids
- Their personal understanding of their own relationship with opioids
Citation: Miller, K, Willson, S, Scanlon, P, Vickers, B, Massey, M, Creamer, L. 2019 Evaluation of opioid-related questions for federal household surveys. 2022.
Study year: 2018