Step 4 – Gather Credible Evidence

Overview and Importance

Step 4 of the Evaluation Framework builds on the evaluation design from Step 3 to determine the evidence needed to answer the evaluation questions. During this step, evaluators will develop a data collection strategy, which involves the selection of data sources and associated measures that align with the evaluation purpose and questions.

Refer to the full-length CDC Program Evaluation Action Guide for additional information, examples and worksheets to apply the concepts discussed in this step.

Establish Expectations

Evaluators will want to collaborate with interest holders and set expectations up front that are realistic and will produce the evidence needed to address their informational needs. Examples of engaging with interest holders during this step include:

• Collaboratively establishing and managing expectations about the evidence that is needed to answer the evaluation questions, and the type and level of results that will be used to answer those evaluation questions.
• Identifying indicators that are relevant and align with the evaluation purpose and questions. Interest holders can validate indicators or suggest alternatives.
• Discussing expectations of what constitutes success or a positive finding and the rationale for such expectations.
• Identifying what data collection methods are practical, what data sources are reliable, and other expectations about the quality, type, and quantity of data needed to build credible evidence.

Choose Appropriate methods

An evaluator can choose between quantitative (numeric) or qualitative (narrative) data collection methods. Both quantitative and qualitative methods have benefits and limitations to their use.

Quantitative data methods rely on numerical data to draw conclusions and compare results. However, these methods are less able to capture unknowns about a particular topic or context and have less flexibility to be adaptive.

Qualitative data methods rely on words to gather deeper insights on topics or concepts that are not well understood. They are also useful when an understanding/study of evaluation participants' experience is needed to provide context on a topic. However, qualitative data are unable to offer comparability.

Choosing the appropriate data collection method will be influenced by several factors^[2]:

• Resource availability (such as time, cost, and staffing)
• Ethical considerations
• Size and scope of the evaluation
• Validity (how well the evaluation measures what's intended)
• Reliability (ability to replicate findings)

In determining the type of data to collect, evaluators and interest holders can explore data that is already available that may be able to answer the evaluation questions. Available data may include surveillance data, census data, or other datasets.

In some cases, an evaluator may use a mixed methods approach, purposefully integrating both qualitative and quantitative methods. Using a mixed methods approach can improve the accuracy of results by providing deeper insights and understanding on the "why" behind concepts, topics, or context more comprehensively^[1]. Ensure the data collection method selected to be used is culturally responsive to produce trustworthy and accurate data.

Common Data Collection Methods

Survey or Questionnaire (Quantitative)

Collection of data by gathering standardized information from all respondents. Typically involves various sampling methods and techniques. Surveys can be in person or virtual through various methods such as interviews, phone, mail-in, text, email, website, or social media.

Use when:

• you need to collect information quickly.
• you want to monitor changes or trends over time.
• you want to identify individuals' knowledge, attitudes or behaviors.

Interviews (Qualitative)

Structured conversations to gather information. Interviews can be conducted individually or in group settings, in person, on the telephone, or video.

Use when:

• you need rich, detailed information
• you need a deeper understanding of individuals' experiences, motivations, or emotions on a particular topic or issue
• you need deeper understanding of the context

Observations (Qualitative)

Conducted by watching or listening to behaviors. There are several considerations for conducting observations, including ethics, sampling methods and objectivity.

Use when:

• you need to explore a new topic of which little is known
• you need to understand individuals' behavior or context as it occurs naturally
• you need to discover silent norms and values

Focus Groups (Qualitative)

Interactive discussion involving small groups (usually 6-8) of participants on specific topics under the guidance of a trained moderator.

Use when:

• you need to explore a new or specific topic of which little is known
• you need to gain a range of view about issues in a single data collection

Case Study (Qualitative)

In-depth exploration of individuals, groups, events, programs, sites, or locations. Relies on various sources of information such as interviews, observations, documents, etc.

Use when:

• you want to gain concrete, in-depth knowledge about a specific topic
• conducting extensive research is not feasible

Document or Record Reviews (Qualitative)

Assessing existing documents to gain historical understanding of the program, participants, etc.

Use when:

• you need to gather background information on the program
• you need answers to specific evaluation questions related to "what"

Identify Relevant Indicators

Indicators differ from measures and metrics. They:

• Are measurable statements that measure a construct or concept (e.g., knowledge of health risks of smoking or awareness of smoking campaign)
• Help evaluators understand whether interventions or programs have achieved progress
• Help programs identify areas for improvement and make decisions based on credible evidence
• Help ensure that the evaluation is credible and effective
• Align with the evaluation focus area and evaluation questions

Below is an example of the relationship between the evaluation question, concept/construct, indicator and measure.

The program roadmap or logic model contains the activities and outcomes which can be helpful for guiding indicator development as well as the expectations established with the interest holders. These include activity indicators which measure the qualities of program implementation efforts and associated output indicators that measure products of the program activities, as well as outcome indicators measuring whether the program effects were achieved. Similarly, each indicator will have an associated measure/metric that directly aligns to it and quantifies it. The measure can be quantitative or qualitative depending on the evaluation question needing to be answered.

Developing Indicators

• Identify and define the expected changes or outcomes that the program is hoping to achieve.
• Choose indicators that are directly related to the evaluation purpose and questions. Clearly outline how the outcome or activity will be measured to make sure that the program constructs align with the measures.
• Use Specific, Measurable, Achievable, Relevant, Timebound, Inclusive and Equity (SMARTIE) criteria to develop the indicator and ensure there is sufficient specificity and detail to facilitate accurate data collection.
• Align the indicators with the evaluation questions to ensure there are appropriate measures to answer the questions.

Determine Appropriate Data Sources, Data Quantity, and Data Quality

When selecting data sources, consider the relevance of the source to the purpose of the evaluation. Primary data are new data specifically collected for that evaluation, and secondary data are existing, previously collected data. Using multiple data sources can or may enhance the credibility of the evaluation by providing different information that can answer the evaluation questions.

Data quantity refers to the amount of data that needs to be collected to answer the evaluation questions. Data quality refers to the appropriateness and integrity of data used in the evaluation1. In considering data quantity and quality, evaluators may:

• Assess the amount of data needed to answer the evaluation questions effectively (i.e. what information is important to know based on Step 3)
• Balance the amount of data to be collected with avoiding burden on the respondents or those involved with data collection
• Ensure the sample size is sufficient to do a detailed analysis

Data collection protocols that are thorough and clear with well-defined indicators will improve the likelihood that the evaluation will collect quality data. Additionally, the data collection instrument design, procedures, trainings, source selection, coding, and error checking are other factors that affect high quality data.

Data collection plans may need to be updated or modified during an evaluation, and documenting these data collection charges will be a useful resource when reviewing, analyzing, and interpreting data. Considerations when checking data quality include:

Accuracy

• Are there errors or inaccuracies?
• Are the data collection methods reliable?
• How were the data validated?
• Were there any limitations or biases?
• How did you minimize inaccuracies?

Completeness

• Are there missing data or gaps in the data?
• Were the procedures for data collection designed in a way that made sure the information captured is complete?

Consistency

• Do the data align with previous records related to the evaluation focus and are there inconsistencies?
• Were protocols for data collection consistenly followed across all sources?
• Were there sufficient data checks?
• If there were discrepencies, were the data resolved or explained to justify the discrepancy?

Timeliness

• How often is the data collected and updated?
• Were the data collected according to the timeline established in the evaluation plan?
• Were there delays in data collection? How were the delays addressed?

Relevance

• Do the data relate back to the objectives/goals of the evaluation?
• Were there any adjustments made to data collection methods at any point during collection?
• Did they enhance relevance to objectives?
• Were interest holders effectively engaged to ensure data is relevant for all parties?

While data quality and quantity are both important, also consider the cultural context and norms when making decisions regarding gathering data. Evaluators ensure persons providing data know and understand their rights, any risks, and how data will be handled, stored, and used. Evaluators also gather data that is secure, confidential, and compliant with program standards. Working collaboratively with interest holders on data collection and data security procedures will help align the procedures with cultural norms and the project setting1.

Develop Data Collection Instruments and Implement Data Collection

Data collection instruments lay the foundation for data and information gathering. This section explores key steps in developing data collection instruments such as:

• Defining the data needs
• Determining if there are existing instruments that can be adapted to fit the evaluation context or if there is a need to develop new instruments
• Developing a protocol guide for the instrument use to ensure consistency and completeness of data
• Developing questions or prompts that align with the previously selected indicators
• Developing the format of response options for the items (such as open-ended, multiple choice, or ranking)
• Piloting the data collection instruments with interest holders to make sure the questions are clear
• Revising the data collection instrument based on feedback from the pilot test
• Finalizing the data collection instrument.
• Training the data collection team on how to use the instruments and any associated protocols
• Developing a data management plan
• Developing a quality assurance plan
• Monitoring the data collection process

Applying the Evaluation Standards and Cross-Framework Actions

As with all the evaluation framework steps, it is important to integrate the evaluation standards and cross-cutting actions when gathering credible evidence in Step 4. See Table 8 in the CDC Program Evaluation Framework, 2024 to determine if you have effectively applied the evaluation standards and cross-cutting actions.