Purpose
The purpose of this tool is to assist state and local public health programs as they develop objectives and questions to guide evaluation of their tuberculosis (TB) programs.
Overview
For TB program evaluation, appropriate objectives and corresponding evaluation questions help ensure that the evaluation produces useful information for program improvement.
To ensure that objectives can be used to track progress and meet the expectations of the evaluation, make the objectives Specific, Measurable, Achievable, Relevant, and Time bound (SMART). SMART objectives and evaluation questions can be developed during brainstorming sessions and discussions with internal and external partners.
Take the time to develop an evaluation plan that includes priority objectives and evaluation questions with short term timelines. Use timelines that reflect annual report submission deadlines, as these can lead to more feasible evaluation projects.
Examples by Focus Area
Below are examples of SMART objectives and associated evaluation questions. Relevant National TB Program Objectives and National Tuberculosis Indicators Project (NTIP) indicators are listed under each focus area. Although each example includes several evaluation questions, in practice, a smaller number of questions would be chosen for an evaluation plan.
These examples are not meant to be copied directly. They are intended to give ideas that can be applied to develop objectives and questions specific to each TB program.
Relevant National TB Objective
Increase the proportion of TB patients with a positive or negative HIV test result that are reported to the TB program to 98% by 2030.
Relevant NTIP Indicator(s)
Known HIV Status
Example TB Program SMART Objectives
By end of budget period (time bound) determine why or why not (measurable and achievable) patients treated for TB in the cohort year have or do not have an HIV test result recorded (specific and relevant).
By end of budget period (time bound) identify patient and provider barriers to HIV testing (specific, measurable, achievable, and relevant).
Example Evaluation Questions
- To what extent are patients offered an HIV test, by provider type (e.g. clinics, hospitals, private providers)?
- What challenges do providers experience in receiving and reporting HIV test results?
- What are patient-related barriers to obtaining HIV tests? (Consider where HIV tests are performed and the costs per HIV test per patient)
- What are practice-related barriers to administering HIV tests?
- What are the barriers to obtaining and recording HIV test results for surveillance?
Relevant National TB Objective
For TB patients with positive sputum culture results, increase the proportion of documented conversion to negative results within 60 days of treatment initiation to 80% by 2030.
Relevant NTIP Indicator(s)
Sputum Culture Conversion
Example TB Program SMART Objectives
By end of budget period (time bound), identify barriers to timely documentation of sputum culture conversion results by providers (measurable, achievable, specific, and relevant).
By end of budget period (time bound), identify two strategies to improve sputum culture conversion results reporting to the state health department (specific, measurable, achievable, and relevant).
Example Evaluation Questions
- When and how are sputum specimen(s) sent to the laboratory for processing?
- How frequently are sputum culture results reported to the TB case manager/state TB staff? How many months after starting TB treatment are results reported?
- What challenges do TB program staff identify in sputum culture collection, processing, and reporting?
- What factors lead to a longer sputum culture conversion time (e.g., clinical characteristics of TB patients, treatment regimen, program/lab operational protocols)?
- To what extent did implementing the Sputum Collection and Reporting Training for State Nurses Program improve the documentation of sputum culture conversion (e.g., improved quality of sputum culture collection, clearer protocols, improved efficiency of data reporting)?
Relevant NTIP Indicator(s)
- Sputum Culture Conversion
- Treatment Initiation
- Completion of Treatment
Example TB Program SMART Objectives
By end of budget period (time bound), identify the benefits and limitations (measurable and achievable) of offering eDOT (specific and relevant).
By end of budget period (time bound), identify challenges and facilitators (measurable and achievable) of offering eDOT (specific and relevant).
By end of budget period (time bound), initiate a strategy to begin implementing eDOT for TB patients beginning treatment in the cohort year (specific, measurable, achievable, and relevant).
By end of budget period (time bound), initiate a strategy to identify and enroll TB patients who are most likely to adhere to telehealth to participate in the eDOT program (specific, measurable, achievable, and relevant).
Example Evaluation Questions
- On average, how far do staff travel and how much time is spent to conduct field DOT each week (e.g., traveling to observe DOT at a patient's location in a week)?
- What are the health department costs associated with conducting field DOT?
- On average, how far do patients travel to come to the TB clinic? How long does it take patients to travel to the TB clinic?
- What are the health department rules and regulations for using eDOT?
- What equipment would be needed to implement an eDOT program? (e.g., webcams, software)
- How would the health department identify eligible patients for an eDOT program?
Relevant NTIP Indicator(s)
- Treatment Initiation
- Completion of therapy
Example TB Program SMART Objectives
By end of budget period (time bound), describe characteristics of current eDOT strategy (specific, measurable, achievable, and relevant).
By end of budget period (time bound), determine the effectiveness of the current eDOT strategy (specific and relevant) to improve on patient outcomes and benefit program capacity (measurable and achievable).
Example Evaluation Questions
- What percentage of TB treatment doses are observed by eDOT compared to doses observed by in-person DOT over the last year?
- How much time was spent traveling for each in-person field dose (so that time per dose can be calculated)?
- What is the cost per observed dose per TB patient using eDOT compared with the cost per observed dose per TB patient using in-person DOT?
- How many doses were administered via synchronous (real-time interaction) vs. asynchronous (not real-time interaction) eDOT?
- What percentage of patients with TB completed treatment within 12 months while participating in eDOT compared to in-person DOT?
- How much, if at all, has eDOT use decreased staff time spent on DOT compared to in-person DOT?
- What are the benefits and challenges of using eDOT compared to in-person DOT according to patients and health care providers?
Relevant National TB Objective
Ensure the 100% completeness of core Aggregate Reports for Tuberculosis Program Evaluation (ARPE) data items reported to CDC by 2030.
Relevant NTIP Indicator(s)
Data reporting - ARPEs
Example TB Program SMART Objectives
By end of budget period, identify barriers (measurable and achievable) that cause missing data items in ARPE data submissions to CDC for the cohort period (specific and relevant).
By end of budget period, identify a data quality assurance strategy to address missing data item issues for the cohort period (specific, measurable, achievable, and relevant).
By end of budget period, assess the effectiveness of newly implemented data quality assurance strategy for the cohort period data (specific, measurable, achievable, and relevant).
Example Evaluation Questions
- How well are each of the core ARPE data items that need to be reported to CDC captured in the monitoring system(s)?
- What are the challenges and barriers to gathering and reporting each of the core data items?
- What strategies can help us improve ARPE data?
- Since applying a new strategy to improve data reporting, how much change is there in the timeliness, completeness, and accuracy of data exchange between health care facilities and the state health department?
Relevant National TB Objective
For contacts to sputum Acid-Fast Bacillus (AFB) smear-positive TB cases, increase the proportion to 90% for those who are examined for infection and disease by 2030.
Relevant NTIP Indicator(s)
Contacts Examined
Example TB Program SMART Objectives
By end of budget period (time bound), identify barriers to completing an examination for TB disease and latent TB infection among contacts in different population groups (specific, measurable, achievable, and relevant).
By end of budget period (time bound), describe success of implementing targeted testing strategies to increase the proportion of contacts to sputum smear-positive TB cases examined (specific, measurable, achievable, and relevant) to inform future program planning.
Example Evaluation Questions
- What barriers are expressed by contacts related to completing their test for TB infection and chest X-ray?
- To what extent are the proportion of contacts examined different in regard to demographic characteristics, risk type (medical and social), type of test (e.g., TST, IGRA), and examination setting (e.g., congregate settings, health care facilities, public health clinics, private providers, schools, or household)?
- How successful were targeted testing strategies in reaching population groups who are at high risk for TB disease and latent TB infection?