Evaluation Briefs No. 4 | February 2009 Developing Process Evaluation Questions Evaluation questions help focus and provide structure to an evaluation, guide the evaluation planning process, facilitate decision making about evaluation methods to use, and inform discussions about how evaluation results can be used to improve school health programs. This brief defines evaluation questions and addresses steps to develop them. Evaluation Questions Defined Evaluation questions define the key issues to be explored by an evaluation. They are developed and prioritized by program staff, evaluation personnel, funders, and other stakeholders. Process evaluation questions address program operations, namely the who, what, when, and how many of program activities and program outputs. Examples of process evaluation questions provide the following information: 1. Whether program activities were accomplished. Were new HIV policies disseminated to all school districts during the past school year? 2. Quality of program components. How much did health education teachers improve their nutrition education knowledge due to our nutrition education curriculum workshops? 3. How well program activities were implemented. To what extent have teachers implemented with fidelity, evidence-based tobacco-use prevention education? 4. Whether the target audience was reached. Did all students identified with asthma receive the Open Airways curriculum? 5. How external factors influenced program delivery. How are the new reading and math achievement test requirements impacting teachers’ ability to provide health education instruction? The answers to these questions allow you to assess whether program activities are occurring as you expected. They also can help determine areas in which a program needs improvement so that you can reach expected outcomes. Steps to Develop Process Evaluation Questions The most useful evaluation questions reflect a diversity of stakeholder perspectives, key components of your program, your most important information needs, and resources available to answer the questions. Steps include the following: 1. Gather your stakeholders. Engage some or all of the stakeholders involved in strategic planning to help you develop evaluation questions or to share the questions with once you have developed them. 2. Review supporting materials such as your strategic plan, program logic model, work plan, and data from the Indicators for School Health Programs (Indicators) and School Health Profiles (Profiles). 3. Brainstorm evaluation questions about your overall program or a specific program activity. Focus on goals, strategies, and objectives in your strategic plan and workplan, and inputs, activities, and outputs in your logic model to generate process evaluation questions. You may identify many questions that can later be prioritized and reduced in number. 4. Sort evaluation questions from your brainstorming session into categories or groups that are relevant to you and your stakeholders. This process will help determine what resources you have for answering the evaluation questions you are most interested in. Decide which evaluation questions to answer. You should prioritize evaluation questions that: • Are important to program staff and stakeholders • Address important program needs • Reflect five-year program goals, strategies, and objectives of your program. • Can be answered with available resources, including funds and personnel expertise. • Can be answered within the available timeframe. • Provide information to make program improvements • Will be supported by your school health program administration 6. Verify that your questions are linked to your program. Once you have determined which evaluation questions you will answer, verify that they are linked to your program strategic plan, logic model, work plan, Indicators and Profiles. 7. Determine who, what, and how to collect the data that will be required for answering the evaluation questions. Determine who will be responsible for collecting the information and analyzing it to answer the evaluation questions. Possible data sources may include persons (e.g., teachers), documents, or records. Possible methods for collecting information include data from Indicators, interviews, and so on. Brief 13-18 provide information about data collection methods for evaluation (see Resources below). Example Process Evaluation Questions Suppose you want to focus your evaluation on your goal to increase implementation of effective physical activity, nutrition, and tobacco-use prevention (PANT) efforts in schools and school districts within a CSH framework. The three strategies for this goal may be to (1) develop model CSH and PANT policies for schools; (2) disseminate model CSH and PANT policies to schools and school districts; and (3) Provide resource and technical assistance (TA) on implementation of PANT within a CSH framework to schools, school districts, and health departments. The goals and strategies in your strategic plan will serve as one tool to guide you in developing your evaluation questions. Based on these goals and strategies you may develop questions such as: • To what extent does our program provide model CSH and PANT policies to support school and school district implementation of school health programs? • How many schools and school districts in the state are aware of our model CSH and PANT policies? • How many state schools and school districts are implementing our model PANT policies within a CSH framework? • How useful was our technical assistance in helping schools and school districts implement our model policies within a CSH framework? Resources Brief 13-18: Data Collection Methods for Evaluation. Available at http://www.cdc.gov/HealthyYouth/evaluation/resources.htm Taylor-Powell E, Steele S, Douglah M. Planning a Program Evaluation. University of Wisconsin-Cooperative Extension. 1996. Available at: http://learningstore.uwex.edu/pdf/G3658-1.PDF W.K. Kellogg Foundation. Evaluation Toolkit. Evaluation Questions. Available at http://www.wkkf.org/Default.aspx?tabid=90&CID=281&ItemID=2810011&NID=2820011&LanguageID=0 For further information or assistance, contact the Evaluation Research Team at ert@cdc.gov. You can also contact us via our Website at: http://www.cdc.gov/healthyyouth/evaluation/index.htm