Error processing SSI file

Conducting an Evaluation

PAGE 20 of 29

Evaluation Tips

Make a list of evaluation questions.
Identify the data sources.
Link the sources.
Proceed with the evaluation.

Evaluation is an ongoing assessment of a drinking water advisory protocol. It is the comparison of Standard Operating Procedure (SOP) criteria to performance. The collection and analysis of subjective experiences along with the analysis of objective forms of data or information provide the foundation for evaluation. Even the most basic evaluation provides insight and can improve future advisories and overall communication.

The scope of an advisory and a water system’s resources and capacities guide the evaluation process. The time, resources, and expertise needed for an evaluation vary with the scope. Evaluations can be conducted over time and in different stages. Evaluations draw on many sources of information, including, operational reports, debriefings, surveys, and public comment. See the Debriefing Evaluation Form. [DOCX – 2 pages]

Different types of data are described below:

Quantitative data, such as water quality data, Web analytics, and epidemiology statistics.
Qualitative data, such as customer comments, media reports, staff memos, etc.

See Appendix C: Online Resources, Data Management for links to information on qualitative data collection.

See the Advisory Feedback Guide [DOCX – 4 pages] and Call Center Data Collection Framework [DOCX – 2 pages] for collecting information from partners.

Surveys

Good Practice

Recognize that data sources and types vary by agency, capacity, and the scope and scale of the drinking water advisory.

Water systems should consider surveying their customers about the advisory. Surveys can be used to determine the effectiveness of the advisory and to determine whether the advisory influenced personal actions.

Customer service surveys may include a section related to an advisory.
Surveys can be designed to measure perceptions, actions, and communication effects and outcomes.
The perceptions and recall of respondents will change with the amount of time that passes between an advisory and a survey.
Surveys can be short and focused or longer to gather a full range of data.
Online surveys offer a fast and efficient way to solicit and analyze feedback from a wide range of respondents.

Tip

Designate a central location to archive evaluation data.

Consult the Post-Advisory Community Survey [DOCX – 6 pages] or CASPER (Community Assessment for Public Health Emergency Response) for ideas about what to ask in a survey and for methods to use when conducting a survey.

Data Management

Manage data as it is collected. Data management activities may include:

Storing data as hard and electronic copies in a central location for easy access.
Requesting information from partners involved with the advisory.
Asking partners about limitations or privacy policies that might limit the ability to report the data.
Working with database experts to ensure that the database structure is optimal for doing the planned analysis.

Data sets are valuable when they are applied to improve future responses. The information collected should be analyzed, synthesized, and reported in a useful format. Use the data to develop questions for public outreach and for the basis of evaluation. Develop a plan for data analysis that will give the answers needed for improvement and decision making.

Appendix C: Online Resources, Data Management contains links to websites on data frameworks.

Table 2 presents examples of different types of resources and data that can be used to evaluate communication activities during a drinking water advisory.

Table 2: Examples of Types and Sources of Drinking Water Advisory Data

Examples of types and sources of drinking water advisory data
Water Systems	Public Health	Media
Water Quality Operations Reports Customer Service Reports/Interviews Distribution Sampling	Surveillance Laboratory Surveys	Logs Archive Review Web Analytics