 |
 
|
 |
 |
| |
 |
|
|
| ||||||||||
|
|
|
|
|
|
|
||||
| ||||||||||
 |
|
|
|
Persons using assistive technology might not be able to fully access information in this file. For assistance, please send e-mail to: mmwrq@cdc.gov. Type 508 Accommodation and the title of the report in the subject line of e-mail. Guidelines for Evaluating Surveillance SystemsPrepared by Douglas N. Klaucke, M.D.* James W. Buehler, M.D.* Stephen B. Thacker, M.D.** R. Gibson Parrish, M.D.** Frederick L. Trowbridge, M.D.*** Ruth L. Berkelman, M.D.* and the Surveillance Coordination Group INTRODUCTION This document describes the evaluation of epidemiologic surveillance systems. Its purpose is to promote the best use of public health resources through the development of effective and efficient surveillance systems. It can serve as a guide for persons conducting their first evaluation and as a reference for those who are already familiar with the evaluation process. Epidemiologic surveillance is the ongoing and systematic collection, analysis, and interpretation of health data in the process of describing and monitoring a health event. This information is used for planning, implementing, and evaluating public health interventions and programs. Surveillance data are used both to determine the need for public health action and to assess the effectiveness of programs. The evaluation of surveillance systems should promote the best use of public health resources by ensuring that only important problems are under surveillance and that surveillance systems operate efficiently. Insofar as possible, the evaluation of surveillance systems should include recommendations for improving quality and efficiency, e.g., eliminating unnecessary duplication. Most importantly, an evaluation should assess whether a system is serving a useful public health function and is meeting the system's objectives. Because surveillance systems vary widely in methodology, scope, and objectives, characteristics that are important to one system may be less important to another. Efforts to improve certain attributes--such as the ability of a system to detect a health event (sensitivity)--may detract from other attributes, such as simplicity or timeliness. Thus, the success of an individual surveillance system depends on the proper balance of characteristics, and the strength of an evaluation depends on the ability of the evaluator to assess these characteristics with respect to the system's requirements. In an effort to accommodate to these objectives, any approach to evaluation must be flexible. With this in mind, the guidelines that follow describe many measures that can be applied to surveillance systems, with the clear understanding that all measures will not be appropriate for all systems. ORGANIZATION OF THIS DOCUMENT This document begins with an outline of the tasks involved in doing an evaluation, which is followed by sections describing each element of an evaluation. The first such section addresses the public health importance of the disease or health condition under surveillance. The second provides a framework for describing the components of a surveillance system. Subsequent sections focus on the attributes of a surveillance system (simplicity, flexibility, acceptability, sensitivity, predictive value positive, representativeness, and timeliness) and demonstrate how these combine to affect the usefulness and cost of a system. The document concludes with a discussion of resources needed to operate the surveillance system and sections detailing conclusions and recommendations. As feedback is received on evaluations based on the guidelines given in this document, relevant examples will be incorporated into future updates. OUTLINE OF TASKS FOR EVALUATING A SURVEILLANCE SYSTEM
Describe the public health importance of the health event. Definition The public health importance of a health event and the need to have that health event under surveillance can be described in several ways. Health events that affect many people or require large expenditures of resources clearly have public health importance. However, health events that affect relatively few persons may also be important, especially if the events cluster in time and place--e.g., a limited outbreak of a severe disease. At other times, public concerns may focus attention on a particular health event, creating or heightening the sense of importance. Diseases that are now rare because of successful control measures may be perceived as "unimportant," but their level of importance should be assessed in light of their potential to re-emerge. Finally, the public health importance of a health event is influenced by its preventability. Measures Parameters for measuring the importance of a health event--and, therefore, the surveillance system with which it is monitored--include:
effect of existing control measures. For example, the number of cases of vaccine-preventable illness has declined following the implementation of school immunization laws, and the public health importance of these diseases would be underestimated by case counts alone. In such instances, it may be possible to estimate the number of cases that would be expected in the absence of control programs (1). Preventability can be defined at several levels--from preventing the occurrence of disease (primary prevention); through early detection and intervention with the aim of reversing, halting, or at least retarding the progress of a condition (secondary prevention); to minimizing the effects of disease and disability among those already ill (tertiary prevention). From the perspective of surveillance, preventability reflects the potential for effective public health intervention at any of these levels. Attempts have been made to quantify the public health importance of various diseases and health conditions. Dean et al. describe such an approach using a score that takes into account age-specific mortality and morbidity rates as well as health- care costs (2). SYSTEM DESCRIPTION Tasks
Methods Objectives may include detecting or monitoring outbreaks, monitoring trends, identifying contacts and administering prophylaxis, enrolling case-patients in a study, and generating hypotheses about etiology. The objectives of the system define a framework for evaluating the specific components. The next task is to describe the components of a surveillance system. This can be done by answering the following questions:
of the health-event reports by the system, and then to depict these steps in a flow chart (Figure 1). USEFULNESS Tasks
Definition A surveillance system is useful if it contributes to the prevention and control of adverse health events, including an improved understanding of the public health implications of such events. A surveillance system can also be useful if it helps to determine that an adverse health event previously thought to be unimportant is actually important. Methods An assessment of the usefulness of a surveillance system should begin with a review of the objectives of the system and should consider the dependence of policy decisions and control measures on surveillance. Depending on the objectives of a particular surveillance system, the system may be considered useful if it satisfactorily addresses at least one of the following questions. Does the system:
Discussion Usefulness may be affected by all the attributes of surveillance. Increased sensitivity may afford a greater opportunity for identifying epidemics and understanding the natural course of an adverse health event in a community. Improved timeliness allows control and prevention activities to be initiated earlier. Increased predictive value positive enables public health officials to focus on productive activities. A representative surveillance system will better characterize the epidemiologic characteristics of a health event in a defined population. Systems that are simple, flexible, and acceptable also tend to be more useful. SYSTEM ATTRIBUTES Task Evaluate the system for each of the following attributes:
Simplicity Definition The simplicity of a surveillance system refers to both its structure and ease of operation. Surveillance systems should be as simple as possible while still meeting their objectives. Methods A chart describing the flow of information and the lines of response in a surveillance system can help assess the simplicity or complexity of a surveillance system. A flow chart for a generic surveillance system is illustrated in Figure 1. The following measures might be considered in evaluating the simplicity of a system:
Discussion It may be useful to think of the simplicity of a surveillance system from two perspectives: the design of the system and the size of the system. An example of a system that is simple in design is one whose case definition is easy to apply and in which the person identifying the case will also be the one analyzing and using the information. A more complex system might involve some of the following:
amount of resources that are required to operate the system. B. Flexibility Definition A flexible surveillance system can adapt to changing information needs or operating conditions with little additional cost in time, personnel, or allocated funds. Flexible systems can accommodate, for example, new diseases and health conditions, changes in case definitions, and variations in reporting sources. Methods Flexibility is probably best judged retrospectively, by observing how a system responded to a new demand. For example, when acquired immunodeficiency syndrome (AIDS) emerged in 1981, the existing notifiable disease reporting system of state health departments was used to report cases, and AIDS surveillance has adapted to rapidly advancing knowledge about the disease, its diagnosis, and its risk factors. Another example is the capacity of the gonorrhea surveillance system to accommodate special surveillance for penicillinase-producing Neisseria gonor- rhoeae. Discussion Unless efforts have been made to adapt a system to another disease, it may be difficult to assess the flexibility of that system. In the absence of practical experience, one can look at the design and workings of a system. Generally, simpler systems will be more flexible--fewer components will need to be modified when adapting the system for use with another disease. C. Acceptability Definition Acceptability reflects the willingness of individuals and organizations to participate in the surveillance system. Methods In terms of evaluating a surveillance system, acceptability refers to the willingness to use the system by: a) persons outside the sponsoring agency, e.g., those who are asked to do something for the system and b) persons in the sponsoring agency that operates the system. To assess acceptability, one must consider the points of interaction between the system and its participants (Figure 1), including persons with the condition and those reporting cases. Quantitative indicators of acceptability include:
surveillance report forms, while others would require special studies or surveys. Discussion Acceptability is a largely subjective attribute that encompasses the willingness of persons on whom the system depends to provide accurate, consistent, complete, and timely data. Some factors influencing the acceptability of a particular system are:
Sensitivity Definition The sensitivity of a surveillance system can be considered on two levels. First, at the level of case reporting, the proportion of cases of a disease or health condition detected by the surveillance system can be evaluated. In Table 1 this is represented by A/(A+C). Second, the system can be evaluated for its ability to detect epidemics (3). Methods The sensitivity of a surveillance system is affected by the likelihood that:
surveillance systems that do not fit the traditional disease care-provider model. For example, the sensitivity of a telephone-based surveillance system of morbidity or risk factors is affected by:
system and on the resources available for the evaluation. The measurement of sensitivity in a surveillance system requires a) the validation of information collected by the system and b) the collection of information external to the system to determine the frequency of the condition in a community (4). From a practical standpoint, the primary emphasis in assessing sensitivity--assuming that most reported cases are correctly classified--is to estimate the proportion of the total number of cases in the community being detected by the system. Discussion A surveillance system that does not have high sensitivity can still be useful in monitoring trends, as long as the sensitivity remains reasonably constant. Questions concerning sensitivity in surveillance systems most commonly arise when changes in disease occurrence are noted. Changes in sensitivity can be precipitated by such events as heightened awareness of a disease, introduction of new diagnostic tests, and changes in the method of conducting surveillance. A search for such surveillance "artifacts" is often an initial step in outbreak investigations. E. Predictive Value Positive Definition Predictive value positive (PVP) is the proportion of persons identified as having cases who actually do have the condition under surveillance (5). In Table 1 this is represented by A/(A+B). Methods In assessing PVP, primary emphasis is placed on the confirmation of cases reported through the surveillance system. Its effect on the use of public health resources can be considered on two levels. At the level of an individual case, PVP affects the amount of resources used for case investigations. For example, in some states every reported case of type A hepatitis is promptly investigated by a public health nurse, and family members at risk are referred for prophylactic treatment with immune globulin. A surveillance system with low PVP--and therefore frequent "false-positive" case reports--would lead to wasted resources. The other level is that of detection of epidemics. A high rate of erroneous case reports may trigger an inappropriate outbreak investigation. Therefore, the proportion of epidemics identified by the surveillance system that are true epidemics is needed to assess this attribute. Calculating the PVP may require that records be kept of all interventions initiated because of information obtained from the surveillance system. A record of the number of case investigations done and the proportion of persons who actually had the condition under surveillance would allow the calculation of the PVP at the level of case detection. Personnel activity reports, travel records, and telephone logbooks may all be useful in estimating the PVP at the epidemic detection level. Discussion PVP is important because a low value means that a) non-cases are being investigated and b) epidemics may be mistakenly identified. PFalse-positiveP' reports may lead to unnecessary intervention, and falsely detected PepidemicsP' may lead to costly investigations and undue concern in the community. A surveillance system with high PVP will lead to fewer wild-goose chases and wasted resources. An example of a surveillance evaluation that examined PVP was reported by Barker et al. They reviewed hospital charts to determine the proportion of persons admitted with a diagnosis of stroke who had the diagnosis confirmed (6). Of 1,604 patients admitted to seven acute-care hospitals with a stroke-related diagnosis, 903 (PVP = 56%) were subsequently confirmed to have had strokes. The PVP for a health event is closely related to the clarity and specificity of the case definition. Good communication between the persons who report cases and the receiving agency also can improve PVP. The PVP reflects the sensitivity and specificity of the case definition and the prevalence of the condition in the population (Table 1). The PVP increases with increasing specificity and prevalence. F. Representativeness Definition A surveillance system that is representative accurately describes a) the occurrence of a health event over time and b) its distribution in the population by place and person. Methods Representativeness is assessed by comparing the characteristics of reported events to all such actual events. Although the latter information is generally not known, some judgment of the representativeness of surveillance data is possible, based on knowledge of:
seek to identify a probability sample of all cases. Quality of data is an important part of representativeness. Much of the discussion in this document focuses on the identification and classification of cases. However, most surveillance systems rely on more than simple case counts. Information commonly collected includes the demographic characteristics of affected persons, details about the health event, and notification of the presence or absence of potential risk factors. The quality and usefulness and representativeness of this information depends on its completeness and validity. Quality of data is influenced by the clarity of surveillance forms, the quality of training and supervision of persons who complete surveillance forms, and the care exercised in data management. A review of these facets of a surveillance system provides an indirect measure of quality of data. Examining the percentage of unknown or blank responses to items on surveillance forms or questionnaires is straightforward. Assessing the reliability and validity of responses would require such special studies as chart reviews or re-interviews of respondents. Discussion In order to generalize findings from surveillance data to the population at large, the data from a surveillance system should reflect the population characteristics that are important to the goals and objectives of that system. These characteristics generally relate to time, place, and person. An important result of evaluating the representativeness of a surveillance system is the identification of population subgroups that may be systematically excluded from the reporting system. This process allows appropriate modification of data collection and more accurate projection of incidence of the health event in the target population. For example, an evaluation of reporting of hepatitis in a county in Washington State suggested that cases of type B hepatitis were under-reported among homosexual males and that cases of type non A-non B hepatitis were under-reported among persons given blood transfusions. The importance of these risk factors as contributors to the occurrence of these diseases was apparently underestimated by the selective under-reporting of certain types of hepatitis cases (9). Errors and bias can make their way into a surveillance system at any stage. Because surveillance data are used to identify high-risk groups, to target interventions, and to evaluate interventions, it is important to be aware of the strengths and limitations of the information in the system. So far the discussion of attributes has been aimed at the information collected for cases, but in many surveillance systems morbidity and mortality rates are calculated. The denominators for these rate calculations are often obtained from a completely separate data system maintained by another agency, e.g., the Bureau of the Census. Thought should be given to the comparability of categories (e.g., race, age, residence) on which the numerators and denominators of rate calculations are based. G. Timeliness Definition Timeliness reflects the speed or delay between steps in a surveillance system. Methods The major steps in a surveillance system are shown in Figure 2. The time interval linking any two of the steps in this figure can be examined. The interval usually considered first is the amount of time between the onset of an adverse health event and the report of the event to the public health agency responsible for instituting control and prevention measures. Another aspect of timeliness is the time required for the identification of trends, outbreaks, or the effect of control measures. With acute diseases, the onset of symptoms is usually used. Sometimes the date of exposure is used. With chronic diseases, it may be more useful to look at elapsed time from diagnosis rather than to estimate an onset date. Discussion The timeliness of a surveillance system should be evaluated in terms of availability of information for disease control--either for immediate control efforts or for long-term program planning. For example, a study of a surveillance system for Shigella infections indicated that the typical case of shigellosis was brought to the attention of health officials 11 days after onset of symptoms--a period sufficient for the occurrence of secondary and tertiary transmission. This suggests that the level of timeliness was not satisfactory for effective disease control (10). In contrast, when there is a long period of latency between exposure and appearance of disease, the rapid identification of cases of illness may not be as important as the rapid availability of exposure data to provide a basis for interrupting and preventing exposures that lead to disease. In another time frame, surveillance data are being used by public health agencies to track progress toward the 1990 Objectives for the Nation and to plan for the Year 2000 Objectives. The need for rapidity of response in a surveillance system depends on the nature of the public health problem under surveillance and the objectives of that system. Recently, computer technology has been integrated into surveillance systems and may promote timeliness (11,12). RESOURCES FOR SYSTEM OPERATION Task Describe the resources that are used to operate the system (direct costs). Definitions This document covers only the resources directly required to operate a surveillance system. These are sometimes referred to as Pdirect costsP' and include the personnel and financial resources expended in collecting, processing, analyzing, and disseminating the surveillance data. Methods In estimating these resources consider the following:
system (e.g., person-time expended per year of operation). If desired, these measures can be converted to dollar estimates by multiplying the person-time by appropriate salary and benefit figures. b. Other resources These may include the cost of travel, training, supplies, equipment, and services (e.g., mail, telephone, and computer time). The application of these resources at all levels of the public health system--from the local health-care provider to municipal, county, state, and Federal health agencies--should be considered. The costs of surveillance systems from two studies are illustrated in Tables 2 and 3 below (7,13). Discussion This approach to assessment of resources includes only those personnel and material resources required for the operation of surveillance and excludes a broader definition of costs that might be considered in a more comprehensive evaluation. Estimating the overall costs of a surveillance system can be a complex process. The estimates may include the estimation of a) indirect costs, such as follow-up laboratory tests or treatment incurred as a result of surveillance; b) costs of secondary data sources (e.g., vital statistics or survey data); and c) costs averted (benefits) by surveillance. Costs are often judged relative to benefits, but few evaluations of surveillance systems are likely to include a formal cost-benefit analysis, and such analyses are beyond the scope of this document. Estimating benefits (e.g., savings resulting from preventing morbidity through surveillance data) may be possible in some instances, although this approach does not take into account the full spectrum of benefits that may result from surveillance systems. More realistically, costs should be judged with respect to the objectives and usefulness of a surveillance system. Examples Examples of resource estimation for surveillance systems operated in Vermont and Kentucky follow. Vermont example (7) Two methods of collecting surveillance data in Vermont have been compared. The PpassiveP' system was already in place and consisted of unsolicited reports of notifiable diseases to the district offices or state health department. The PactiveP' system was implemented in a probability sample of physician practices. Each week, a health department employee called these practitioners to solicit reports of selected notifiable diseases. In comparing the two systems an attempt was made to estimate their costs. The estimates of resources directly applied to the surveillance systems are shown in Table 2. Kentucky example (13) Another example is provided by an assessment of the costs of a surveillance system involving the active solicitation of case reports of type A hepatitis in Kentucky. Table 3 summarizes the costs of this system. The resources that went into the direct operation of the system were for personnel and telephone and were estimated as $3,764 and $535, respectively. This system found nine more cases than would have been found through the passive surveillance system, and an estimated seven hepatitis cases were prevented through prophylaxis of the contacts of the nine case-patients. CONCLUSIONS AND RECOMMENDATIONS Tasks List your conclusions and recommendations. These should state whether the system is addressing an important public health problem and is meeting its objectives. Recommendations should address the continuation and/or modification of the surveillance system. Discussion The attributes and costs of a surveillance system are interdependent. Before recommending changes in a system, interactions among the attributes and costs should be considered to ensure that benefits resulting from strengthening one attribute do not adversely affect another attribute. Efforts to increase sensitivity, PVP, timeliness, and representativeness tend to increase the cost of a surveillance system, although savings in efficiency with automation may offset some of these costs (12). As sensitivity and PVP approach 100%, a surveillance system is more likely to be representative of the population being monitored. However, as sensitivity increases, PVP may decrease. Efforts to increase sensitivity and PVP tend to make a surveillance system more complex--potentially decreasing its acceptability, timeliness, and flexibility. For example, a study comparing health-department-initiated (active) surveillance and provider-initiated (passive) surveillance did not improve timeliness, despite increased sensitivity (8). SUMMARY STATEMENT Evaluating surveillance systems is not easy. There is no perfect system; trade-offs must always be made. Each system is unique and therefore requires a balancing of the effort and resources put into each of its components if the system is to achieve its intended goal. This document has presented guidelines--not absolutes--for the evaluation of surveillance systems. Attributes have been described that can be examined and evaluated to assess a system's ability to achieve the objectives for which it was designed. Our goal has been to make the evaluation process more explicit and objective. Suggestions on how we may improve these guidelines would be welcomed. References
Disclaimer All MMWR HTML documents published before January 1993 are electronic conversions from ASCII text into HTML. This conversion may have resulted in character translation or format errors in the HTML version. Users should not rely on this HTML document, but are referred to the original MMWR paper copy for the official text, figures, and tables. An original paper copy of this issue can be obtained from the Superintendent of Documents, U.S. Government Printing Office (GPO), Washington, DC 20402-9371; telephone: (202) 512-1800. Contact GPO for current prices. **Questions or messages regarding errors in formatting should be addressed to mmwrq@cdc.gov.Page converted: 08/05/98 |
| |||||||
This page last reviewed 5/2/01
|
