Promoting the Science and Practice of Implementation Evaluation in Public Health

Leonard Jack Jr, PhD, MSc

Suggested citation for this article: Jack L Jr. Promoting the Science and Practice of Implementation Evaluation in Public Health. Prev Chronic Dis 2018;15:180645. DOI: http://dx.doi.org/10.5888/pcd15.180645.

Leonard Jack Jr, PhD, MSc

In early 2018, PCD addressed these important considerations by introducing Implementation Evaluation, a new article type that provides the journal’s readers (program planners, policy makers, evaluators, researchers, and diverse stakeholders) with information on how to refine evaluation methods, make health system improvements, strengthen collaborations and partnerships, build organizational infrastructure, measure return on investments, and enhance data collection approaches (www.cdc.gov/pcd/for_authors/types_of_articles.htm). Implementation Evaluation articles provide insights into factors that affect the ability of public health practice to successfully package and disseminate effective interventions implemented and evaluated in real-world settings. PCD’s interest in this area extends to research that examines which factors positively or negatively impact the diffusion of proven interventions and the degree of integrity needed to generate success. Specific program elements such as “adherence to intervention, exposure, or dose, quality of delivery, participant responsiveness and program differentiation” are all factors associated with implementation fidelity (4). Implementation Evaluation articles published by PCD offer readers timely research that examines in comprehensive ways how evidence-based interventions are implemented in comparable real-world settings.

PCD was fortunate in its inaugural year of introducing this new article type to receive many outstanding submissions. The journal is excited to present this collection of 5 articles that highlight research findings from implementation evaluation efforts that address a variety of topics:

1. A call to action for public health professionals to advance dissemination and implementation science;
2. an enhancement of a framework for evaluating behavioral health systems;
3. the application of a pragmatic framework to guide health care systems in assessing implementation and impact of an evidence-based physical activity program;
4. an assessment of the effectiveness and cost benefit of a program for weight loss and diabetes prevention in a rural setting; and
5. an evaluation of activities to reduce the intake of sodium in community settings.

As part of its effort to provide more research on topics related to implementation evaluation, PCD has recruited associate editors and editorial board members with considerable experience and expertise in implementation dissemination, implementation science, and implementation evaluation. The evolution of work occurring in these areas has expanded over the past 25 years, with the fundamental goal of better identifying program components in public health that contribute to achieving success in population health outcomes. A major component of this goal is to find cost-effective ways to disseminate effective interventions in alignment with local context and real-world settings. An essay from authors Estabrooks, Brownson, and Pronk of our editorial board and associate editor teams provides an overview of dissemination and implementation science, including a review of frameworks, models, theories, concepts, and principles over the past 25 years. These authors discuss the importance of developing individual and team-based skills and abilities among public health professionals that increase adoption and scalability of evidence-based interventions.

Public health surveillance systems are an important aspect of implementation evaluation in collecting and analyzing timely data and disseminating findings that guide public health responses to pressing public health issues (5). Public health surveillance systems, when developed with input from stakeholders, can be implemented and sustained on an ongoing basis (6). Azofeifa and colleagues describe the use of a working group to generate a revised framework for examining various aspects of behavioral health among adults (7). They identified and discussed a range of factors — organizational barriers, limitations to data sources, issues related to stigma and confidentiality — that affect the development and implementation of this surveillance system (6,7). They also discuss ways to address these issues to increase adoption, use, and sustainability of the proposed behavioral health surveillance evaluation framework (7).

In addition to public health surveillance systems, other systems-based approaches must strike a balance between rigor and relevance in considering ways to evaluate the adoption, scalability, and sustainability of interventions (8). Hence, implementation science research, evaluation, and practice should use tailored evaluation designs that carefully align with the components of the intervention (8). Stoutenberg and coauthors applied the RE-AIM framework, an approach to planning and evaluating factors related to internal and external validity, to guide health care systems in assessing the implementation and impact of the Exercise is Medicine (EIM) program (9). EIM is an initiative that integrates physical activity assessment, prescription, and patient referrals as a standard of care (9). Authors provide recommendations and insights into ways the EIM in health systems can be effectively implemented and evaluated.

Economic evaluations are another aspect of implementation evaluation that is becoming increasingly helpful in informing decision-making to operationalize and sustain implementation strategies and best practices (10). Economic evaluations are critical to public health professionals, health care organizations, and funders interested in deciding how to maximize use of limited fiscal and human resources (10). McKnight and associates assessed the effectiveness and cost benefit of replicating a 12-week wellness program targeting adults in 4 rural locations (11). Researchers reported information on participation, completion, and changes in several health outcomes and discussed how a combination of factors influenced researchers’ ability to achieve results similar to those derived in the original wellness program.

Finally, the collection includes research on reducing intake of sodium in community settings, which has remained a national public health issue (12). This public health goal is particularly important given that diets high in salt are linked to high blood pressure, which is a major risk factor for stroke among adults (13). Community-based salt reduction programs may be effective in a range of settings, but more robust evaluation methods are needed. Scaling up these efforts in coordination with national initiatives could provide the most effective and sustainable approach to reducing population salt intake (14,15). In 2016, the Centers for Disease Control and Prevention (CDC) launched the Sodium Reduction in Communities Program (SRCP) to help increase consumers’ options for lower-sodium foods and create healthier food environments in communities (16). CDC’s SRCP funded and provided technical assistance to 8 recipients to increase the availability and purchase of lower-sodium food options by implementing 1) food service guidelines and nutrition standards, 2) procurement practices, 3) meal and/or menu modifications, and 4) environmental strategies and behavioral economics approaches to increase consumers’ options of lower-sodium foods (16). Long and coauthors present findings generated from baseline and 1-year follow-up from the SRCP implemented in Arkansas (17). Researchers describe how program staff worked closely with personnel in a school district and in a community meal program to implement intervention activities to reduce dietary sodium among the food options available and served. Researchers reported that mean sodium content of meals was reduced among participants in both the schools and the community meal program.

This collection of articles from PCD’s first year of Implementation Evaluation articles represents an exciting new area of focus for the journal. PCD will continue to identify and publish cutting-edge implementation evaluation research that helps all populations benefit from the dissemination of new and proven discoveries. Toward that end, the journal seeks to gain a deeper understanding of how factors like staffing capacity, economics, leadership support, and intervention fidelity influence scaling up and sustaining proven, culturally appropriate, and setting-relevant interventions. PCD is also committed to publishing articles that use implementation evaluation findings to identify circumstances under which intervention activities should be reduced or discontinued because of factors such as premature adoption (implementing intervention activities before or without having proven evidence of effectiveness), harmful effects, or wasteful use of fiscal or human resources (18). PCD encourages authors to visit the Author’s Corner section of the journal’s website at www.cdc.gov/pcd/for_authors/index.htm to learn more about requirements for submitting an Implementation Evaluation manuscript for consideration.

Top

Author Information

Corresponding Author: Leonard Jack, Jr, PhD, MSc, Preventing Chronic Disease, National Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control and Prevention, Mailstop F-80, 4770 Buford Highway, NE, Atlanta, GA 30341-3717. Telephone: 770-488-1886. Email: ljj0@cdc.gov.

.

Top

References

1. Bauer UE, Briss PA, Goodman RA, Bowman BA. Prevention of chronic disease in the 21st century: elimination of the leading preventable causes of premature death and disability in the USA. Lancet 2014;384(9937):45–52. CrossRef PubMed
2. Century J, Rudnick M, Freeman C. A framework for measuring fidelity of implementation: a foundation for shared language and accumulation of knowledge. Am J Eval 2010;31(2):199–218. CrossRef
3. Breitenstein SM, Gross D, Garvey CA, Hill C, Fogg L, Resnick B. Implementation fidelity in community-based interventions. Res Nurs Health 2010;33(2):164–73. PubMed
4. Kershner S, Flynn S, Prince M, Potter SC, Craft L, Alton F. Using data to improve fidelity when implementing evidence-based programs. J Adolesc Health 2014;54(3, Suppl):S29–36. CrossRef PubMed
5. Groseclose SL, Buckeridge DL. Public health surveillance systems: recent advances in their use and evaluation. Annu Rev Public Health 2017;38(1):57–79. CrossRef PubMed
6. Althouse BM, Scarpino SV, Meyers LA, Ayers JW, Bargsten M, et al. Enhancing disease surveillance with novel data streams: challenges and opportunities. EPJ Data Sci 2015 4;17.
7. Azofeifa A, Stroup DF, Lyerla R, Largo T, Gabella BA, Smith CK, et al. Behavioral Health Surveillance Working Group. Evaluating behavioral health surveillance systems. Prev Chronic Dis 2018;15:170459. CrossRef PubMed
8. Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract 2008;14(2):117–23. CrossRef PubMed
9. Stoutenberg M, Galaviz KI, Lobelo F, Joy E, Heath GW, Hutber A, et al. A pragmatic application of the RE-AIM framework for evaluating the implementation of physical activity as a standard of care in health systems. Prev Chronic Dis 2018;15:170344. CrossRef PubMed
10. Hoomans T, Severens JL. Economic evaluation of implementation strategies in health care. Implement Sci 2014;9(1):168. CrossRef PubMed
11. McKnight T, Demuth JR, Wilson N, Leider JP, Knudson A. Assessing Effectiveness and Cost-Benefit of the Trinity Hospital Twin City Fit For Life Program for Weight Loss and Diabetes Prevention in a Rural Midwestern Town. Prev Chronic Dis 2018;15:170479. CrossRef PubMed
12. Kochanek KD, Murphy SL, Xu J, Arias E. Mortality in the United States, 2013. NCHS Data Brief, No. 178. Hyattsville (MD): National Center for Health Statistics, Centers for Disease Control and Prevention, US Department of Health and Human Services; 2014.
13. Johnson C, Santos JA, McKenzie B, Thout SR, Trieu K, McLean R, et al. The Science of Salt: A regularly updated systematic review of the implementation of salt reduction interventions (September 2016-February 2017). J Clin Hypertens (Greenwich) 2017;19(10):928–38. CrossRef PubMed
14. Mugavero K, Losby JL, Gunn JP, Levings JL, Lane RI. Reducing sodium intake at the community level: the sodium reduction in communities program. Prev Chronic Dis 2012;9:120050. CrossRef PubMed
15. Christoforou A, Trieu K, Land MA, Bolam B, Webster J. State-level and community-level salt reduction initiatives: a systematic review of global programmes and their impact. J Epidemiol Community Health 2016 Nov;70(11):1140-1150.
16. Centers for Disease Control and Prevention. Sodium Reduction in Communities Program (SRCP): 2016–2021 comprehensive evaluation. https://www.cdc.gov/dhdsp/docs/SRCP_Eval_Design_Briefing-508.pdf. Accessed on November 8, 2018.
17. Long C, Rowland B, Langston K, Faitak B, Sparks K, Rowe V, et al. Reducing the intake of sodium in community settings: evaluation of year one activities in the Sodium Reduction in Communities Program, Arkansas, 2016–2017. Prev Chronic Dis 2018;15:180310. CrossRef
18. Escoffery C, Lebow-Skelley E, Haardoerfer R, Boing E, Udelson H, Wood R, et al. A systematic review of adaptations of evidence-based public health interventions globally. Implement Sci 2018;13(1):125. CrossRef PubMed

Top