|
|
Volume
6: No. 2, April 2009
TOOLS AND TECHNIQUES
Evaluating Partnerships to Prevent and Manage Chronic Disease
Frances Dunn Butterfoss, PhD
Suggested citation for this article: Butterfoss FD. Evaluating partnerships to prevent and manage chronic disease. Prev Chronic Dis 2009;6(2):A64.
http://www.cdc.gov/pcd/issues/2009/ apr/08_0200.htm. Accessed [date].
Abstract
To be effective and sustain themselves over time, public-private partnerships must
make evaluation a priority. Specifically, partnerships should evaluate 1) their infrastructure, function, and processes; 2) programs designed to achieve their mission, goals, and objectives; and 3) changes in health and social status, organizations, systems, and the broader community. This article describes how to 1) develop a comprehensive evaluation strategy based on partnership theory; 2) select short-term,
intermediate, and long-term indicators to measure outcomes; 3) choose appropriate methods and tools; and 4) use evaluation results to provide accountability to stakeholders and improve partnership function and program implementation.
Back to top
Introduction
Public-private partnerships fundamentally bring together the expertise of the private and public sectors and allow each to do what it does best,
so that products and services can be delivered efficiently and effectively. These partnerships also can help overcome organizational boundaries and allow parties to work together on a shared goal. For example, even though collaborating with
private-sector businesses may cause tension among nonprofit public-sector partners, the
businesses may bring new skills and funding to the
partnership and enhance the partnership’s scope of influence. An illustration: 3 partnerships were sponsored by a national association of health plans to
improve the quality of diabetes care in New Mexico, Missouri, and New York. The main evaluation finding was that the competing health plans and local organizations established and sustained viable partnerships around a shared goal, despite significant challenges (1).
The process of evaluation discovered the effects of these vital partnerships by answering the following questions: 1)
What attributes and events resulted in competing organizations coming together? 2)
What is necessary to sustain these partnerships? 3) What recommendations can
help replicate this approach for other chronic conditions in other communities?
and 4) To what extent do local market characteristics and structures set partnership direction and influence success?
Evaluation is a critical task for any partnership and determines whether the organization and its activities are sustained over time. Effective evaluation provides ongoing, systematic information that strengthens
the partnership during implementation and provides outcome data to assess the extent of change among participants or within systems (2).
Both the for-profit and nonprofit sectors place a high value on evaluation and regard it as a necessity rather than an enhancement. However, their underlying value systems and motivating factors may differ.
For the nonprofit sector, partnership evaluation fulfills underlying process goals, such as identifying new approaches, increasing community awareness and support, informing policy decisions, and contributing to the
understanding of what works (3). The public health issues have developed over time, and the
public health community’s expectations are that desired evaluation outcomes may take years or even decades to accomplish.
Conversely, the for-profit sector is driven by increasing its market share, improving
its performance through continuous innovation,
having good financial management and stability, increasing efficiency,
increasing customer accountability and satisfaction, improving its access to
government officials, and improving
public relations. The time horizon for outcomes in the business sector is often
a few months to 2 years. As the value systems of the for-profit and nonprofit sectors converge, evaluation also will become
more of a shared value. Partnering certainly can enable this process.
This article describes how evaluation is viewed by nonprofit and
for-profit sectors, levels of partnership evaluation, and a step-by-step
model for evaluating partnerships. I conclude by presenting the challenges
to evaluating partnerships and recommending solutions.
Back to top
Views on Participatory Approaches to Evaluating Partnerships
Many partnership evaluations are based on collaborative or participatory approaches according to who controls the process, who participates, and how much (4). These approaches are well suited to partnerships, although they have disadvantages
(Appendix A) (5).
Participatory approaches to evaluation are generally comfortable and customary for the public sector. The private sector may not be as familiar with these types of evaluations
nor as patient with the extra time and effort that it takes to be democratic
and attentive to the needs of all partners, priority populations, and communities served. Discussing the extra benefits that result from such
evaluations, such as better understanding and acceptance of findings that may
improve performance, may enable for-profit partners to be more open to these
approaches and learn by participating in the evaluation process
(6).
When diverse partners work together, evaluation approaches and terms
must be clarified. The private sector and the public sector may differ in their
approaches to the evaluation or assessment process, the standards and methods they use to gather data, how they define terms, and the kinds of indicators they plan to measure.
For example, the VERB campaign used the best practices of private-sector
marketing to children. The VERB brand created an emotional affinity between the
product (physical activity) and the user (tween), and engaged tweens at key
places and times when they might be both inactive and receptive to the brand
(7). The public sector redefined its terms — education had to include
persuasion, VERB was not a program but a brand, and pooling resources
with the private sector allowed the public agency, the Centers for Disease
Control and Prevention (CDC), to buy media time and the talents of marketing experts.
The agency also adjusted its concepts of performance measures and outcomes to
meet both private investors’ needs and public health’s goals.
Coming to consensus on definition of terms, methods, and measures is a crucial step in building trust when beginning a partnership evaluation. Although public health professionals may feel comfortable using the terms process, impact, and outcome measures, using the terms short-term, intermediate, and long-term indicators
is more descriptive and avoids confusion when working with partners from diverse
professional backgrounds. Different evaluation terms that the for-profit and nonprofit sectors may use in a public-private partnership are listed in
Table 1.
Back to top
Levels of Partnership Evaluation
In public-private partnerships, evaluation may measure 1) processes that sustain and renew partnership infrastructure and function; 2) programs intended to accomplish targeted activities or those that work directly toward the partnership’s goals; and 3) changes in health status or the community.
Appendix B details these 3 levels and sample measures for each. The aim of every partnership should be to evaluate something in each level.
Conducting a member survey to assess satisfaction with how the organization functions (level 1), evaluating 1 program or activity that the partnership conducts (level 2), and collecting extant data on key health indicators (level 3) are reasonable expectations for an annual evaluation plan.
Back to top
Steps in Partnership Evaluation
Many practical frameworks and models exist that can help partnerships develop evaluation plans, but the focus here is on the Framework for Program Evaluation in Public Health (8,9). The framework guides its users in selecting evaluation strategies that are useful, feasible, ethical, and accurate — its 6 steps help increase understanding of a partnership’s context as well as its outcomes. The steps have been refined and described below to apply to partnership evaluation.
Step 1: Engage the stakeholders
Engaging stakeholders means fostering participation and power sharing among people invested in the evaluation and its findings. Stakeholders include 1) those involved in program operations (eg, partners, public relations professionals, lawyers, sponsors, funders, collaborators, administrators, managers, business owners, staff), 2) those served or affected by the program (eg, clients, customers, families, neighborhood organizations, academic institutions, elected officials, advocacy groups,
professional associations, opponents), and 3) primary users of the evaluation. Stakeholders must understand the organizational structure, history, and goals of the partnership and how politics affect program implementation and impact. This
understanding can be attained by creating an environment where stakeholders discuss their values, philosophies and assumptions, and capabilities. Stakeholders may
1) provide resources for evaluation such as staff and in-kind supplies, 2)
clarify partnership goals and objectives, 3) identify and prioritize evaluation questions, 4) develop
and pilot evaluation methods and tools, 5) collect data, and 6) interpret and
report results (10).
Stakeholders’ needs, concerns, and demands for specific outcomes differ widely, even though they may agree with the partnership’s goals and objectives. To motivate
the stakeholders to participate in the partnership and its evaluation, data
could be gathered from them about what they need the evaluation to measure. A sample set of
criteria is included in
Table 2 (11).
Step 2: Describe the partnership
This description should focus on the purpose, goals, objectives, resources, current and planned activities, expected outcomes, stage of development, and environmental context of the partnership.
In a public-private partnership, objectives are based on compromise among partners with different political, social, and economic aims. Divergent interests concerning actions and expected impacts must be taken into account (11). Developing a logic model is
1 way
to help stakeholders clarify the partnership’s rationale, strategies, and conditions. It serves as a road map of the program, prioritizes the sequence of activities, summarizes expected change by linking processes to eventual outcomes, shows how partnership programs are linked to other ongoing efforts, and displays the infrastructure needed to support the partnership (9). An effective logic model will be refined and changed many times, as the partners learn about how and why the
partnership works. A sample logic model for a state partnership is in CDC’s National Heart Disease and Stroke Prevention Program’s Evaluation Guide (10).
Partnerships develop in stages: 1) formation — initial building of the organization, 2) implementation — strategic planning and conducting of activities to address goals, 3) maintenance — sustaining activities until goals are accomplished, and 4) institutionalization — collaborative attainment of goals
in permanent structures within the community (12,13). The current stage of the partnership should be assessed to determine the proper focus for evaluation (8). For example,
evaluation activities should focus on identifying and recruiting partners if the partnership is in the formation stage; communication and decision making in the maintenance stage; and community changes in the institutionalization stage. A comprehensive evaluation of a mature partnership includes measures at each stage.
Step 3: Focus the evaluation design
The evaluation should focus on issues of greatest concern to stakeholders, while efficiently
using time and resources. A written plan that summarizes evaluation goals and procedures and outlines the stakeholders’ roles and responsibilities is essential. The plan should include evaluation questions and practical methods for sampling, data collection, data analysis, and interpretation. Stakeholders can help prioritize the questions to determine which are critical, are likely
to improve the partnership, and can be answered with available resources. Questions may include
the following:
- What should the partnership accomplish and how will it be demonstrated?
- What activities will the partnership undertake to accomplish its goals?
- What factors might help or hinder the accomplishment of its goals?
- Who are the partners (number, diversity, and participation levels)?
- How do partners work together?
- What partnership outcomes should be measured?
The evaluation design is linked to the priority questions, and the choice of design has implications for what data will be collected and how. A pretest-posttest design uses a comparison group, measures the partnership on given parameters before and after it implements planned improvement strategies,
or both. A case study design is used to study the partnership’s context, history, structure, and function. Case studies usually rely on multiple sources of information such as observations,
interviews, audiovisual material, documents, and reports.
Appendix C provides a sample evaluation plan for
partnerships (14).
Step 4: Gather credible evidence
After deciding on the evaluation questions and design, the partnership must
decide what data it needs to answer the questions, where and how the data can be obtained, and how the data should be analyzed and used. Adequate data may be available and easily accessed, or new data may have to be collected. Evaluation data should provide a well-rounded picture of the partnership and its programs
so stakeholders can perceive the results as believable and relevant. Integrating
qualitative and quantitative data increases the likelihood that data will be balanced and accepted by all stakeholders (15).
For each evaluation question, at least 1 indicator or data point must be defined and tracked. Examples of indicators for partnerships might include measures of 1) partnership effectiveness (eg, participation in meetings and activities, usefulness of partnership structures), 2) partnership activities (eg, participation rate, completion of state plan objectives), and 3) partnership effects (eg, number of
policies or practices that were amended or adopted, health status changes). Practitioners and researchers have summarized measures that document changes in partnership knowledge, attitudes, practices, community environment, policies, and health status (2,16,17).
For each evaluation question and indicator, sources of data must be identified. Data from documents, key informant interviews, meeting observations, member surveys, and focus groups provide different perspectives of the partnership and enhance the comprehensiveness and credibility of the evaluation. Census data (including economic data and demographics), health survey data (eg, Behavioral Risk Factor Surveillance System survey
results), or behavioral outcome data (eg, emergency medical
transports, hospital admissions) represent likely data sets that partnerships may use to assess health and quality of life status. A rule of thumb is to collect only data that will be used and to use all data collected.
In deciding what instruments to use, partnerships may develop their own questionnaires or interview frameworks, use validated and reliable tools, or modify an existing tool to fit their priority population(s), community culture, and issues. Coalitions and Partnerships in Community Health (2) lists tools and resources for these evaluations. Stakeholders should develop clear procedures for gathering, analyzing, and interpreting data, and training staff, partnership members, and volunteers
to collect quality data.
Step 5: Justify conclusions
After designing an evaluation, data must be collected, described, analyzed, and synthesized to summarize the findings, then interpreted to decide what it means in the context of the partnership. Investing enough time and resources in analysis and interpretation is critical because this is when decisions are made and actions are taken. Once data are collected, they are
returned to stakeholders for reflection and verification. Stakeholders should look beyond the raw data to
ask what the results mean, what led to the findings, and whether they are significant. Each partner has different criteria for judging success and weighs them differently; using multirater analysis may help (11). Conclusions are justified and will be used with confidence when they are supported by data, consistent with the agreed-on values of the stakeholders, and linked to recommended actions.
Step 6: Ensure use and share lessons learned
The partnership should provide continuous feedback to stakeholders regarding interim findings to ensure that evaluation conclusions lead to appropriate decisions or actions. Stakeholders are more likely to use evaluation results if they feel they own the evaluation process and if they function cohesively as a team. During each planning and implementation step, stakeholders should discuss the best ways to communicate evaluation findings and use them. Frequent communication will increase
the commitment to act on the results and refine the evaluation design, questions, methods, and interpretations. Having a positive experience with evaluation changes participants’ attitudes; they begin to base decisions on judgments instead of assumptions (6).
Back to top
The Challenges of Evaluating Partnerships
Researchers agree that partnerships are difficult to evaluate. Measuring system-level changes is more difficult than evaluating program outcomes because multiple levels and community readiness must be considered (18). Other practical and methodologic issues include the following:
- The partnership’s planning process does not include evaluation. Resources are often inadequate or are more likely to be spent on interventions. Because evaluation is costly
in time and resources, the partnership is not always committed to do it. If
evaluation is supported, staff are motivated to make partnership programs look effective to maintain funding or jobs. Evaluation may not be based on a solid logic model or theory, and partnerships may fail to find the right evaluators or
tools for evaluating partnership processes and outcomes (19).
- Each partnership is unique. Partnerships are embedded within communities and responsive to their cultural contexts.
For this reason, they tend to be unique, difficult to replicate, and unrepresentative of other partnerships, even those that address similar issues.
- The design and methods of the evaluation can make generalization difficult. Establishing and measuring outcomes, controlling extraneous variables that interact with outcomes, accounting for secular trends over the partnership’s development, and addressing the political realities to satisfy funders make it challenging to detect systems-level change (20).
Extraneous variables (eg, new government programs, changes in funding streams)
are difficult to control and may interact with each other or influence outcomes
by changing how programs are implemented. Partnerships may not identify outcome
measures or link them to appropriate intermediate outcomes. Even when
comparable long-term outcomes are measured across sites, baseline data may not
be available. Finally, distinguishing between cause and effect or the percentage
of the outcome that can be attributed to each partnership activity is difficult
(20-22).
Back to top
Recommendations to Improve Partnership Evaluation
Partnerships may be the best vehicles available to address the chronic diseases of our time, so evaluation methods continuously
need to be refined. The following are proposed solutions to partnership evaluation issues (2,3,20,23-27):
- Use innovative, qualitative evaluation methods. Rely on qualitative methods that represent the community and try to figure out how partnerships make a difference. Innovative methods need to be developed to address the dynamic nature of partnerships (25).
- Focus on evaluating practice-proven strategies and measurable outcomes. Partnerships are best suited to assessment and priority setting rather than implementing projects (4). Evaluators must concern themselves with short-term, immediate, and long-term effects of the partnership. In addition to health and social outcomes, evaluation should focus on how partnerships build capacity by improving outcomes related to participation, leadership, networks, skills, resources, and sense
of community (28). Similarly, evaluators may determine whether a partnership is on track to become empowered and sustained by noting outcomes such as community infrastructure improvements, economic enhancements, educational opportunities, and policy changes (29).
- Provide needed training and technical assistance. Appropriate training, technical assistance, and resources for conducting effective evaluations should be available to partnerships, so they learn how to translate evaluation results into actionable tasks.
- Help partnerships “begin where they are.” Most partnerships view evaluation as a formidable task and choose not to evaluate. They are overwhelmed by technical tasks, time and financial costs, and concerns that they might not “measure up.” Partnerships should be encouraged to start small and evaluate something. They might choose to evaluate 1 aspect of the partnership from each of 3 levels (short-term, intermediate, and long-term) as a starting point. Existing data can be evaluated with
little or no cost. As examples, partner diversity can be determined by assessing the roster; attendance patterns can be derived from the meeting minutes. As confidence and skills grow, partners may be encouraged to engage in new and more complex evaluation tasks.
Back to top
Conclusion
Public-private partnerships can be powerful agents for preventing and
managing chronic disease. However, such partnerships become more complex as the
public sector works more closely with private-sector partners. The following
issues must be considered in developing evaluations that lead to improvements in
partnerships and their programs and services:
- What common evaluation criteria can be agreed on (eg, frequency of evaluation)?
- How can the environment for public-private partnerships be assessed?
- How can partnerships obtain adequate resources to conduct effective evaluations?
- What are the roles of various stakeholders in the evaluation? How can technical and evaluation capacity be fostered?
- How can evaluation be built into the framework of the partnership?
- Can existing tools and methods be adapted to meet public and private partners’ needs?
How well these issues are addressed will help determine the effectiveness of partnership evaluations. Evaluations that meet stakeholder needs and focus on mutually acceptable and measurable systems-level outcomes will make partnership support and sustainability more likely in the end.
Back to top
Author Information
Frances Dunn Butterfoss, PhD, Coalitions Work, 1109 Moore House Rd, Yorktown, VA 23690. Telephone: 757-898-7454. E-mail:
fran.butterfoss@coalitionswork.com.
Back to top
References
- Palsbo SE, Kroll T, McNeil M. Addressing chronic conditions through community partnerships: a formative evaluation of
Taking on Diabetes. New York (NY): The Commonwealth Fund; 2004.
- Butterfoss FD. Coalitions and partnerships in community health. San Francisco (CA): Jossey-Bass; 2007.
- Butterfoss FD, Francisco VT.
Evaluating community partnerships and
coalitions with practitioners in mind. Health Promot Pract 2004;5(2):108-14.
- Butterfoss FD, Francisco V, Capwell EM. Stakeholder participation in
evaluation. Health Promotion Practice 2001;2(2):114-9.
- Dugan MA. Participatory and empowerment evaluation. In: Fetterman DM, Kafterian SJ, Wandersman
A, editors. Empowerment evaluation: knowledge and tools for self-assessment and accountability. Thousand Oaks (CA): Sage
Publications; 1996.
p. 277-303.
- Patton MQ. Toward distinguishing empowerment evaluation and placing it in a larger context. Eval Pract 1997;18(2):147-63.
- McKinnon R.
Lessons from VERB:
A case for branding in nutrition education. J Nutr Ed Behav 2007;39(2 Suppl):S53-4.
- Butterfoss FD, Kegler MC. Toward a comprehensive understanding of community
coalitions: moving from practice to theory. In: DiClemente RJ, Crosby RA, Kegler MC, editors. Emerging theories in health promotion practice and research. San Francisco (CA): Jossey-Bass;
2002. p. 157-93.
- Centers for Disease Control and Prevention.
Framework for program evaluation in public health. MMWR
Recomm Rep 1999;48(RR-11):1-40.
- Centers for Disease Control and Prevention. Evaluation guide: fundamentals of evaluating partnerships. Atlanta (GA): US Department of Health and Human Services; 2007.
- Toulemonde J, Fontaine C, Laudren E, Vincke P. Evaluation in partnership: practical suggestions for improving their quality. Evaluation 1998;4(2):171-88.
- Butterfoss FD, Gilmore LA, Krieger JW, LaChance LL, Lara M, Meurer JR, et
al.
From formation to action: How Allies Against Asthma coalitions are getting
the job done. Health Promot Pract 2006;7(2 Suppl):34S-43S.
- Nicholas EA, Dwyer M, Murphy A, Taylor-Fishwick JC, Cohn JH, Kelly CS, et
al.
Coalition-based approaches for addressing environmental issues in childhood
asthma. Health Promot Pract 2005;7(2 Suppl):108S-16S.
- Francisco VT, Capwell EM, Butterfoss F. Getting off to a good start with your evaluation.
Health Promot Pract 2000;1(2):126-31.
- Francisco VT, Butterfoss FD, Capwell EM. Key issues in evaluation:
quantitative and qualitative methods and research design. Health Promot Pract 2001;2(1):20-3.
- Granner ML, Sharpe PA.
Evaluating community
coalition characteristics and
functioning: a summary of measurement tools. Health Educ Res 2004;19(5):514-32.
- Butterfoss FD.
Process evaluation for community participation. Annu Rev Public Health 2006;27:323-40.
- Goodman RM, Wandersman A, Chinman M, Imm P, Morrisey E.
An ecological assessment of community-based interventions for prevention and health promotion:
approaches to measuring community coalitions. Am J Community Psychol 1996;24(1):33-61.
- Wolff T. A practical approach to evaluation of collaborations. In: Backer TE, editor. Evaluating community collaborations. New York (NY): Springer Publishing Co; 2003.
- Berkowitz B.
Studying the outcomes of community-based coalitions. Am J Community Psychol 2001;29(2):213-27.
- Kreuter MW, Lezin NA, Young LA. Evaluating community-based collaborative
mechanisms: implications for practitioners. Health Promot Pract 2000;1(1):49-63.
- Roussos S, Fawcett ST.
A review of collaborative partnerships as a strategy for improving community health. Annu Rev Pub Health 2000;21:369-402.
- Birkby B. Community partnerships: questions, controversy, and context. Prevention Evaluation Perspectives 2003;1(1):1-7.
- Zakocs RC, Edwards EM.
What explains community coalition effectiveness?: A review of the literature. Am J Prev Med 2006;30(4):351-61.
- Gabriel RM. Methodological
challenges in evaluating community partnerships and coalitions: still crazy
after all these years. Journal of Community Psychology 2000;28(3):339-52.
- Wandersman A, Florin P.
Community interventions and effective prevention. Am Psychol 2003;58:441-8.
- Yin RK, Ware AJ. Using outcome data to evaluate community drug prevention initiatives:
pushing the state-of-the-art. Journal of Community Psychology 2000;28(3):323-38.
- Kegler MC, Norton BL, Aronson RE. Evaluation of the five-year expansion program of California Healthy Cities and Communities (1998-2003). Sacramento (CA): Center for Civic Partnerships; 2003. http://www.civicpartnerships.org/docs/publications/ExecSummary.pdf.
Accessed November 25, 2008.
- Reininger B, Martin DW, Ross M, Sinicrope PS, Dinh-Zarr T.
Advancing the theory and measurement of collective empowerment:
a qualitative study. Int Q Community Health Educ 2005-2006;25(3):211-38.
Back to top
|
|