An Evaluation of Health Impact Assessments in the United States, 2011 - 2014

Emily Bourcier, MPH, MHA; Diana Charbonneau, MIT; Carol Cahill, MLS; Andrew L. Dannenberg, MD, MPH

Suggested citation for this article: Bourcier E, Charbonneau D, Cahill C, Dannenberg AL. An Evaluation of Health Impact Assessments in the United States, 2011–2014. Prev Chronic Dis 2015;12:140376. DOI: http://dx.doi.org/10.5888/pcd12.140376external icon.

PEER REVIEWED

PEER REVIEWED

Abstract

Introduction
The Center for Community Health and Evaluation conducted a 3-year evaluation to assess results of health impact assessments (HIAs) in the United States and to identify elements critical for their success.

Methods
The study used a retrospective, mixed-methods comparative case study design, including a literature review; site visits; interviews with investigators, stakeholders, and decision makers for 23 HIAs in 16 states that were completed from 2005 through 2013; and a Web-based survey of 144 HIA practitioners.

Results
Analysis of interviews with decision makers suggests HIAs can directly influence decisions in nonhealth-related sectors. HIAs may also influence changes beyond the decision target, build consensus and relationships among decision makers and their constituents, and give community members a stronger voice in decisions that affect them. Factors that may increase HIA success include care in choosing a project or policy to be examined’ selecting an appropriate team to conduct the HIA; engaging stakeholders and decision makers throughout the process; crafting clear, actionable recommendations; delivering timely, compelling messages to appropriate audiences; and using multiple dissemination methods. Challenges to successful HIAs include underestimating the level of effort required, political changes during the conduct of the HIA, accessing relevant local data, engaging vulnerable populations, and following up on recommendations.

Conclusion
Results of this study suggest HIAs are a useful tool to promote public health because they can influence decisions in nonhealth-related sectors, strengthen cross-sector collaborations, and raise awareness of health issues among decision makers.

Top

Introduction

A health impact assessment (HIA) is used to communicate between public health professionals and decision makers in other sectors (eg, public policymakers) to increase stakeholder input and use of public health data in decisions that affect the public but are unrelated (or seemingly unrelated) to public health. HIAs convey to decision makers the potential health effects of proposed projects and policies such as those related to land use and transportation, and they make recommendations that promote the beneficial and mitigate the adverse health effects of such projects and policies. An HIA is “a systematic process that uses an array of data sources and analytic methods and considers input from stakeholders to determine the potential effects of a proposed policy, plan, program, or project on the health of a population and the distribution of those effects within the population. HIA provides recommendations on monitoring and managing those effects” (1).

Use of HIAs has been increasing since their introduction in Europe in the 1990s. Over 300 HIAs were completed or in progress in the United States as of mid-2014 (2). Prior reports called for evaluation to determine whether HIAs have the expected effects, to improve HIA methods, and to justify continuing investing in HIAs (1,3). Types of HIA evaluation include process, impact, and outcome evaluations (1). Process evaluation compares the processes followed in conducting the HIA with the investigators’ intended plans or the guidelines for an HIA (4,5). Impact evaluation, the focus of this article, examines the HIA’s effect on subsequent decisions and related events. Outcome evaluation examines changes in health status or health determinants resulting from the HIA; such evaluation is difficult and rarely done.

Two process evaluations (1 of 81 HIAs [6] and 1 of 23 HIAs [7]) in the United States show substantial variation in the conduct of HIAs in relation to practice guidelines and standards. Results of process evaluations were published for some individual HIAs (8,9). An impact evaluation of 88 HIAs, primarily in Europe, identified enablers and barriers to HIAs’ influencing decisions similar to those reported in this study (10). A process and impact evaluation of 17 case studies found that a wide variety of processes were being used by HIA practitioners in Europe (11). A 4-cell matrix for assessing HIA effectiveness developed in that evaluation was used in subsequent HIA evaluations (6,12). A process evaluation of 55 HIAs in Australia and New Zealand found that 65% of the reports that used a standard report review package were adequate (12). An impact evaluation of 11 of those 55 HIAs found most were effective, usually by directly influencing the decision process of nonhealth-related policymakers or raising stakeholder awareness of health issues (13).

The Robert Wood Johnson Foundation (RWJF), a major funder of HIAs in the United States, commissioned the Center for Community Health and Evaluation to study what constitutes success for HIAs, document their benefits, clarify and explain factors that contribute to successful HIAs, and look for opportunities to improve the HIA field. This is the first national study of HIAs that emphasizes the perspectives of decision makers, who are in the best position to indicate whether an HIA influenced their decision.

Top

Methods

We conducted a literature review of previous HIA evaluations and of factors that increase the likelihood of HIA success. HIA success was initially defined as the degree to which its recommendations were incorporated into a decision in a nonhealth related sector. A logic model (Figure) was created to define the intervention being evaluated, depict the intermediate and long-term outcomes that HIAs are intended to achieve, and guide development of the data collection instruments. An 8-member advisory committee with representatives from universities, philanthropic organizations, and HIA-focused organizations provided guidance on study design, interview and web-based survey questions, and analysis.

Figure 1. Logic model of a typical health impact assessment (HIA). The evaluators developed the logic model to depict the intermediate and long-term outcomes HIAs are intended to achieve and to guide development of the data collection instruments. The activities are aligned with the recommended six-step HIA framework (1). ¹ Includes financial resources, staffing, technical assistance, local knowledge, and advocacy. ² Improved capacity to conduct HIAs, enhanced decision-making ability. ³ Outcomes that are not within the scope of this evaluation. [A text version of this figure is also available.]

Case studies

For this evaluation, the researchers used a retrospective comparative case study approach (14), including in-depth case studies of 23 completed HIAs. At the request of the evaluation’s funder, the study sample included 10 HIAs supported by the Health Impact Project, a collaboration of RWJF and The Pew Charitable Trusts, and 4 by the RWJF Active Living Research initiative. An additional 13 HIAs were found through consultation with HIA experts and considered by them to have elements that were successful. The study team recruited the investigators of 23 of these 27 HIA cases to participate in the evaluation; 2 HIA investigators declined to participate, 1 HIA did not self-identify as an HIA, and 1 HIA was replaced by another to add more geographic and investigator diversity. Overall the sample was purposefully selected for variety in geography, sector, and funding source.

The investigators scheduled site visits at least 6 months after the HIA was released for 21 cases; less time had passed for the other 2 cases. The evaluation team worked with each HIA lead investigator to identify key informants and then scheduled interviews with the HIA team, 1 or 2 decision makers, and 1 or 2 community stakeholders for each HIA.

The investigators developed a semi-structured interview guide, with open-ended questions tailored to HIA team members, decision makers, and community stakeholders. Questions covered origins of the HIA, decision maker and stakeholder involvement, time and resource use, how the HIA used data to support its recommendations, the degree to which recommendations were implemented, major impacts of the HIA, factors that facilitated successes or created challenges, and opportunities to increase the success of HIA. Detailed narrative data were gathered on each HIA from multiple perspectives. Decision makers included federal, state, and local elected and appointed officials; high-level agency staff; and private sector leaders. Interviews ranged from 30 to 90 minutes and were conducted between March 2012 and March 2013 with 166 key informants: 119 interviews were in person and 47 were by phone. Informants were 78 HIA practitioners, 47 decision makers, and 41 community members. For quality assurance, 6 of the 23 HIA site visits were conducted by 2 members of the evaluation team, and interviews for 2 HIAs were independently coded by 2 people.

Transcripts were uploaded into the qualitative analysis software program Atlas.ti 6.0 (Atlas.ti, Corvallis, Oregon). Using grounded theory techniques (15) and a code list based on the evaluation questions and the logic model, team members who conducted the interviews coded the data. The coded data were analyzed using an immersion/crystallization approach (16), which emphasizes gaining an in-depth knowledge of the data to identify key themes. Themes were aggregated into a case study template. Each case study was reviewed by all 3 team analysts to ensure accurate representation of the data. The evaluation team then reviewed the data again to identify cross-cutting themes and draw conclusions. The team organized the evaluation domains of interest (eg, impact, success factors, challenges) into 8 tables to synthesize the themes across the 23 cases (17).

Web-based survey

In January 2013, a national Web-based survey of HIA practitioners was sent via SurveyMonkey (www.surveymonkey.com) to a convenience sample of 121 people associated with HIAs who were identified from the Health Impact Project and University of California, Los Angeles–HIA websites. In March 2013 this survey was sent to an additional 22 people who were among attendees of the HIA of the Americas meeting. In February and March 2013 the survey was promoted to recipients of email newsletters from the Health Impact Project and Human Impact Partners. Multiple reminders were sent, and the survey closed in April 2013. The questions were primarily closed-ended and focused on the impact of HIAs and success factors.

Top

Results

Case studies

The sample consisted of 23 HIAs in 16 states completed from 2005 through 2013 (Table 1). Sectors represented among these HIAs were 11 built environment, 3 transportation, 3 natural resources and energy, 2 food and agriculture, 1 housing, 1 economic policy, 1 climate change, and 1 labor and employment. Of the 23 HIAs, 14 were funded by RWJF, 4 by government agencies, 3 by other foundations, and 2 by other sources.

Identified impact

Table 2 presents the identified impacts of the 23 HIAs. In 11 HIAs, decision makers reported their decisions would have been markedly different without the HIA. Decision target outcomes — the way projects, plans, and policies were developed or implemented — could be directly linked to specific HIA recommendations in 11 HIAs. This, in turn, altered the trajectory of a policy or plan in ways that were intended to improve health or mitigate potentially adverse health consequences. Fourteen HIAs influenced changes beyond the decision under consideration, resulting in the incorporation of health objectives into plans, policies, and programs established by nonhealth-related agencies, and 8 HIAs — some of which overlap with the aforementioned 14 — contributed to longer-term outcomes beyond initial decision targets.

HIAs contributed to various other impacts. Seventeen HIAs institutionalized or strengthened pre-existing relationships among individuals and organizations, or they created new and enduring relationships between public health and other agencies such as transportation or planning departments. Sixteen HIAs helped decision makers and stakeholders understand how health is connected to seemingly unconnected issues. HIAs helped build consensus around controversial topics. In 9 HIAs, the process amplified community member voices in the decision making process.

Reported success factors

The evaluation identified components that HIA teams can incorporate into their assessments to increase the likelihood of HIA success. These components (or “success factors”) clustered into 7 themes (Table 3):

Screen and choose HIA targets wisely because different decision contexts require different HIA approaches, and HIA is not always the right tool. To help decide whether to conduct an HIA and to determine its scope, HIA practitioners recommend the HIA screening step include assessment of the following issues: whether momentum is already building for the issue, whether the decision makers have basic knowledge about health-related issues, what relationships exist between those conducting the HIA and the decision makers, and how the HIA timing aligns with the decision-making process.

Invest in the right team to conduct the HIA. The composition of the HIA team is crucial, since successful HIAs rely on many kinds of expertise and require sustained collaborative effort. Teams should include experts in the content related to the decision under consideration, knowledge of the decision-making process, and skills in project management and stakeholder engagement.

Engage key stakeholders. Community members, influential champions, and other stakeholders can build momentum for considering and adopting HIA recommendations. Stakeholders are valuable sources of political expertise and community knowledge.

Engage decision makers throughout the process. Bringing decision makers on board as stakeholders or team members increases the likelihood that HIA recommendations will receive appropriate consideration.

Craft clear recommendations that spark action. To increase the likelihood of adoption and implementation, recommendations should be actionable, realistic, and sector-specific; consider the implementer’s authority to act; and address timelines and costs. A decision maker reading an HIA report should have a clear idea of what to do next.

Deliver compelling messages to the right audiences at the right times. HIA teams should make complex information accessible to many audiences throughout the HIA process. In retrospect, some HIA team members wished they had communicated HIA information through multiple channels and earlier, not only at the recommendations stage. Many advised that HIAs should consider both the content and strategic timing of their dissemination efforts and tailor these to the needs of different audiences (eg, decision makers, business audiences, community members).

Take advantage of HIA credibility. Because HIAs combine stakeholder input and robust public health evidence, they can provide a credible evidence-based rationale for recommendations in the setting of complex policy and planning decisions.

Reported challenges

Common challenges to conducting successful HIAs cluster into 6 themes (Table 3). For most HIAs, the time and resources required to conduct the HIA were greater than expected. Engaging stakeholders and decision makers proved more difficult than anticipated for most HIAs because of competing time demands and political considerations. In some cases the decision-making process moved faster than the HIA process, thereby undermining the usefulness of recommendations.

In some cases, relevant data were not available at a sufficiently local level to document disparities in impacts. Some HIA investigators reported difficulties in engaging members of vulnerable populations, and approaches used reflected different purposes and philosophies. Finally, adequate dissemination and follow-up on implementation were absent for many HIAs, thus reducing the likelihood of the recommendations having an impact on decisions.

Web survey results

Of the 173 survey respondents, 144 had conducted at least 1 HIA and self-identified as HIA practitioners. In this convenience sample, the number of practitioners who received the survey and the resulting response rate are unknown. Considering the number of HIAs completed in the United States, the survey respondents probably constitute a substantial proportion of all US HIA practitioners. Of the 144 respondents, 47% worked for government agencies, 23% for academic institutions, 20% for nonprofit organizations, 7% for for-profit organizations, and 2% for other entities. The 135 respondents who identified their location were from 1 of 32 states or 5 non-US countries. The amount of overlap between the anonymous survey respondents and case study informants is unknown.

In response to a closed-ended question, the most important outcomes expected by respondents from an HIA were recommendations’ influence on decision-making (87%), inclusion of health-related objectives in other sector plans (68%), enduring cross-sector partnerships (64%), a common language and new ways of framing health issues (35%), increased community capacity to make decisions and conduct HIAs (33%), and changed attitudes (25%). For respondents’ most successful HIA completed to date, the response of the decision maker to the HIA was reported as “thrilled” for 8% and “receptive” for 62%; 15% of respondents reported a mixed decision maker response, 10% a neutral response, 3% decision maker pushback, and 2% did not know. Eighty percent reported engaging representatives from vulnerable populations in their most recent HIA.

Most respondents (84%) had been involved in an HIA they considered successful. Types of success observed by these respondents included contributions to cross-sector coalitions or partnerships (35%), influence on decision-making (19%), and contributions to positive, detectable changes in the community (6%). These proportions of observed impacts are lower than the respondents’ expected impacts noted above.

Top

Discussion

By obtaining information directly from decision makers who know what influenced their decisions, this impact evaluation provides strong evidence that HIAs can influence decisions to promote health considerations in projects and policies outside of the health sector. This study also documents how HIAs can influence changes beyond the decision target, by raising decision makers’ awareness of the effects of their decisions on people’s health and by giving community members a stronger voice in decisions that affect them. Success for HIAs should therefore be defined by both their impacts on decisions as well as on the environments in which those decisions are made.

On the basis of study results, we have several recommendations for improving the conduct of HIAs and building the field. Pay more attention to the needs of vulnerable populations by focusing on practical factors such as the resources and skills needed, the HIA timeline, and how shared expectations are developed and managed. HIA teams could make strong calls for action to increase the likelihood of the recommended actions being implemented. Provide an implementation plan in their recommendations, including a list of who should be responsible for each task.

Some HIAs are done by investigators with little HIA experience who welcome technical assistance. We have recommendations also for providers of such technical assistance: focus more on HIA components that are challenging, such as engaging stakeholders in meaningful ways (18,19); cultivate relationships with decision makers and potential adversaries; facilitate high-functioning HIA teams; obtain and use locally relevant data; tailor separate communication products for various audiences; and suggest tools to help prospective HIA investigators determine whether an HIA is appropriate and how to define its scope.

On the basis of information from key informants, we have several recommendations for funders on how to improve the success of HIAs. For example, be more flexible about timelines and deliverables because HIA teams must often respond to changing political and decision-making conditions; build community stakeholder engagement into their rating criteria, and fund elements that support community engagement, such as facilitation training and food for meetings; and require and support an implementation plan as part of the HIA report, while recognizing that the plan may extend beyond the original funding period.

This report adds to the growing literature on the evaluation of HIAs as a field (6–13). Study strengths include the decision maker perspective on impact and which elements make HIA more influential and useful, its relatively large and diverse sample size, and the resources to gather data from multiple perspectives. Study staff achieved excellent cooperation with investigators, stakeholders, and decision makers of the individual HIAs and conducted over 120 hours of in-person interviews with these key informants. The consensus-based approach and use of 3 evaluation staff members to both gather and analyze the data ensured rigorous analysis of the enormous amount of qualitative data. This study was also strengthened by the expertise of its investigators in evaluation and in HIA, by interactions with an experienced advisory committee, and by collaboration with investigators of other HIA evaluations simultaneously under way in the United States and Australia.

Several limitations should be considered in interpreting study results. First, the 23 HIAs examined in this report are not necessarily representative of all HIAs in the United States; the sample was purposefully selected for diversity and was not a random sample. Second, it is difficult to separate impacts of HIA from other influences on decisions. In any decision, health considerations need to be balanced with economic, political, social, and other factors, and a decision maker may be unable to state how much weight was given to each factor. Third, the amount of time from the HIA’s completion to the informant interviews varied. In some cases decisions had been made several years ago, thus introducing possible recall bias, whereas in other cases decisions were ongoing at the time of the interview. Fourth, some HIAs involved politically sensitive topics about which some informants may have been reluctant to provide full information.

This report documents that HIAs can influence decision making and strengthen cross-sector collaborations in ways that improve the health of a community. The most promising tactics for increasing HIA effectiveness include choosing HIA targets well, gathering the right HIA team, engaging stakeholders and decision makers, creating actionable recommendations, delivering compelling messages, and establishing and following through on an implementation plan.

Top

Acknowledgments

This work was supported by a contract from RWJF to the Group Health Research Institute. RWJF staff participated in periodic telephone calls with study investigators during the study and commented on a draft of this article. We thank all informants who participated in the detailed interviews. We also appreciate the advice from members of the project advisory committee, especially Aaron Wernham and Naima Wong.

Top

Author Information

Corresponding Author: Emily Bourcier, MPH, MHA, Center for Community Health and Evaluation, Group Health Research Institute, 1730 Minor Ave, Ste 1600, Seattle, WA 98101. Telephone: 206-287-2035. Email: bourcier.e@ghc.org.

Author Affiliations: Diana Charbonneau, Carol Cahill, Center for Community Health and Evaluation, Group Health Research Institute, Seattle, Washington; Andrew L. Dannenberg, University of Washington School of Public Health, Seattle, Washington.

Top

References

1. Committee on Health Impact Assessment. Improving health in the United States: the role of health impact assessment. Washington (DC): The National Academies Press; 2011. http://www.nap.edu/catalog.php?record_id=13229. Accessed August 19, 2014.
2. HIA in the United States. Washington (DC): Health Impact Project; 2014. http://www.healthimpactproject.org/hia/us. Accessed August 19, 2014.
3. Dannenberg AL, Bhatia R, Cole BL, Dora C, Fielding JE, Kraft K, et al. Growing the field of health impact assessment in the United States: an agenda for research and practice. Am J Public Health 2006;96(2):262–70. CrossRefexternal icon PubMedexternal icon
4. Bhatia R, Branscomb J, Farhang L, Lee M, Orenstein M, Richardson M. Minimum elements and practice standards for health impact assessment, version 2. Oakland (CA): North American HIA Practice Standards Working Group; 2010. http://www.humanimpact.org/doc-lib/finish/11/9. Accessed August 19, 2014.
5. Hebert KA, Wendel AM, Kennedy SK, Dannenberg AL. Health impact assessment: a comparison of 45 local, national, and international guidelines. Environ Impact Assess Rev 2012;34(34):74–82. CrossRefexternal icon
6. A review of health impact assessments in the U.S.: current state-of-science, best practices, and areas for improvement. Washington (DC): Environmental Protection Agency; 2013. http://www.epa.gov/research/healthscience/health-review-hia.htm. Accessed August 19, 2014.
7. Schuchter J, Bhatia R, Corburn J, Seto E. Health impact assessment in the United States: has practice followed standards? Environ Impact Assess Rev 2014;47(47):47–53. CrossRefexternal icon
8. Mathias KR, Harris-Roxas B. Process and impact evaluation of the Greater Christchurch Urban Development Strategy Health Impact Assessment. BMC Public Health 2009;9:97. CrossRefexternal icon PubMedexternal icon
9. Hoehner CM, Rios J, Garmendia C, Baldwin S, Kelly CM, Knights DM, et al. Page Avenue health impact assessment: building on diverse partnerships and evidence to promote a healthy community. Health Place 2012;18(1):85–95. CrossRefexternal icon PubMedexternal icon
10. Davenport C, Mathers J, Parry J. Use of health impact assessment in incorporating health considerations in decision making. J Epidemiol Community Health 2006;60(3):196–201. CrossRefexternal icon PubMedexternal icon
11. Wismar M, Blau J, Ernst K, Figueras J. The effectiveness of health impact assessment: scope and limitations of supporting decision-making in Europe. Brussels (BE): European Observatory on Health Systems and Policies; 2007. http://www.euro.who.int/__data/assets/pdf_file/0003/98283/E90794.pdf. Accessed August 19, 2014.
12. Haigh F, Harris E, Ng Chok H, Baum F, Harris-Roxas B, Kemp L, et al. Characteristics of health impact assessments reported in Australia and New Zealand, 2005–2009. Aust N Z J Public Health 2013;37(6):534–46. CrossRefexternal icon PubMedexternal icon
13. Haigh F, Baum F, Dannenberg AL, Harris MF, Harris-Roxas B, Keleher H, et al. The effectiveness of health impact assessment in influencing decision-making in Australia and New Zealand, 2005-2009. BMC Public Health 2013;13:1188. CrossRefexternal icon PubMedexternal icon
14. Yin RK. Case study research: design and methods. Thousand Oaks (CA): Sage Publications; 2003.
15. Corbin JM, Strauss AL. Basics of qualitative research: techniques and procedures for developing grounded theory. Thousand Oaks (CA): Sage Publications; 2008.
16. Borkan J. Immersion/crystallization. In: Crabtree BF, Miller WL, editors. Doing qualitative research. Thousand Oaks (CA): Sage Publications; 1999.
17. Bourcier E, Charbonneau D, Cahill C, Dannenberg A. Do health impact assessments make a difference? A national evaluation of HIAs in the United States: supplement. Seattle (WA): Center for Community Health and Evaluation; 2014. http://www.rwjf.org/content/dam/farm/reports/issue_briefs/2014/rwjf409204/subassets/rwjf409204_1. Accessed August 19, 2014.
18. Tamburrini A, Gilhuly K, Harris-Roxas B. Enhancing benefits in health impact assessment through stakeholder consultation. Impact Assessment and Project Appraisal 2011; 29(3):195-204. http://www.tandfonline.com/doi/abs/10.3152/146155111X12959673796281. Accessed November 24, 2014.
19. Stakeholder Participation Working Group of the 2010 HIA of the Americas Workshop. Guidance and best practices for stakeholder participation in health impact assessments. Version 1.0. March 2012. http://www.humanimpact.org/downloads/guidance-and-best-practices-for-stakeholder-participation-2012/ Accessed November 24, 2014.

Top

Tables

Table 1. Characteristics of 23 Health Impact Assessments (HIAs) Examined in Depth in Center for Community Health and Evaluation HIA Evaluation Study, 2011–2014

Table 2. Reported Impacts of Health Impact Assessments (HIAs) in Case Study Sample (N = 23), With Illustrative Quotes and Examples, Community Health and Evaluation HIA Evaluation Study, 2011–2014

Abbreviations: DOT, Department of Transportation.

Table 3. Reported Success Factors and Challenges to Conducting Health Impact Assessments (HIAs), With Illustrative Quotes and Examples, Community Health and Evaluation HIA Evaluation Study, 2011–2014

Top