A Cohort Review Approach Evaluating Community Health Worker Programs in New York City, 2015–2017

Alexis Feinberg, MPH¹; Lois Seidl, MPH²; Rachel Dannefer, MPH, MIA²; Katarzyna Wyka, PhD³; Elizabeth Drackett, MPA²; La’Shawn Brown-Dudley, MS²; Nadia Islam, PhD¹; Lorna E. Thorpe, PhD¹ (View author affiliations)

Suggested citation for this article: Feinberg A, Seidl L, Dannefer R, Wyka K, Drackett E, Brown-Dudley L, et al. A Cohort Review Approach Evaluating Community Health Worker Programs in New York City, 2015–2017. Prev Chronic Dis 2019;16:180623. DOI: http://dx.doi.org/10.5888/pcd16.180623.

PEER REVIEWED

Abstract

The objective of this study was to describe how a cohort review approach was applied as an evaluation framework for a community health worker intervention among adult residents in 5 public housing developments in New York City in 2015–2017. The cohort review approach involved systematically monitoring participants engaged in the Harlem Health Advocacy Partners program during a given time period (“cohort”) to assess individual outcomes and program performance. We monitored participation status (completed, still active, disengaged, on leave, or died) and health outcomes. In this example of a cohort review, levels of enrollment and program disengagement were higher in cohort 1 than in cohort 2. For 6-month health outcomes, the percentage of participants with hypertension who had controlled blood pressure was static in cohort 1 and improved significantly in cohort 2. The percentage of participants with diabetes who self-reported controlled hemoglobin A_1c increased significantly in cohort 1 at 6-month follow-up. The cohort approach highlighted important outcome successes and identified workload challenges affecting recruitment and retention.

Top

Introduction

Although evidence for the effectiveness of community health workers (CHWs) is mounting, reviews of interventions related to chronic disease prevention and management demonstrate inconsistent results (1–3). One key issue is that many CHW programs are funded through grants or operating budgets that are often unpredictable, unstable, and time limited (4). Such funding mechanisms pose unique challenges: with short-term funding, some health outcomes may not emerge within funded evaluation time frames, and positive benefits of programs, including the adoption and maintenance of behavior change, may not have the opportunity to accrue or be sustained. Another problem is inconsistency in how results are reported.

These challenges have affected other public health interventions focused on sustained patient interactions, and monitoring tools developed in response to these challenges can be adapted for CHW program evaluation. For example, the introduction of an annual review process known as a “cohort review” was an important innovation in the monitoring and evaluation of tuberculosis control efforts (5); it involved systematic monitoring of groups of patients beginning treatment within a given period (“cohort”). Structured indicators allowed local and national comparisons, as well as measurement against previous cohorts, to assess improvements in program recruitment, retention, and outcomes.

We adapted cohort review methods to the evaluation of a CHW program. By standardizing participant status definitions and tracking outcome milestones, CHWs and evaluators can develop an analytic framework to better monitor participation status, participant characteristics, and health outcomes. The cohort process also allows for the assessment of trends of program performance indicators that are actionable for decision makers, particularly when comparison groups are unavailable or are no longer supported by funding sources.

Top

Purpose and Objective

The objective of this study was to describe how the cohort review approach was applied as an evaluation framework for a community-focused CHW intervention, the Harlem Health Advocacy Partners (HHAP) program, in New York City. HHAP is an ongoing municipal project that aims to improve the health of adults residing in 5 public housing developments in East/Central Harlem. Despite rich histories of community organizing, East/Central Harlem has been subject to policies and processes such as redlining, broken windows policing, and “benign neglect” that have contributed to high levels of poverty and poor health outcomes. HHAP was launched to address health and social conditions in the neighborhood, with the aim of closing racial/ethnic gaps in health and social outcomes between public housing residents in East/Central Harlem and other New Yorkers (6,7). We developed and applied the cohort review approach to the health coaching component of HHAP to assess program implementation and outcomes during the first 2 years of the program.

Top

Intervention Approach

During the first year of HHAP, 224 participants were enrolled from February through August 2015 (cohort 1), and subsequent cohorts followed an annual enrollment cycle. Cohort 2 enrolled 348 participants from September 2015 through August 2016. Concurrent to cohort 1 enrollment, we recruited a 1-year comparison sample of 176 residents from 5 nearby developments, selected on the basis of frequency-matched sociodemographic characteristics and proximity to the intervention developments (8). After cohort 1, comparison groups were not available.

In addition to a residence requirement, eligibility criteria for health coaching and the comparison group included being aged ≥18 and having at least one of 3 self-reported chronic conditions (asthma, diabetes, or hypertension). Participants who reported ever having received a physician diagnosis of asthma, hypertension, or diabetes were defined as adults with these conditions, on the basis of the following question: “Have you ever been told by a doctor, nurse, or other health professional that you have . . . ?” Both cohort 1 and cohort 2 participants were recruited primarily via community outreach conducted by CHWs, who canvassed the grounds of the selected public housing developments and collaborated with community and senior centers in each development to promote the program. The comparison group was recruited from a random sample telephone survey (9). CHWs attempted to deliver core intervention components within 6 to 12 months of enrollment.

The HHAP intervention includes 4 components: 1) health coaching, 2) navigation of the health care system, 3) wellness activities, including peer support and walking groups, and 4) advocacy to build leadership among residents to address community health needs and improve systems and conditions that influence neighborhood health. The health coaching provided by CHWs also included referrals, emergency interventions during acute-risk situations (eg, morbidly high blood pressure readings, mental health crises), and the setting of one or more SMART (specific, measurable, achievable, results-focused, and time-bound) goals. Additional health care navigation support was available through referrals to a partner organization that assists residents in obtaining medical services and ensures they receive the care to which they are entitled. A full description of the HHAP model is available elsewhere (8).

Top

Evaluation Methods

For cohort 1, CHWs conducted intake assessments as part of participant enrollment (baseline), and an academic research team from the NYU–CUNY Prevention Research Center (PRC) conducted follow-up assessments. The academic research team conducted all comparison group assessments. Cohort 1 and comparison participants received a $20 cash incentive for completing surveys. For cohort 2, CHWs conducted baseline and follow-up assessments. Surveys were conducted at 3 months, 6 months, 9 months, or 12 months after enrollment. Among participants enrolled in cohort 1 and cohort 2, 209 of 224 (93.3%) in cohort 1 and 233 of 348 (67.0%) in cohort 2 completed any follow-up assessment survey. For this analysis, we tabulated data on 6-month follow-up from both years; the response rate was 85.7% (192 of 224) for cohort 1, 92.6% (163 of 176) for the comparison group, and 41.7% (145 of 348) for cohort 2.

We categorized all HHAP participants into mutually exclusive and exhaustive categories of participation in health coaching: completed, enrolled active, disengaged, on leave, or died (Box). CHWs assigned and updated participant status. The NYU-CUNY PRC collected data on health outcomes in the baseline surveys and follow-up surveys. These outcomes were blood pressure control, blood pressure control among participants with hypertension, and self-reported hemoglobin A_1c (HbA_1c) control among participants with diabetes. Blood pressure was the average of 3 measurements taken at each survey point, and we defined control as systolic blood pressure under 140 mm Hg or diastolic blood pressure under 90 mm Hg (10). We dichotomized self-reported status of glycemic control as controlled if a health professional told a participant their diabetes was within goal and as “uncontrolled or don’t know” if they were told it was not within goal or if they were unaware of their status.

Using SAS version 9.4 for all analyses (SAS Institute Inc), we compared the baseline characteristics of cohort 1 with the baseline characteristics of cohort 2 and the comparison group with t test for continuous variables and χ² test for categorical variables. For health outcome variables, we tested significance by cohort between enrollment and 6-month post-enrollment by using the McNemar χ² test. We chose this test because it is widely used and easy to interpret.

Top

Results

A greater percentage of residents participating in HHAP health coaching than in the comparison group were aged 65 or older and self-reported hypertension (Table 1). Most participants were female and either Hispanic or non-Hispanic black, reflecting the population of the public housing developments (9). Participants were demographically similar to one another across cohorts, except that a greater proportion of cohort 2 participants than cohort 1 or comparison group participants were Hispanic.

Enrollment increased 55.4% from cohort 1 to cohort 2, from 224 to 348 participants. Of the 224 cohort 1 participants, 216 (96.4%) participants were still active in the program after 6 months, 5 (2.2%) had disengaged, 1 (0.4%) was on leave, and 2 (0.9%) had died. Of the 348 participants enrolled in cohort 2, 303 (87.1%) were still active after 6 months, 39 (11.2%) had disengaged, 2 (0.6%) were on leave, and 2 (0.6%) had died.

The percentage of participants with self-reported hypertension in cohort 1 and controlled blood pressure did not change from baseline to 6-month follow-up (58.8% to 60.1%, P = .79) (Table 2). Blood pressure control among residents with hypertension in the comparison group may have worsened from baseline to 6-month follow-up (61.0% to 53.3%, P = .16). In cohort 2, the percentage of participants with diagnosed hypertension and controlled blood pressure increased significantly, from 57.7% to 73.9% (P = .002). The percentage of participants with self-reported diabetes who reported their HbA_1c as controlled increased significantly in cohort 1 (50.0% to 64.3%, P = .02), whereas self-reported HbA_1c control did not improve among comparison group participants (65.7% to 64.2%, P = .74). Although the change was not significant, we found improvements in HbA_1c control among cohort 2 participants (72.3% to 83.0%, P = .20).

Top

Implications for Public Health

Our findings from the first 2 years of HHAP’s health coaching component demonstrate the utility of the cohort review approach in providing a structure for evaluating a multiyear program, particularly when an ongoing comparison group is not available. The approach highlighted successes in health outcomes among participants retained in the program and challenges in program retention.

The assessment showed that more participants in cohort 2 than in cohort 1 disengaged from the program after 6 months. One reason for the higher level of disengagement in cohort 2 could be the challenge of maintaining health-coaching participants carried over from cohort 1 while recruiting for cohort 2, since CHWs in cohort 2 were also responsible for managing participants from the previous year. In addition, in the beginning of cohort 2, programmatic operations were transferred from an external organization to the New York City Department of Health and Mental Hygiene, which may have resulted in a disruption for some participants. Finally, the incentive offered in cohort 1 may have positively influenced program retention and the number of follow-up interviews. Because the cohort process cycle emphasizes continuous monitoring and improvement, the HHAP program addressed retention and workload issues in cohort 3.

Our cohort assessment quantified improvements in key health outcomes shown in previous studies, namely in blood pressure (11) and glycemic control (2,12). The increase from cohort 1 to cohort 2 in the number of participants with controlled blood pressure suggests that the ability of CHWs to enhance care increases over time. This care includes efforts to keep participants connected with their primary care physician and to motivate participants to take all routine tests and medications for their conditions.

In planning for evaluating CHW programs using a cohort approach, metrics for the implementation process should be developed a priori and aligned with program objectives. Our analysis underscored the challenge of defining the participation status of a participant as complete. The definition was challenging because the criteria for program completion changed over 2 cohort years; awareness of this challenge helped formalize the definition of completion. Moreover, the program further disaggregated the disengaged group into 4 new categories: withdrew, lost to follow-up, transferred out of health coaching, and unavailable (ie, unable to fit health coaching into their schedule). It will be important to monitor these categories to assess whether participants are not interested or able to participate in the program, which would reflect a poor fit between the program and a participant’s needs (withdrew), or the program is unable to maintain contact with participants because of other factors (lost to follow-up).

We found the cohort review approach adaptable to new program goals. For example, to better HHAP’s efforts to address the social determinants of health in addition to disease management, we developed outcome metrics for social determinants of health for cohort 3, and the program will continue to monitor other variables that may contribute to health outcomes.

Our study has several limitations. Our findings in part reflect differences in HHAP programmatic operations between cohort 1 and 2. Cohort 1 data were collected by both CHWs and an academic research team and participants in cohort 1 received a cash incentive for completing surveys, whereas cohort 2 data were collected by CHWs only, often with fewer follow-up assessments, and cohort 2 participants did not receive an incentive. Differences in data collection may have biased comparisons between cohort 1 and cohort 2. Some health outcome data were self-reported; however, any bias introduced by self-report is unlikely to be differential across cohorts, except if selection bias was introduced because of higher loss to follow-up in cohort 2. Finally, given the large number of disengaged participants in cohort 2, we are not fully confident that improved outcomes were solely a function of programmatic improvements. Improved outcomes may reflect differential disengagement of participants who would have been less likely to improve.

Although previous CHW evaluations focused on individual-level outcomes, we found the cohort monitoring approach to be an effective method for evaluating the implementation process of CHW programs while also addressing issues associated with resource constraints and limited program duration (13). Adapting a cohort approach can begin to fill this gap (4,14).

Top

Acknowledgments

Research was supported by Harlem Health Advocacy Partners, a project funded and administered by the New York City Department of Health and Mental Hygiene. The efforts of Drs. Thorpe and Islam are supported in part by the Centers for Disease Control and Prevention (CDC) Grant U48DP001904. The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of the New York City Department of Health and Mental Hygiene or CDC. The authors declare no conflicts of interest. No copyrighted materials were used.

Top

Author Information

Corresponding Author: Alexis Feinberg, MPH, Department of Population Health, New York University School of Medicine, 180 Madison Ave, New York, NY 10016. Email: Alexis.Feinberg@nyulangone.org.

Author Affiliations: ¹Department of Population Health, New York University School of Medicine, New York, New York. ²New York City Department of Health and Mental Hygiene, Long Island City, New York. ³City University of New York Graduate School of Public Health and Health Policy, New York, New York.

Top

References

1. Kim K, Choi JS, Choi E, Nieman CL, Joo JH, Lin FR, et al. Effects of community-based health worker interventions to improve chronic disease management and care among vulnerable populations: a systematic review. Am J Public Health 2016;106(4):e3–28. CrossRef PubMed
2. Palmas W, March D, Darakjy S, Findley SE, Teresi J, Carrasquillo O, et al. Community health worker interventions to improve glycemic control in people with diabetes: a systematic review and meta-analysis. J Gen Intern Med 2015;30(7):1004–12. CrossRef PubMed
3. Shah M, Kaselitz E, Heisler M. The role of community health workers in diabetes: update on current literature. Curr Diab Rep 2013;13(2):163–71. CrossRef PubMed
4. Albritton E. How states can fund community health workers through Medicaid to improve people’s health, decrease costs, and reduce disparities. 2016.2016 https://familiesusa.org/product/how-states-can-fund-community-health-workers-through-medicaid. Accessed May 13, 2019.
5. Centers for Disease Control and Prevention. Understanding the TB cohort review process: instruction guide. Atlanta (GA): Centers for Disease Control and Prevention, Division of Tuberculosis Elimination; 2006.
6. Fullilove MT, Wallace R. Serial forced displacement in American cities, 1916–2010. J Urban Health 2011;88(3):381–9. CrossRef PubMed
7. Bailey ZD, Krieger N, Agénor M, Graves J, Linos N, Bassett MT. Structural racism and health inequities in the USA: evidence and interventions. Lancet 2017;389(10077):1453–63. CrossRef PubMed
8. Lopez PM, Islam N, Feinberg A, Myers C, Seidl L, Drackett E, et al. A place-based community health worker program: feasibility and early outcomes, New York City, 2015. Am J Prev Med 2017;52(3S3):S284–9. CrossRef PubMed
9. Feinberg A, Seidl L, Levanon Seligson A, Pinzon J, Mata A, Gray L, et al. Launching a neighborhood-based community health worker initiative: Harlem Health Advocacy Partner (HHAP) Community Needs Assessment. New York (NY): NYU-CUNY Prevention Research Center, New York City Department of Health and Mental Hygiene, New York City Housing Authority, and Community Service Society; 2015. https://med.nyu.edu/prevention-research/sites/default/files/prevention-research2/FINAL%20MAY%202016%20%20-%20HHAP%20Neighborhood%20Based%20Comm%20Report_0.pdf. Accessed June 3, 2019.
10. US Department of Health and Human Services Health Resources and Services Administration. Hypertension control. https://www.hrsa.gov/sites/default/files/quality/toolbox/508pdfs/hypertensioncontrol.pdf. Accessed April 4, 2018.
11. Levine DM, Bone LR, Hill MN, Stallings R, Gelber AC, Barker A, et al. The effectiveness of a community/academic health center partnership in decreasing the level of blood pressure in an urban African-American population. Ethn Dis 2003;13(3):354–61. PubMed
12. Pérez-Escamilla R, Damio G, Chhabra J, Fernandez ML, Segura-Pérez S, Vega-López S, et al. Impact of a community health workers-led structured program on blood glucose control among Latinos with type 2 diabetes: the DIALBEST trial. Diabetes Care 2015;38(2):197–205. CrossRef PubMed
13. Katigbak C, Van Devanter N, Islam N, Trinh-Shevrin C. Partners in health: a conceptual framework for the role of community health workers in facilitating patients’ adoption of healthy behaviors. Am J Public Health 2015;105(5):872–80. CrossRef PubMed
14. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4(1):50. CrossRef PubMed

Top

Tables

^a The Harlem Health Advocacy Partners program is an ongoing municipal project that aims to improve the health of adults residing in 5 public housing developments in East/Central Harlem, New York City (8).
^b Recruited from February through August 2015.
^c Concurrent to cohort 1 enrollment, a 1-year comparison sample of 176 residents was recruited from 5 nearby developments, selected on the basis of frequency-matched sociodemographic characteristics and proximity to the intervention developments.
^d Recruited from September 2015 through August 2016.
^e P value determined by t test for continuous variables and χ² test for categorical variables and compares cohort 1 characteristics with characteristics of cohort 2 and comparison group.
^f Eligibility criteria for health coaching and the comparison group included being aged ≥18 and having at least 1 of 3 self-reported chronic conditions (asthma, diabetes, or hypertension). Participants who reported ever having received a physician diagnosis of asthma, hypertension, or diabetes were defined as adults with these conditions, on the basis of the following question: “Have you ever been told by a doctor, nurse, or other health professional that you have . . . ?”
^g Smoking was dichotomized into an indicator for smoking every day or some days by using the following question: “Do you now smoke cigarettes every day, some days, or not at all?”

Abbreviations: —, does not apply; HbA_1c, hemoglobin A_1c.
^a The Harlem Health Advocacy Partners program is an ongoing municipal project that aims to improve the health of adults residing in 5 public housing developments in East/Central Harlem, New York City (8).
^b Recruited from February through August 2015.
^c Concurrent to cohort 1 enrollment, a 1-year comparison sample of 176 residents was recruited from 5 nearby developments, selected on the basis of frequency-matched sociodemographic characteristics and proximity to the intervention developments.
^d Recruited from September 2015 through August 2016.
^e Difference between values at intake and 6-month follow-up examined by using McNemar χ² test.
^f Blood pressure measurements were not obtained for every participant because of equipment malfunction, technical errors, or participant refusal.
^g Defined as systolic blood pressure <140 mm Hg or diastolic blood pressure <90 mm Hg.
^h Participants who reported ever having received a physician diagnosis of hypertension or diabetes were defined as adults with these conditions, on the basis of the following question: “Have you ever been told by a doctor, nurse or other health professional that you have . . . ?”

Top