|
|
Volume
7: No. 3, May 2010
ORIGINAL RESEARCH
Evaluating an Insurance-Sponsored Weight Management Program
With the RE-AIM Model, West Virginia, 2004-2008
Christiaan G. Abildso, PhD, MPH; Sam J. Zizzi, EdD; Bill Reger-Nash, EdD
Suggested citation for this article: Abildso CG, Zizzi SJ, Reger-Nash B. Evaluating an insurance-sponsored weight management program
with the RE-AIM model, West Virginia, 2004-2008. Prev Chronic Dis 2010;7(3):A46. http://www.cdc.gov/pcd/issues/2010/
may/09_0114.htm. Accessed [date].
PEER REVIEWED
Abstract
Introduction
Evaluations of weight management programs in real-world settings are lacking. The RE-AIM model (reach,
effectiveness, adoption, implementation, maintenance) was developed to address this deficiency. Our primary objective was to evaluate a 12-week insurance-sponsored weight management intervention
by using the RE-AIM model, including short-term and long-term individual outcomes and setting-level implementation factors.
Our secondary objective was to critique the RE-AIM model and its revised
calculation methods.
Methods
We created operational definitions for components of the 5 RE-AIM indices and used standardized effect size values from various statistical procedures to measure multiple components or outcomes within each index. We used χ2
analysis to compare categorical variables and repeated-measures analysis of
variance to assess the magnitude of outcome changes over time.
Results
On the basis of data for 1,952 participants and surveys completed by administrators at 23 sites,
RE-AIM indices ranging from 0 to 100 revealed low program reach and adoption (5.4 and 8.8, respectively), moderate effectiveness (43.8), high implementation (91.4), low to moderate individual maintenance (21.2), and moderate to high site maintenance (77.8). Median (interquartile range) weight loss was 13 lb (6.5-21.4 lb) among participants who completed
phase I (12 weeks; 76.5%) and 15 lb (6.1-30.3 lb) among those who completed phase II (1 year; 45.7%).
Conclusions
This program had a significant, positive effect on participants and has been
sustainable but needs to be expanded for more public health benefit. The RE-AIM model provided a useful framework to determine program strengths and weaknesses and to present them to
the insurance agency and public health decision makers.
Back to top
Introduction
Regardless of the success of diet and physical activity programs in controlled research settings (1-6), if large-scale programs are not effective or easily implemented in real-world settings they fall short of their intended purpose. In the Strategic Plan for NIH Obesity Research, the National Institutes of Health Obesity Research Task Force (7) outlined a national strategy for obesity research emphasizing the use of translational research. However, evaluation of the translation of
effective programs to real-world settings is lacking (8).
The RE-AIM model (9) was developed as a tool to evaluate health promotion programs along 5 dimensions:
reach, effectiveness, adoption, implementation, and maintenance. Reach refers to the percentage of potential participants who enroll and their representativeness of the target population; effectiveness, to the changes in participant outcomes during the program; adoption, to the number and representativeness of sites willing to conduct
the program; implementation, to site adherence to program
protocols; and maintenance, to individual outcomes after final intervention contact and program continuation at sites over the long term (8-10).
The RE-AIM model has been used to evaluate a limited number of programs, including 14 WISEWOMAN cardiovascular screening interventions in North Carolina (11), a church-based physical activity intervention in South Carolina (12), and 2 diabetes self-management interventions (13). To our knowledge, no evaluation of a large-scale weight management program using RE-AIM methods has been published, but studies have shown the model is applicable to a variety of health behaviors and programs.
Additionally, researchers have recommended revising RE-AIM index calculation methods to reduce the likelihood of difficult-to-interpret negative values (13).
Our primary objective was to use RE-AIM to evaluate a multisite, insurance-sponsored weight management program that produced positive short-term physical and psychosocial changes in participants in a university laboratory setting. Our secondary objective was to critique the RE-AIM model’s usefulness as an
evaluation tool for health promotion programs. We analyzed 3- and 12-month participant outcomes and site-level program implementation information.
Back to top
Methods
This study evaluated a weight management program that has been offered since
2004 by a large public employees’ insurance agency in West Virginia as a benefit to its members. Obese insured members (body mass index [BMI] ≥30 kg/m2) are eligible for the program, as are overweight members (BMI 25.0-29.9 kg/m2) who also report being treated for hypertension, diabetes, heart disease, metabolic syndrome, or sleep apnea. Participants enroll in the
program by calling the insurance agency to be screened for height, weight, and health information and must have written approval from their primary care physician.
The insurance agency also screens site providers to ensure the site has adequate safety devices, exercise equipment, and staffing to accommodate high-risk participants. At the time of this study, the insurance agency had approved 31 fitness facilities (“sites”) to accept participants: 2 housed in university cardiac rehabilitation/exercise physiology laboratories, 2 in physical therapy facilities, 5 in hospitals, and 22 private wellness/fitness centers. Searches of available print (14) and
online (www.wvha.com, www.yellowpages.com) resources yielded 352 of these types of facilities in West Virginia. Each approved site is required to designate an administrator to act as liaison with the insurance agency, handle participant contact, coordinate site services, and enter participant data into a database.
The weight management program evaluated in this study is a 1-year benefit, during which weight loss is promoted
by using a combination of behavior therapy (eg, food logs), individualized site-based exercise programming, and individualized diet therapy. Case management nurses track participant progress
on the basis of monthly entry of the following body measurements by the sites into a Web-based database: weight and BMI, systolic and diastolic blood pressure, body fat percentage, waist circumference,
and number of site visits. Each site determines its measurement protocols on the
basis of available instruments and staff training. Sites are compensated via monthly member copayments and insurance agency payments for the provision of on-site services in 2 phases.
Phase I (months 1-3)
Services provided during phase I include 1) access to the exercise site, 2) a 60-minute fitness evaluation and exercise program designed by an exercise physiologist, 3) a 60-minute evaluation and nutrition program designed by a registered dietitian, 4) monthly 30-minute exercise sessions with a personal trainer, and 5) 60-minute follow-up reassessments by the exercise physiologist and registered dietitian at the end of month 3. Participants may be removed from the program if they fail to
exercise at the site at least twice per week; turn in food logs periodically; attend appointments with the exercise physiologist, registered dietitian, or personal trainer; or have body measurements taken by site staff monthly. Phase I is considered the “intervention” period for this evaluation. If participants meet the behavioral requirements and lose 12 lb (or show substantial improvements in other physical measures) they may continue to
phase II.
Phase II (months 4-12)
Phase II spans the remainder of the first year (9 months). This phase is considered the postintervention period for the current evaluation because it is more like a private gym membership than a weight-loss intervention; it provides access to the facility and 1 personal training session per month. Participants are expected to continue exercising at the facility twice per week and have physical measurements and visit data submitted monthly by the sites to maintain the benefit.
The insurance agency dictates what services are offered, but not how the services are implemented. The use of relevant weight management guidelines of the American College of Sports Medicine (ACSM) (15) and the American Dietetic Association (16) is suggested. For example, the exercise physiologist may choose from fitness testing methods recommended by ACSM (eg, 6-minute walk test, 1-mile walk test) and the registered dietitian may use his or her discretion in diet composition or outcome
goals.
Data sources
We used archival participant data and site administrator surveys to evaluate the weight management program. The insurance agency granted us access to an online password-protected database containing demographic and physical measurement data of all program participants. The university’s institutional review board approved the study for the protection of human
participants. We downloaded, verified, de-identified, and imported archival data of all participants into SPSS version 14.0 (SPSS,
Inc, Chicago, Illinois).
To gather site-level program evaluation data, we recruited site administrators by mail, e-mail, and telephone (17) to complete an online survey. The
survey assessed 1) site staffing, equipment, and facilities; 2) program implementation methods; and 3) barriers to program implementation. Section 2 of the survey was relevant to this study;
it included items that ascertained the procedures (open-ended) or frequency (categorical) of conducting
14 program components during phase I as recommended or required by the insurance agency: measuring weight, waist circumference, body fat percentage, and blood pressure; providing individualized nutrition and fitness assessments, nutrition classes, fitness plan, one-on-one personal training, and follow-up nutrition and fitness assessments; tracking participant attendance; and reviewing home exercise and food logs
or journals. Responses were recoded to dichotomous variables (yes/no) and the sum of
yes responses tallied to represent the number of program components implemented.
Statistical analyses of RE-AIM indices
Box. RE-AIM Components and Operational Definitions |
RE-AIM Component |
Definition |
Individual participation rate (IPR) |
The percentage of eligible members of the insurance agency who participated in the weight management program |
Demographic representativeness (DR) |
How well participants represented the population eligible for the program |
Individual completion rate (ICR) |
The percentage of participants who completed each of the phases of the weight management program |
Outcomes (O) |
Weight change experienced by completers of each phase of the weight management program |
Differential impact (DI) |
A measure of differential changes in weight experienced among phase I and II completers by sex and age |
Setting participation rate (SPR) |
The percentage of eligible cardiac rehabilitation centers, physical therapy clinics, wellness/fitness centers, and health clubs in the state that participated in the weight management program |
Component implementation rate (CIR) |
The percentage of program protocols implemented at each site |
Setting continuation rate (SCR) |
The percentage of program sites that have elected to continue conducting the program for more than 1 year |
Presenting results using the RE-AIM model first requires creating operational definitions for multiple components in each of the 5 RE-AIM dimensions (10) (Box). These components were used to calculate RE-AIM indices (Appendix). Index calculations used standardized effect size (18) values from various statistical procedures to measure multiple components or outcomes in each index. We used χ2 analysis to compare categorical variables and repeated measures analysis of
variance (ANOVA) to assess the magnitude of outcome changes over time. In the indices, “positive” effects were reduced by differential or “negative” effects (eg,
program attrition). This index calculation method has resulted in negative
values in prior studies (13) for 2 reasons. First, in calculating index values,
differential effects have been subtracted directly from outcome effects,
overstating the “negative” program effect. To limit this potential distortion,
we calculated a “proportion of positive effect not explained by differential
effects” by first subtracting the differential (“negative”) effect size
from 1 and then multiplying the difference by the “positive” effect (Appendix).
Second, a negative index value may still result if the effect size is more than 1. We resolved this challenge by using
effect size measures whose upper limit was 1.
Effect size measures (and interpretation) included the squared Cramer phi (φc2) for χ2 tests (small [φc2 ≈ .01], moderate [φc2 ≈ .09], or large [φc2 > .25]) and the squared curvilinear correlation coefficient
(partial eta squared; η2) for ANOVA (small [η2 ≈ .01],
moderate [η2 ≈ .06], or large [η2 > .14]) (18,19). All raw index scores ranging from 0 to 1 were multiplied by 100 for summary index values of 0 to 100. Missing data were excluded from analyses, which were conducted by using SPSS version 14.0.
Back to top
Results
The data for 1,952 participants from 31 sites were exported, and administrators from 27 of 31 active sites (87.1% response rate) completed online surveys. Responses from 4 new sites whose participants had not yet completed
phase I of the program were excluded from analysis, resulting in 23 completed site administrator surveys.
The distribution of most of the participants’ measurement data was skewed, and
data are presented as median (interquartile range) unless otherwise noted.
Individual-level impact ([reach * effectiveness] / 100)
A total of 60,041 adult members were covered by the insurance agency (N. Henderson, health promotions director, written communication, January 2008).
By using the state obesity (31%) and overweight (36%) prevalence rates (20), and national prevalence of overweight adults with at least 1 comorbid condition (66%) (21),
we estimated that 32,878 insurance members may have been eligible for the program. At the time of data collection, 1,952 members had participated (5.9% individual participation rate)
(Table
1).
Baseline physical measurements for men and women are presented in
Table 2. A significantly larger percentage of participants than the eligible population were women (80.0% vs 54.1%; P < .001, φc2 = .073), and
more of them were aged 45-54.9 years and fewer of them were aged 65 years or older (P < .001,
φc2 = .113), yielding
a reach value (R)
of 5.4 (Table 1).
Phase I outcome changes are presented in
Table 3. Of 1,647 participants who could have completed
phase I (those who had started the program at least 12 weeks before data collection), 76.5% had done so.
Fewer women (74.8%) than men (83.7%) completed phase I (P = .001, φc2 = .007), and completion rates tended to rise with age. Participants who completed
phase I had significant weight loss (13 lb [6.5-21.4 lb]; P < .001,
η2 = .592), and men lost significantly more weight than women (P < .001,
η2 = .050). Differences in weight loss were also revealed among age groups (P = .003,
η2= .014). These components produced
an effectiveness value (E) of 43.8 and an individual-level impact ([R * E] / 100) of 2.4 (Table 1).
Setting-level impact ([adoption *
implementation] / 100)
There were 31 active weight management program sites out of a total of 352 potential sites in West Virginia at the time of data collection, resulting in an adoption
value (A)
of 8.8 (Table 1). Site survey responses (N = 23) showed that sites implemented a mean of 12.8 (standard deviation [SD] 1.0) of 14 program components during
phase I. All sites measured weight, waist circumference, body fat percentage, and blood pressure; tracked attendance; and provided initial and follow-up nutrition assessments with
a registered dietitian, nutrition classes designed by the registered dietitian, and an individualized exercise prescription. The least frequently implemented component, by 15 sites (65%), was having participants maintain home exercise logs. Implementation survey data yielded
an implementation value (I) of 91.4 and
a setting-level impact ([A * I] / 100) of 8.0 (Table 1).
Long-term maintenance ([individual maintenance * setting maintenance] / 100)
Phase II outcome changes are presented in
Table 4. Of 762 participants who could have completed
phase II (those who had started the program at least 1 year before data collection), 348 (45.7%) had done so (ICRPhII = .457) (Table 1). Excluding the small samples of participants aged 18-24.9 years (n = 2) and 65 years
or older (n = 9), completion rates tended to increase with age and be higher in men in each age group. Participants who completed
phase II achieved significant
weight loss from baseline (15 lb [6.1-30.3 lb]; P < .001, η2 = .467), shedding 6.7% (2.7%-12.7%) of baseline body weight. Weight loss was similar among
phase II completers of different age groups (P = .61, η2 = .011) and between sexes (P = .21, η2 = .005). These values combined to yield an
individual maintenance value (MI) of 21.2 (Table 1).
Four of 18 sites (22%) that had been approved to accept participants at least 1 year
before this study had stopped or had been
disallowed to continue accepting participants, resulting in a setting
maintenance value (MS) of 77.8 and a long-term maintenance
value
([MI * MS] / 100) of 16.5 (Table 1).
Back to top
Discussion
We achieved our primary objective of using RE-AIM to evaluate a weight management program. We found moderate program effectiveness and high implementation, suggesting the program has been beneficial for participants and can be implemented in a variety of settings. We found low program reach and adoption, suggesting the program could be improved by recruiting new participants and sites. Recruitment may prove difficult, however, because participants must be highly motivated to enroll in the
program, and sites are required to offer services by highly trained personnel often unavailable in rural areas of West Virginia. In the long term, site maintenance was high, but individual maintenance was fairly low, indicating the program is sustainable but the services of
phase II may need to be revised to improve participant outcomes. The summary results suggest this weight management program has potential to be expanded for
more translation and public health benefit and should be
considered a viable model for other public and private insurers.
Individual short-term and long-term outcome changes are comparable with those of other behavioral programs and clinical trials. Short-term attrition from this weight management program (23.5%) was slightly higher than is commonly seen in behavioral programs of similar length (10%-15%), though the median weekly weight loss in this program is comparable (1.23 lb vs 1.1 lb) (22). This program also compares favorably with clinical trials of similar length, which average 85%-95% completion rates
and approximately 1 lb of weekly weight loss (1,5).
Long-term individual results also compare favorably with other behavioral programs and clinical trials. Participants who completed
phase II (n = 348) lost a mean of 20.9 (SD, 22.3) lb from baseline, with some recidivism.
Slightly more than half of phase II completers (51.5%) maintained phase I weight loss or continued losing weight in
phase II. In comparison, approximately 60% to 70% of weight loss is maintained for a year
after treatment in other short-term behavioral interventions (22).
One-year results indicate more average weight losses but lower completion rates than randomized control trials of similar length (1,5).
We also achieved the secondary objective of critiquing the RE-AIM model and revised index calculations. The strength of the RE-AIM model is its ability to quantify for decision makers a program’s strengths and weaknesses. Comparison
with other health promotion program evaluations that used RE-AIM is limited at this point in the model’s refinement because no 2 studies have used the same index calculation methods. We believe this study advances the RE-AIM model by 1) providing
methods for assessing long-term maintenance at the individual and site levels and 2) addressing 2 methodologic concerns with existing index calculation methods (ie, negative index values and effect sizes with varying maximum values). The revised methods in this study produced positive R and MI values, whereas previously used methods (10) would have yielded negative values.
Limitations
The study is limited in a number of ways. Multiple sources of measurement error may have affected the data, including 1) lack of standardized procedures and instruments for measuring health outcomes, 2) missing outcome data, and 3) social desirability of sites when entering participant data and completing survey items. Potential error was addressed in multiple ways. Trained exercise professionals took measurements using accepted professional standards of practice, the insurance agency
periodically audited site data, we contacted sites to collect missing data, and all survey recruiting material stressed the informative (not punitive) nature of the study and independence of the investigators from the insurance agency. Measurement error would be more important in a small clinical trial assessing an intervention’s efficacy than in this study with its large sample and focus on standardized effect sizes.
Conclusions
Numerous questions remain to be answered before the public health effect of this and other weight management programs can be better understood. This program had a significant
positive effect on participants and was sustainable, but needs to be expanded. The RE-AIM model provided a framework through which the translation of this program could be evaluated and presented to public health decision makers. In fact, the summary data of this project were presented to the insurance agency and used
as evidence for changing the program benefit to address the low reach, adoption, and long-term individual maintenance. We encourage continued study of RE-AIM index calculations and application of the model to evaluate obesity treatment programs. It may be beneficial to add qualitative process data to these outcomes to work toward developing a set of best practices in behavioral weight management (11,12). Additionally, because insurance agencies often provide
benefits for multiple weight management modalities (eg, behavioral, surgical, pharmacologic), RE-AIM methods should be used to evaluate multiple modalities concurrently to allow for side-by-side comparison and facilitate decision making about resource allocation. These evaluations will need to link program costs, insurance claims, individual outcomes, and future cost savings. Such analyses may affect the benefit and incentive structure of this
insurer and others to improve the translation of
knowledge regarding clinical obesity treatment into innovative programming that can be widely implemented.
Back to top
Acknowledgments
This research was funded by the State of West Virginia Public Employees Insurance Agency.
Back to top
Author Information
Corresponding Author: Christiaan G. Abildso, PhD, MPH, West Virginia University College of Physical Activity and Sport Sciences, PO Box 6116, Morgantown, WV 26506-6116. Telephone: 304-293-0860. E-mail:
cgabildso@mail.wvu.edu.
Author Affiliations: Sam J. Zizzi, Bill Reger-Nash, West Virginia University, Morgantown, West Virginia.
Back to top
References
- Andersen RE, Wadden TA, Bartlett SJ, Zemel B, Verde TJ, Franckowiak SC.
Effects of lifestyle activity versus structured aerobic exercise in obese women: a randomized trial. JAMA 1999;281(4):335-40.
- Dunn AL, Garcia ME, Marcus BH, Kampert JB, Kohl HW, Blair SN.
Six-month physical activity and fitness changes in Project Active, a randomized trial. Med Sci Sports Exerc 1998;30(7):1076-83.
- Dunn AL, Marcus BH, Kampert JB, Garcia ME, Kohl HW III, Blair SN.
Comparison of lifestyle and structured interventions to increase physical activity and cardiorespiratory fitness: a randomized trial. JAMA 1999;281(4):327-34.
- Neter JE, Stam BE, Kok FJ, Grobbee DE, Geleijnse JM.
Influence of weight reduction on blood pressure: a meta-analysis of randomized controlled trials. Hypertension 2003;42(5):878-84.
- Wadden TA, Berkowitz RI, Womble LG, Sarwer DB, Phelan S, Cato RK, et al.
Randomized trial of lifestyle modification and pharmacotherapy for obesity. N Engl J Med 2005;353(20):2111-20.
- Wadden TA, Vogt RA, Andersen RE, Bartlett SJ, Foster GD, Kuehnel RH, et al.
Exercise in the treatment of obesity: effects of four interventions on body composition, resting energy expenditure, appetite, and mood. J Consult Clin Psychol 1997;65(2):269-77.
- Strategic plan for NIH obesity research. Washington (DC): US Department of Health and Human Services, National Institutes of Health Obesity Research Task Force; 2004.
- Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks P.
The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med 2004;27(1):3-12.
- Glasgow RE, Vogt TM, Boles SM.
Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999;89(9):1322-7.
- Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM.
Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res 2006;21(5):688-94.
- Farris RP, Will JC, Khavjou O, Finkelstein EA.
Beyond effectiveness: evaluating the public health impact of the WISEWOMAN program. Am J Public Health 2007;97(4):641-7.
- Bopp M, Wilcox S, Laken M, Hooker SP, Saunders R, Parra-Medina D, et al. Using the RE-AIM framework to evaluate a physical activity intervention in churches. Prev Chronic Dis 2007;4(4).
http://www.cdc.gov/pcd/issues/2007/oct/06_0155.htm. Accessed November 17, 2007.
- Glasgow RE, Nelson CC, Strycker LA, King DK.
Using RE-AIM metrics to evaluate diabetes self-management support interventions. Am J Prev Med 2006;30(1):67-73.
- West Virginia Department of Health and Human Resources Health Statistics Center. The burden of cardiovascular disease in West Virginia; 2002. http://www.wvdhhr.org/bph/oehp/hsc/burdencvd/default.htm. Accessed November 17, 2007.
- American College of Sports Medicine. ACSM’s guidelines for exercise testing and prescription. 7th edition. Hagerstown (MD): Lippincott, Williams, and Wilkins; 2005.
- Seagle H, Strain GW, Makris AP, Reeves RS.
Position of the American Dietetic Association: weight management. J Am Diet Assoc 2009;109(2):330-46.
- Dillman DA. Mail and internet surveys. 2nd edition. New York (NY): Wiley; 2000.
- Cohen J. Statistical power analysis for the behavioral sciences. 2nd edition. Hillsdale (NJ): Lawrence Erlbaum Associates; 1988.
- Witte RS, Witte JS. Statistics. 6th edition. Orlando (FL): Harcourt, Inc; 2001.
- Behavioral Risk Factor Surveillance System: 2006 prevalence data. Centers for Disease Control and Prevention. http://apps.nccd.cdc.gov/brfss/. Accessed November 2, 2007.
- Must A, Spadano J, Coakley EH, Field AE, Colditz G, Dietz WH.
The disease burden associated with overweight and obesity. JAMA 1999;282(16):1523-9.
- Brownell KD, Wadden TA.
Etiology and treatment of obesity: understanding a serious, prevalent, and refractory disorder. J Consult Clin Psychol 1992;60(4):505-17.
Back to top
|
|