Weeks (2017) (1) highlights an important issue: higher baseline costs in an intervention group may bias the results of difference-in-difference analyses if the parallel trends assumption does not hold. He argues that when cost (or utilization) is an important study outcome, researchers should use pre-intervention cost (or utilization) as a matching variable in the propensity score matching algorithm.
Unfortunately, VA cost and utilization data were not made available to us until after the matched sample was created. Thus, matching on pre-intervention costs was not possible. We examined several alternative models to assess the impact of possible imbalance in pre-intervention costs between treatment and control groups on our results. First, we included pre-intervention costs as an additional covariate in the outcomes models (Model 1). Second, we used our current sample and further matched the intervention and comparison groups on prior year VA costs. With the further matched sample, we first estimated the same difference-in-difference as in our paper (Model 2). Because this difference-in-difference model cannot accommodate propensity matched weights, we also estimated a generalized linear model (GLM) on post-intervention costs, taking into account propensity matched weights (Model 3). All estimation models controlled for covariates included in our original model (age, race/ethnicity, marital status, Elixhauser Index, urban residence, year of enrollment), with baseline cost as an additional control. Results were similar to those originally reported and suggest reductions in both cost measures but estimates were not consistently statistically significant (Table 1). This could be due to the smaller sample size from further matching on baseline costs.
Table 1.
Model Source of Claims |
Estimation Method |
Coefficients a (SE) |
Observations b |
Number of Individuals |
P values |
---|---|---|---|---|---|
Original Model c | Difference in Differences | ||||
VA | −0.336 (0.173) | 922 | 482 | 0.050 | |
VA + Medicare | −0.233 (0.149) | 957 | 488 | 0.119 | |
Model 1 d | Difference in Differences | ||||
VA | −0.358 (0.173) | 922 | 482 | 0.038 | |
VA + Medicare | −0.249 (0.149) | 957 | 488 | 0.095 | |
Model 2 e | Difference in Differences | ||||
VA | −0.298 (0.196) | 603 | 312 | 0.129 | |
VA + Medicare | −0.242 (0.169) | 622 | 315 | 0.154 | |
Model 3 f | GLM, gamma family, log link | ||||
VA | −0.347 (0.228) | 316 | 316 | 0.129 | |
VA + Medicare | −0.311 (0.193) | 316 | 316 | 0.108 |
Because the dependent variable is log transformed, coefficients estimates are proportional changes in cost from a one-unit change in the independent variables.
Observations with $0 in claims are dropped in log transformation.
(Original model) Samples matched on diagnosis, facility, HBPC services, age, race/ethnicity, marital status, urban/rural, service connected disability, Elixhauser Index. Model controlled for age, race/ethnicity, marital status, Elixhauser Index, urban residence, and year of enrollment.
(Model 1) Same sample as in original model. Outcome model additionally included pre-intervention cost as a covariate.
(Model 2) Samples additionally matched on pre-intervention cost. Outcome model additionally included pre-intervention cost as a covariate.
(Model 3) Same sample as in model 2. Outcome model additionally included pre-intervention cost as a covariate. Propensity matched weights accounted for.
It is important to note that the conclusions of our original paper (2) focused heavily on the absence of significant cost increases that might hinder adoption of REACH. Specifically, in our Discussion we note: “…both REACH II and REACH VA have been shown to provide benefit for dementia caregivers at a cost of less than $5/day; however, concerns about additional healthcare costs may have hindered REACH’s widespread adoption.” Both the analysis of REACH VA as a retrospective cohort study, along with the arguably stronger analysis of REACH II, as a randomized control trial, provided no evidence that there was an increase in VA or Medicare expenditures for either REACH intervention. After reaching this conclusion, we noted that for VA patients, REACH was associated with significantly lower healthcare costs, and this may have been related to the addition of a structured format for addressing the caregiver’s role in managing complex ADRD care. In light of the aforementioned concerns about differences in baseline cost in REACH VA, we believe that our speculative language was reasonable and appropriate.
Concerning Weeks’ comment about multiple comparisons, we transparently noted in our Discussion section that we did not adjust for multiple comparisons. We chose this approach because we did not believe adjustment was warranted for the primary analyses or would have affected between group comparisons.
Finally, Weeks’ conclusion that RCTs provide ‘gold standard’ evidence is, in general, true. However, understanding how REACH performs in the real world, outside the rarified atmosphere of clinical trials, is critically important. Only through implementation research and observational study can we explore the possibility that REACH may have created synergies between the coordination of guideline-driven care and the integration of a health system to better meet the needs of chronically ill patients and support their families.
Acknowledgments
Funding/Support: This study was supported by the National Institute on Aging (AG-046224-03), with additional support from the University of Tennessee Health Sciences University and the Memphis Veterans Affairs Medical Center.
Footnotes
Author Contributions: Dr. Nichols had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Nichols, Martindale-Adams, Waters, Zhu
Acquisition of data: Kaplan, Lum, Zhu, Zuber
Analysis of data: Kaplan, Lum, Zhu, Zuber
Interpretation of data: All authors
Drafting of the manuscript: Nichols, Kaplan, Waters, Zhu
Obtained funding: Nichols
Study supervision: Nichols, Martindale, Waters, Zhu
Critical revision of the manuscript for important intellectual content: All authors.
Conflict of Interest Disclosures: None reported.
Role of the Funder/Sponsor: The NIA, UTHSC and VAMC had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.
Publisher's Disclaimer: Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the views of NIH, the University of Tennessee Health Sciences University, the VA, or the United States government.
Conflict of Interest Disclosures: Below is the checklist authors completed.
Elements of Financial/Personal Conflicts |
*Linda Nichols |
Jennifer Martindale- Adams |
Carolyn Zhu |
Erin Kaplan | Jeff Zuber | Jessica Lum | Teresa Waters |
|||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Yes | No | Yes | No | Yes | No | Yes | No | Yes | No | Yes | No | Yes | No | |
Employment or Affiliation | X | X | X | X | X | X | X | |||||||
Grants/Funds | X | X | X | X | X | X | X | |||||||
Honoraria | X | X | X | X | X | X | X | |||||||
Speaker Forum | X | X | X | X | X | X | X | |||||||
Consultant | X | X | X | X | X | X | X | |||||||
Stocks | X | X | X | X | X | X | X | |||||||
Royalties | X | X | X | X | X | X | X | |||||||
Expert Testimony | X | X | X | X | X | X | X | |||||||
Board Member | X | X | X | X | X | X | X | |||||||
Patents | X | X | X | X | X | X | X | |||||||
Personal Relationship | X | X | X | X | X | X | X |
Contributor Information
Linda O. Nichols, Co-Director, Caregiver Center, Veterans Affairs Medical Center Memphis, Professor, Departments of Preventive Medicine and Internal Medicine, University of Tennessee Health Science Center, VAMC (11-H), 1030 Jefferson Avenue, Memphis, TN 38104; (901) 523-8990 *5082, (901) 577-7439 (fax)
Jennifer Martindale-Adams, Associate Professor, Department of Preventive Medicine, University of Tennessee Health Science Center, Co-Director, Caregiver Center, Veterans Affairs Medical Center Memphis, VAMC (11-H); 1030 Jefferson Avenue, Memphis, TN 38104; (901) 523-8990 *5080
Carolyn W. Zhu, Associate Professor, Department of Geriatrics and Palliative Medicine, Icahn School of Medicine at Mount Sinai, Geriatric Research, Education, and Clinical Center, James J. Peters VA Medical Center
Erin K. Kaplan, Assistant Professor, Department of Economics, Rhodes College
Jeffrey K. Zuber, Data Analyst, Department of Preventive Medicine, University of Tennessee Health Science Center, Veterans Affairs Medical Center Memphis
Jessica Lum, Statistician, Geriatric Research, Education, and Clinical Center, James J. Peters VA Medical Center.
Teresa M. Waters, Professor and Chair, Department of Preventive Medicine, University of Tennessee Health Science Center.
References
- 1.Weeks W. Over-REACHing conclusions. J Am Geriatr Soc. 2017 doi: 10.1111/jgs.14949. [DOI] [PubMed] [Google Scholar]
- 2.Nichols LO, Martindale-Adams J, Zhu CW, Kaplan EK, Zuber J, Waters TM. Impact of the REACH II and REACH VA dementia caregiver interventions on healthcare costs. J Am Geriatr Soc. 2017 doi: 10.1111/jgs.14716. [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]