The Association of Mental Health Program Characteristics and Patient Satisfaction

Abstract

OBJECTIVES:

Satisfaction with care is an important patient-centered domain of health system quality. However, satisfaction measures are costly to collect and not directly modifiable. Therefore, we assessed the relationships between veterans’ satisfaction and measures of modifiable aspects of Veterans Health Administration (VHA) mental health care programs.

STUDY DESIGN:

For a sample of 6990 patients who received mental health care from the VHA in 2013, we used survey and administrative data to investigate the association of a suite of access and encounter satisfaction measures with a large collection of measures of program characteristics.

METHODS:

We estimated risk-adjusted correlations between 6 satisfaction measures (across 2 domains: access and encounter satisfaction) and 28 mental health care program characteristics (across 4 domains: program reach, psychosocial service access, program intensity, and treatment continuity).

RESULTS:

We found that satisfaction with access to care was higher than experiences with care encounters, but that broad measures of mental health care program reach and intensity were positively associated with both kinds of satisfaction. No measures of psychosocial service access were positively associated with access and encounter satisfaction. Most measures of treatment continuity were consistently and positively associated with both kinds of satisfaction.

CONCLUSIONS:

As the VHA strives to increase access to, and provision of, mental health care, policy makers and program managers should be aware that satisfaction with care, as it is currently measured, may not rise as more patients initiate treatment, unless continuity of care is maintained or enhanced.

Satisfaction with care is an important, patient-centered measure of health system performance because it can identify gaps in quality that could be missed by other measures,¹ help detect cross-population disparities,² and serve as a catalyst for quality improvement.^3,4 Yet, the relationship between patient satisfaction and quality of care, although generally positively correlated,⁵ is not fully understood.⁶ Studies have found patient satisfaction to be associated with hospital process quality⁷; lower rates of readmissions, heart attack mortality,^8,9 and surgical quality¹⁰; and better long-term outcomes.¹¹ Fenton et al¹² found it also positively correlated with higher healthcare utilization, costs, and all-cause mortality.

Assessment of what objective health system measures drive patient satisfaction is important for 2 reasons. First, because it relies on patient surveys, satisfaction remains expensive and challenging to measure at a high frequency.¹³ However, many other measures of health system performance are easily obtained at a high frequency from administrative data (eg, process quality measures or readmissions). If satisfaction is highly correlated with these other administrative measures, they offer supplements to satisfaction surveys—ways to monitor and improve aspects of care related to satisfaction during longer intervals between measurement. Second, satisfaction is not directly modifiable; improvements must come from changes in the processes of care or investments in services that patients value.

For these reasons, we studied the relationships between a set of patient satisfaction measures and a large collection of mental health program characteristics for patients with a recent mental health encounter in the Veterans Health Administration (VHA), the largest provider of mental health care in the United States.¹⁴ Prior work has documented variation in satisfaction across VHA patients with psychiatric diagnoses. Rosenheck et al¹⁵ found that VHA patients who were discharged from the hospital with a primary psychiatric or substance use diagnosis were more likely to be satisfied with their care if they were older, in better health, or had a long length of stay.

Burnett-Zeigler et al¹⁶ reported that VHA patients with psychiatric diagnoses who were younger, nonwhite, or lower-income; had a service-connected disability; or had received a posttraumatic stress disorder (PTSD) or a substance use disorder diagnosis were less likely to be satisfied with their care. Hepner et al¹⁷ examined perceptions of behavioral health care among VHA patients who received mental health care. Seventy-four percent said they were helped by treatment, but only 32% reported an improvement in symptoms. Holcomb et al¹⁸ found that the satisfaction of midwestern VHA patients with psychiatric diagnoses positively correlated with better self-reported outcomes. Patients with co-occurring substance use and psychotic disorders who were treated in VHA residential substance use disorder treatment programs that had more positive perceptions and satisfaction exhibited greater engagement in care and experienced better outcomes.¹⁹ Finally, Hoff et al²⁰ reported lower levels of satisfaction among VHA patients with psychiatric diagnoses than those with medical diagnoses.

METHODS

Since 2002, the VHA Office of Quality and Performance has fielded the Survey of Healthcare Experiences of Patients (SHEP), an ongoing monthly mail survey of patients’ experiences during their most recent VHA encounter. Modeled on the Consumer Assessment of Healthcare Providers and Systems survey and based on a stratified design that selects from the specialty care domains as well as new and established primary care patients within each facility,²¹ SHEP samples about 30,000 ambulatory care patients each month who visit the VHA and who were not surveyed in the prior year. The 2013 version of SHEP is our source of satisfaction measures, with an overall response rate of about 44% and slightly higher response rates for males and substantially higher response rates for older patients (eAppendix Table A1 [eAppendices available at ajmc.com]).

In 2010, the Department of Veterans Affairs (VA)’s Office of Mental Health Operations (OMHO) implemented the Mental Health Information System (MHIS) Dashboard,²² which includes facility-level quality metrics consistent with the goals of the VA’s Uniform Mental Health Services Handbook.^23,24 In addition, the mental health domain of the VHA Strategic Analytics for Improvement and Learning (SAIL) includes 25 administrative data–based performance measures related to access, continuity of care, patient safety, and quality of care at a facility level.²⁵ We used the 2013 MHIS Dashboard and precursors to SAIL mental health domain report metrics (MHIS and SAIL are refined on an ongoing basis), shared with us by OMHO, to predict patient-level satisfaction responses to the 2013 SHEP.

SHEP surveys patients with a recent VHA encounter (the “index” encounter). To merge facility-level MHIS/SAIL-based mental health program characteristics, we associated each SHEP respondent with the VHA facility where they had the index encounter. For risk adjustment, we also merged, at the patient level, demographic and Elixhauser²⁶ comorbidity data from VHA administrative files.

Our interest was in the relationships between mental health care program characteristics and patient satisfaction, so we used data from a subset of SHEP respondents—those with a recent mental health encounter. To accomplish this, we restricted the SHEP sample to respondents with index encounters in the same quarter and year as encounters for mental health. Because most SHEP respondents complete and return surveys 2 or more months after the index visit, this approach guaranteed that the majority would have had a recent mental health encounter prior to providing satisfaction feedback. Therefore, although some of the survey questions ask patients to report satisfaction based on the prior 12 months of care, it would be likely that patients’ impressions were more heavily influenced by their most recent mental health encounter. Nevertheless, unlike prior analyses of satisfaction among VHA mental health patients,^1,17 we were not directly assessing satisfaction with mental health care services. Our final sample included 6990 patients across 165 VHA facilities, although not all patients responded to all survey items due to SHEP question skip patterns (eAppendix Table A2). All analyses were conducted at the patient level.

Patient Satisfaction Variables

Satisfaction with timeliness of care, which we termed “access satisfaction,” is measured by SHEP asking respondents how often they were able to obtain needed care right away and were able to get VHA appointments as soon as they thought they needed care, excluding the times they needed urgent care. Access to VHA tests or treatments is measured by SHEP asking how easy it was to access that care in the last 12 months. Response options for the above 3 measures included “always,” “usually,” “sometimes,” or “never.” There was no cardinal meaning to these categorical responses. Therefore, we dichotomized them to eliminate fine gradations in the ordinal scale Specifically, following Prentice et al,²⁷ we dichotomized these to 1 for responses of “always” or “usually” and 0 otherwise (Table 1).

TABLE 1.

Patient-Level Demographics and Satisfaction Measures

Variable	Description	Mean	SD
Demographics
Age	In years	61.86	11.74
Married	1 = married, 0 = not married	0.55	0.50
Female	1 = female, 0 = male	0.08	0.26
Black	1 = black, 0 = nonblack	0.12	0.32
Access satisfactiona
Got needed appointment	In the last 12 months, when you needed care right away, how often did you get care as soon as you thought you needed? (1 = “always” or “usually”; 0 = “sometimes” or “never”)	76%	43%
Got wanted appointment	In the past 12 months, not counting the times you needed care right away, how often did you get an appointment as soon as you thought you needed? (1 = “always” or “usually”; 0 = “sometimes” or “never”)	76%	43%
Got needed care	In the past 12 months, how often was it easy to get the care, tests, or treatment you thought you needed through the VHA? (1 = “always” or “usually”; 0 = “sometimes” or “never”)	78%	42%
Encounter satisfactiona
VHA rating	Using any number from 0 to 10, where 0 is the worst healthcare possible and 10 is the best healthcare possible, what number would you use to rate all your VHA healthcare in the last 12 months? (dichotomized to 1 for a response of “9” or “10” and 0 otherwise)	53%	50%
Clinician rating	Using any number from 0 to 10, where 0 is the worst personal doctor/nurse possible and 10 is the best personal doctor/nurse possible, what number would you use to rate your personal VHA doctor/nurse? (dichotomized to 1 for a response of “9” or “10” and 0 otherwise)	66%	47%
Visit rating	All things considered, how satisfied were you with the VHA during your recent visit? (Responses were on a 7-point scale where 1 = completely dissatisfied and 7 = completely satisfied; dichotomized to 1 for a response of “6” or “7” and 0 otherwise.)	47%	50%

VHA indicates Veterans Health Administration.

^aNumber of responses and skip patterns provided in eAppendix Table A2.

Encounter satisfaction, which measures satisfaction with the care received or provider seen, is measured by SHEP asking respondents to rate VHA healthcare in the last 12 months on a scale of 0 to 10, where 0 indicates the “worst healthcare possible” and 10 the “best healthcare possible.” Satisfaction with the respondents’ personal doctor/nurse is also assessed on a 0-to-10 scale. For the same reasons given above, we dichotomized these to 1 for responses of 9 or 10 and 0 otherwise.²⁷ Satisfaction with the most recent VHA visit is assessed on SHEP with a scale ranging from 1 to 7, where 1 indicates “completely dissatisfied” and 7 “completely satisfied.” We dichotomized this to 1 for responses of 6 or 7 and 0 otherwise.²⁷

Program Characteristics Variables

The mental health program measures we considered are listed and defined in Table 2 and are organized into 4 areas of focus: 1) program reach (eg, the proportion of patients receiving mental health care), 2) psychosocial service access (eg, the proportion of patients initiating psychosocial treatment or psychotherapy), 3) program intensity (eg, the number of encounters per year), and 4) treatment continuity (eg, the proportion of discharged patients with follow-up within 7 days). Within each area, we examined 5 or more performance metrics. Transitional work visits and supportive employment visits mentioned in Table 2 are occupational therapy treatment modalities.^28,29

TABLE 2.

Facility-Level Program Characteristics (independent variable) Definitions and Means^a

Variable	Description	Mean	SD
Program reach
MH patients	% of veterans service-connected for a MH condition who reside in the facility catchment area who received VHA MH care	51%	4%
Trans’l work	% of patients with SMI with transitional work visitsb	2%	2%
Sup empl	% of patients with SMI who received supportive employment visitsb	5%	6%
Homeless	% of homeless VHA patients who received any MH care	89%	5%
PTSD	% of patients with PTSD who received specialty outpatient care for PTSD	20%	9%
SUD spec	% of patients with SUD who initiated intensive treatment in a specialty setting	8%	6%
Case mgmnt	% of patients with SMI who received MH intensive case management for psychosis	3%	3%
Psychosocial service access
SMI	% of patients with SMI who initiated psychosocial treatment or psychotherapy	61%	10%
PTSD	% of patients with PTSD who initiated psychotherapy	59%	9%
SUD psy	% of patients with SUD who initiated psychosocial treatment or psychotherapy in any setting	47%	7%
Depression	% of patients with depression who initiated psychotherapy	39%	7%
Program intensity
Overall	Number of MH encounters per VHA patient	2.91	1.11
MH patients	Number of MH encounters per patient with any MH encounters	11.87	3.71
PRRC	Number of encounters at psychosocial rehab and recovery centers per VHA patient	28.13	34.39
Homeless	Number of MH encounters per homeless veteran with any MH encounters	29.89	13.36
SUD	Weeks of intensive outpatient SUD treatment per patient with any intensive outpatient SUD treatment visits	3.44	1.70
Trans’l work	Number of transitional work visitsb per patient with any transitional work visits	6.21	7.61
Sup empl	Number of supportive employment visitsb per patient with any supportive employment visits	6.22	4.33
Treatment continuity
Num resid	Number of outpatient visits in the 6 months post residential MH discharge	27.11	11.54
Num inpt	Number of outpatient visits in the 6 months post inpatient MH discharge	30.36	12.56
F/u resid	% of patients with 7-day follow-up after residential discharge	52%	23%
Gap	% of patients with an MH outpatient visit in 1 year who went 6 months without a second visit	43%	5%
SMI	% of patients with SMI with 8 psychotherapy or psychosocial treatment visits in 14 weeks	8%	4%
PTSD	% of patients with PTSD with 8 visits for PTSD psychotherapy in 14 weeks	14%	5%
Inpatient detox	% of patients who had inpatient detox with outpatient follow-up within 7 days	42%	12%
Outpatient detox	% of patients who had outpatient detox with outpatient follow-up within 7 days	37%	16%
SUD	% of patients with SUD with 8 SUD psychotherapy or psychosocial treatments in 14 weeks	21%	8%
Depression	% of patients with depression with 8 psychotherapy visits in 14 weeks	6%	3%

Empl indicates employment; F/u, follow-up; inpt, inpatient; mgmnt, management; MH, mental health; num, number; PRRC, psychosocial rehab and recovery center; psy, psychosocial; PTSD, posttraumatic stress disorder; resid, residential; SD, standard deviation; SMI, serious mental illness; SUD, substance use disorder; sup, supportive; trans’l, transitional; VHA, Veterans Health Administration.

^aAll measures calculated within a year unless specified otherwise. N = 6990 patients.

^bTransitional work visits and supportive employment visits are occupational therapy treatment modalities.^27,28

RESULTS

Descriptive Statistics

Tables 1 and 2 report means of demographic control variables, dependent variables (patient-level satisfaction), and key independent variables (facility-level program characteristics). eAppendix Table A3 reports the means for diagnostic risk-adjustment variables. Table 1 shows that the average age of patients in our sample was 62 years, 55% were married, 8% were female, and 12% were black. In addition, 17% of our sample had an alcohol use disorder; 9%, a drug use disorder, 37%, psychosis; and 48%, a depression diagnosis (all as defined by Elixhauser²⁵ and listed in eAppendix Table A3). These figures were higher than the general population because we deliberately selected a sample of patients with a VHA mental health visit. Table 1 also shows that, across our sample, most patients reported high levels of satisfaction for all but 1 measure. Table 2 shows facility-level program characteristics organized by the 4 domains.

The facility-level program characteristics in Table 2 were each computed by OMHO on the full sample of patients implied by each characteristic, not just the patients in our study sample. For example, the program reach characteristic of “PTSD” is defined as “% of patients with PTSD who receive specialty outpatient care for PTSD.” This means that this measure captures, for each facility and year, the percentage of patients with PTSD seen by the facility in that year who received specialty outpatient care for PTSD.

Multivariate Analysis

Separately, for each satisfaction measure (the dependent variable) and each program characteristic (the key independent variable), we estimated an ordinary least squares (OLS) model, controlling for age (in years), marital status (1 = married, 0 = not married), sex (1 = female, 0 = male), race (1 = black, 0 = nonblack), and comorbidities.²⁶ We accounted for heteroscedasticity with robust standard errors. In sensitivity analyses, we also ran models with clustering, facility random effects, and logistic regression. These produced similar results, which are not shown.

Based on the OLS models, for program reach, psychosocial service access, program intensity, and treatment continuity measures, Tables 3, 4, and 5 indicate a positive, negative, or not statistically significant association of program characteristics with satisfaction measures. (Coefficient estimates are provided in eAppendix Tables A4-A7.)

TABLE 3.

Risk-Adjusted Statistically Significant^a Associations of Program Reach Measures With Satisfaction Measures

	Access Satisfactionb			Encounter Satisfactionb
Performance Measure	Got Needed Appointment	Got Wanted Appointment	Got Needed Care	VHA Rating	Clinician Rating	Visit Rating
MH patients: % of MH service-connected veterans in facility catchment area who received VHA MH care		+	+	+	+
Trans’l work: % of patients with SMI who had transitional work visits	+	+	+		+	+
Sup empl: % of patients with SMI who received supportive employment visits
Homeless: % of homeless VHA patients who received any MH care
PTSD: % of patients with PTSD who received specialty outpatient care for PTSD
SUD spec: % of patients with SUD who initiated intensive SUD treatment in a specialty setting
Case mgmnt: % of patients with SMI who received MH intensive case management for psychosis	−	−	−

+ or − indicate positive or negative association, respectively; empl, employment; F/u, follow-up; inpt, inpatient; mgmnt, management; MH, mental health; num, number; PRRC, psychosocial rehab and recovery center; psy, psychosocial; PTSD, posttraumatic stress disorder; resid, residential; SMI, serious mental illness; SUD, substance use disorder; sup, supportive; trans’l, transitional; VHA, Veterans Health Administration.

^aSignificant at the P <.05 level. Results based on ordinary least squares estimates, adjusted for age, marital status, sex, race, and Elixhauser comorbidities²⁶ (eAppendix Table A3).

^bDefined in Table 1.

TABLE 4.

Risk-Adjusted Statistically Significant^a Associations of Psychosocial Service Access and Program Intensity Measures With Satisfaction Measures

	Access Satisfactionb			Encounter Satisfactionb
Performance Measure	Got Needed Appointment	Got Wanted Appointment	Got Needed Care	VHA Rating	Clinician Rating	Visit Rating
Psychosocial Service Access Measures
SMI: % of patients with SMI initiating psychosocial treatment or psychotherapy			−
PTSD: % of patients with PTSD initiating psychotherapy	−		−
SUD psy: % of patients with SUD initiating psychosocial treatment or psychotherapy in any setting						+
Depression: % of patients with depression initiating psychotherapy
Program Intensity Measures
Overall: number of mental health encounters per VHA patient
MH patients: number of MH encounters per patient with any MH encounters	+	+	+	+	+	+
PRRC: number of encounters at psychosocial rehab and recovery centers per VHA patient			−			−
Homeless: number of MH encounters per homeless veteran with any MH encounters		+		+		+
SUD: weeks of intensive outpatient SUD treatment per patient with any intensive outpatient SUD treatment visits	+			+
Transitional work: number of transitional work visits per patient with any transitional work visits
Supportive employment: number of supportive employment visits per patient with any supportive employment visits			+

+ or − indicate positive or negative association, respectively; empl, employment; F/u, follow-up; inpt, inpatient; MH, mental health; num, number; PRRC, psychosocial rehab and recovery center; psy, psychosocial; PTSD, posttraumatic stress disorder; resid, residential; SMI, serious mental illness; SUD, substance use disorder; trans’l, transitional; VHA, Veterans Health Administration.

^aSignificant at the P <.05 level. Results based on ordinary least squares estimates, adjusted for age, marital status, sex, race, and Elixhauser comorbidities²⁶ (eAppendix Table A3).

^bDefined in Table 1.

TABLE 5.

Risk-Adjusted Statistically Significant^a Associations of Treatment Continuity Measures with Satisfaction Measures

	Access Satisfactionb			Encounter Satisfactionb
Performance Measure	Got Needed Appointment	Got Wanted Appointment	Got Needed Care	VHA Rating	Clinician Rating	Visit Rating
Num resid: number of outpatient visits in 6 months post residential MH discharge		+	+	+		+
Num inpt: Number of outpatient visits in 6 months post inpatient MH discharge		+	+			+
F/u resid: % of patients with 7-day follow-up after residential discharge
Gap: % of patients with a MH outpatient visit in 1 year who went 6 months without a second visit	−	−	−	−	−	−
SMI: % of patients with SMI who had 8 psychotherapy or psychosocial treatment visits in 14 weeks	+	+	+	+	+	+
PTSD: % of patients with PTSD who had 8 visits for PTSD psychotherapy in 14 weeks	+	+	+	+	+	+
Inpatient detox: % of patients who had inpatient detox with outpatient follow-up within 7 days	+	+	+	+		+
Outpatient detox: % of patients who had outpatient detox with outpatient follow-up within 7 days
SUD: % of patients with SUD who had 8 SUD psychotherapy or psychosocial treatments in 14 weeks
Depression: % of patients with depression who had 8 psychotherapy visits in 14 weeks	+	+	+	+	+	+

+ or − indicate positive or negative association, respectively; empl, employment; F/u, follow-up; inpt, inpatient; MH, mental health; num, number; PRRC, psychosocial rehab and recovery center; psy, psychosocial; PTSD, posttraumatic stress disorder; resid, residential; SMI, serious mental illness; SUD, substance use disorder; trans’l, transitional; VHA, Veterans Health Administration.

^aSignificant at the P < .05 level. Results based on ordinary least squares estimates, adjusted for age, marital status, sex, race, and Elixhauser comorbidities²⁶ (eAppendix Table A3).

^bDefined in Table 1.

Program Reach

Table 3 shows that the broad measure of program reach, “MH patients”—the percentage of veterans service-connected for a mental health condition who received mental health care—and the percentage of VHA patients diagnosed with psychotic disorders, bipolar, major depression, or PTSD who had transitional work visits (“trans’l work”) were positively associated with at least 2 of 3 measures of access satisfaction over the prior 12 months. They were also each positively associated with 2 of 3 measures of encounter satisfaction.

None of the other 4 (more narrow) program reach measures were positively associated with access or encounter satisfaction. All but the percentage of patients with serious mental illness (SMI) who received mental health intensive case management for psychosis (“case mgmnt”) had no statistically significant association with satisfaction. Consistent with prior work,³⁰ condition-specific measures may have been negatively associated with satisfaction if patients with those conditions generally rated satisfaction lower (because of their condition, not their care) and their representation in our sample was higher at facilities that treat more of them.¹⁶ No matter the reason, the use of measures that were negatively associated with satisfaction, or not associated with it at all, should be justified and validated on other grounds (eg, they measured some aspect of clinically appropriate care). In “case mgmnt,” for example, availability of this treatment has been shown to improve the clinical outcomes of patients with chronic SMI.³¹

Psychosocial Service Access

Of all the categories of program characteristics, those pertaining to psychosocial service access were least associated with satisfaction measures (Table 4). This could be because initiating psychosocial services is a challenging time for patients, so facilities with greater access to it also have more patients who exhibited less satisfaction.

Program Intensity

With 1 exception (psychosocial rehab and recovery center), all program intensity measures were either positively associated with satisfaction or were not associated with any satisfaction measure (Table 4). Although the number of mental health encounters per unique patient seen at a facility (“overall”) was not associated with the satisfaction of its patients who received mental health care, the number of encounters per patient with any encounters (“MH patients”) was positively associated with all 6 satisfaction measures. Because the denominator of the overall measure was all VHA patients who were seen at a facility, it reflects both the proportion of patients in the healthcare system who received mental health treatment and the intensity of services felt by those who received care. A more focused decomposition of the overall measure into metrics that assessed 1) the reach of mental health services among patients requiring them (eg, the “MH patients” reach measure), and 2) the intensity of services among mental health care utilizers (eg, the “MH patients” intensity measure) was easier to interpret.

Measures of intensity of therapeutic and supportive employment program services (“sup empl” and “trans’l work”) were either not correlated with satisfaction or only associated with 1 access measure. It is possible that patients thought of these principally as employment programs and did not consider their experience with them when responding to healthcare satisfaction surveys.

As with other measures of services for patients with serious mental illness, the “psychosocial rehab and recovery centers” measure was negatively associated with satisfaction. Again, this may reflect enrichment of the patient population with patients who tended to rate healthcare services poorly.

Treatment Continuity

With few exceptions, continuity, variously measured, was positively associated with half or more of the access and/or encounter satisfaction measures (Table 5). In the case of the percentage of outpatients who received mental health care without a second visit in 6 months (“gap”), the association was negative with all 6 satisfaction measures, which is still consistent with the idea that less continuity of care is less satisfying to patients.

Limitations

Our analysis has a few limitations. First, it is observational, so we cannot infer causality. Also, our sample is of patients with a recent VHA mental health encounter. As such, it is not necessarily representative of all VHA enrollees or even all VHA enrollees with mental health diagnoses, many of whom may not have had a recent mental health visit. Third, the SHEP survey response rate is relatively low for patients younger than 50 years, which could threaten the generalizability of findings for that group. Finally, the survey instrument was not specifically designed to elicit impressions of mental health care only. It is possible they were also influenced by other aspects of VHA care.

DISCUSSION

In a sample of patients who visited the VHA for mental health conditions, we assessed the relationship between satisfaction and program characteristics, spanning multiple domains. Our results provide some important lessons for policy makers and healthcare managers. We found that satisfaction with VHA access among patients with mental health conditions was higher than satisfaction with care encounters. Broad measures of the program’s reach of mental health care treatment (ie, the proportion of patients served) and intensity (ie, the number of visits received) tended to be positively associated with both access and encounter satisfaction. No measures of access to psychosocial services (ie, the proportion of patients who received psychosocial services regardless of setting of care) and most measures of treatment continuity (ie, measures of outpatient follow-up after inpatient care) were positively associated with both kinds of satisfaction. Also, more narrow performance measures—those that focused on specific diagnostic populations (eg, those with PTSD and SMI)—were less likely to be positively associated with satisfaction. This is consistent with prior work that suggests certain types of patients who receive mental health care are less likely to be satisfied with care, perhaps more of a characteristic of the patients than the treatment programs that serve them.¹⁶

Policy makers and program managers should be aware that as they attempt to increase psychosocial service access, they may not see a positive relationship to satisfaction. Efforts to ensure initial access to psychosocial services to all patients who need them may negatively impact the availability of ongoing or more intensive services for those who initiate, as a larger pool of patients initiating services would compete for available treatment slots. We found that because measures of access to psychosocial services had the weakest relationship to satisfaction, while treatment continuity had the most consistent relationship, further investigation is needed. If one took satisfaction as the only assessment of health system performance, they might conclude that psychosocial treatment access is not valuable. That is not the right interpretation. Psychosocial treatment access is valuable for other reasons. For example, the subpopulation that does not respond adequately to medications may rely on this modality of care for improvement.

On the other hand, continuity was most consistently associated with greater satisfaction. This is in line with a growing body of work showing positive outcomes associated with continuity, such as better quality of life, community functioning, symptom reduction,³² increasing Global Assessment of Functioning scores,³³ and lower mortality risk.³⁴ Boden and Moos¹⁹ also showed that greater engagement with care is associated with higher satisfaction. It is likely that continuity of care was associated with greater satisfaction because those who were not satisfied with care tended to be lost to follow-up. Continuity of care may directly cause greater satisfaction (because patients want it), but the reverse may also be true: patients who are satisfied (for other reasons) may be more likely to return for subsequent appointments, increasing measured continuity of care.

One measure for which we found no relationship to satisfaction is hard to justify on nonsatisfaction-based grounds: the program intensity “overall” measure of the number of mental health encounters per unique VHA patient. Because this measure could go up through a reduction in VHA patients, independent of their mental health care needs, and because there are other valid measures of intensity more specific to mental health patients, this did not appear to be a measure of high value.

Several measures in each domain may capture the same or similar aspects of care. For instance, several reach measures were positively correlated and exhibited similar patterns of relationships to satisfaction. Such redundancy is useful for managers, particularly in the context of incentivized or prioritized measure performance. As has been observed in other work,^13,27 incentives can lead to loss of fidelity in the data underlying metrics,³⁴ as behavior may be modified in direct response to the measure. If a metric tied to incentives starts to deviate considerably from another measuring the same thing in a different way, but it is not tied to incentives, that is a signal that the integrity of the underlying data may have been affected by the incentives.

CONCLUSIONS

With a few exceptions, this research demonstrates that the set of mental health program characteristics used by the VHA exhibits the expected associations with patient satisfaction and should be useful in monitoring patient-centered aspects of care quality.

TAKEAWAY POINTS.

• Broad measures of mental health care program reach and intensity and most measures of treatment continuity were consistently and positively associated with patient satisfaction.

• Because psychosocial services are clinically valuable, policy makers and managers should not interpret a lack of association with satisfaction to justify reducing their availability.

• Policy makers and managers should be aware that satisfaction with care, as currently measured, may not rise as more patients initiate treatment, unless continuity of care is maintained or enhanced.

• Policy makers and managers should continue to track patient satisfaction and to specifically target satisfaction with mental health care.

eAppendix

Table A1.

SHEP Response Rates

Patient Characteristic	Response Rate
All patients	43.6%
Male	44.7%
Under age 50 years	13.4%
Age 50–64 years	38.4%
Age 65–80 years	55.2%
Age 80 or more years	61.0%

SHEP indicates Survey of Healthcare Experiences of Patients

Table A2.

SHEP Skip Pattern and Item Nonresponse

Variable	Question and Skip Pattern	N (of 6990)
Access satisfaction
Got needed app’t	Q: In the last 12 months, when you needed care right away, how often did you get care as soon as you thought you needed? Skipped if answered “no” to prior question about whether needed care right away.	3546 answered of 3848 eligible (92%) because 3142 answered “no” to the indicated prior question
Got wanted app’t	Q: In the past 12 months, not counting the times you needed care right away, how often did you get an appointment as soon as you thought you needed? Skipped if answered “no” to prior question about whether made any appointments excluding those for urgent care.	5333 answered of 5557 eligible (96%) because 1433 answered “no” to the indicated prior question
Got needed care	Q: In the past 12 months, how often was it easy to get the care, tests, or treatment you thought you needed through the VA? Skipped if answered “no” to prior question about whether tried to get any VA care.	6070 answered of 6215 eligible (98%) because 775 answered “no” to the indicated prior question
Encounter satisfaction
VHA rating	Q: Using any number from 0 to 10, where 0 is the worst healthcare possible and 10 is the best healthcare possible, what number would you use to rate all your VHA healthcare in the last 12 months? No skip pattern	6917 answered of 6990 eligible (99%)
Clinician rating	Q: Using any number from 0 to 10, where 0 is the worst personal doctor/nurse possible and 10 is the best personal doctor/nurse possible, what number would you use to rate your personal VHA doctor/nurse? Skipped if don’t have a personal VA doctor/nurse	6289 answered of 6425 eligible (98%) because 565 indicated they didn’t have a personal doctor/nurse
Visit rating	Q: All things considered, how satisfied were you with the VHA during your recent visit? No skip pattern	6990 answered of 6990 eligible (100%)

SHEP indicates Survey of Healthcare Experiences of Patients; VA, Department of Veterans Affairs; VHA, Veterans Health Administration

Table A3.

Means for Elixhauser Comorbidities²⁶ (N = 6990 patients)

Variable	Description	Mean
CHF	Congestive heart failure	0.04
card arr	Cardiac arrhythmias	0.08
valve disease	Valvular disease	0.02
pulm circ	Pulmonary circulation disorder	0.00
periph vasc	Peripheral vascular disorder	0.05
Hypertension	Hypertension	0.62
Paralysis	Paralysis	0.01
oth neuro	Other neurological disorder	0.04
chron pulm	Chronic pulmonary disease	0.18
DM, uncomp	Diabetes, uncomplicated	0.30
DM, comp	Diabetes, complicated	0.08
hypothy	Hypothyroidism	0.08
Renal failure	Renal failure	0.05
Liver disease	Liver disease	0.04
Peptic ulcer	Peptic ulcer disease	0.01
AIDS	AIDS	0.00
Lymphoma	Lymphoma	0.01
metast canc	Metastatic cancer	0.00
Tumor	Solid tumor without metastasis	0.11
Arthritis	Rheumatoid arthritis/collagen vascular disease	0.02
coag	Coagulopathy	0.02
Obesity	Obesity	0.24
Weight loss	Weight loss	0.00
Fluid	Fluid and electrolyte disorders	0.03
Blood loss	Blood loss anemia	0.00
Deficiency	Deficiency anemia	0.08
Alcohol	Alcohol use disorder	0.17
Drug	Drug use disorder	0.09
Psychoses	Psychoses	0.37
Depression	Depression	0.48

Table A4.

Risk-Adjusted OLS Coefficients for Program Reach Measures Models

Performance measure (defined in Table 2)	Access satisfaction (defined in Table 1)				Encounter satisfaction (defined in Table 1)
	got needed app’t	got wanted app’t	got needed care	# positive | # negative	VHA rating	clinician rating	visit rating	# positive | # negative
MH patients	+0.3386	+0.6432a	+0.4443a	2 | 0	+0.5754a	+0.4095a	+0.2039	2 | 0
trans’l work	+0.6216a	+0.8659a	+0.5291a	3 | 0	+0.4282	+0.7584a	+0.6133a	2 | 0
sup empl	−0.0056	+0.0450	+0.0163	0 | 0	−0.0266	−0.0505	+0.0440	0 | 0
homeless	−0.0373	−0.1363	−0.0855	0 | 0	−0.0030	−0.0504	+0.0928	0 | 0
PTSD	−0.1580	+0.0249	+0.0124	0 | 0	+0.1064	+0.0922	+0.0465	0 | 0
SUD spec	+0.0028	+0.1648	+0.0754	0 | 0	+0.0490	+0.1054	+0.1356	0 | 0
case mgmnt	−0.5648a	−0.8411a	−0.6640a	0 | 3	−0.4471	−0.2083	−0.3430	0 | 0

Results based on OLS estimates, adjusted for age, marital status, sex, race, and Elixhauser comorbidities²⁶ (Appendix Table A3)

^aSignificant at the P <.05 level

OLS indicates ordinary least squares; VHA, Veterans Health Administration; MH, mental health; PTSD, post-traumatic stress disorder; SUD, substance use disorder

Table A5.

Risk-Adjusted OLS Coefficients for Psychosocial Service Access Measures Models

Performance measure (defined in Table 2)	Access satisfaction (defined in Table 1)				Encounter satisfaction (defined in Table 1)
	got needed app’t	got wanted app’t	got needed care	# positive | # negative	VHA rating	clinician rating	visit rating	# positive | # negative
SMI	−0.0864	−0.0795	−0.1189a	0 | 1	−0.0190	−0.0424	+0.0083	0 | 0
PTSD	−0.2546a	−0.0591	−0.1288a	0 | 2	−0.0318	−0.0812	−0.0617	0 | 0
SUD psy	−0.0667	+0.1306	−0.0359	0 | 0	+0.1359	+0.1206	+0.1727a	1 | 0
depression	−0.1060	−0.0439	−0.1151	0 | 0	−0.0051	−0.0758	−0.0064	0 | 0

Results based on OLS estimates, adjusted for age, marital status, sex, race, and Elixhauser comorbidities²⁶ (Appendix Table A3)

^aSignificant at the P <.05 level

OLS indicates ordinary least squares; PTSD, post-traumatic stress disorder; SMI, serious mental illness; SUD, substance use disorder; VHA, Veterans Health Administration

Table A6.

Risk-Adjusted OLS Coefficients for Program Intensity Measures Models

Performance measure (defined in Table 2)	Access satisfaction (defined in Table 1)				Encounter satisfaction (defined in Table 1)
	got needed app’t	got wanted app’t	got needed care	# positive | # negative	VHA rating	clinician rating	visit rating	# positive | # negative
Overall	+0.0025	+0.0070	+0.0011	0 | 0	+0.0066	+0.0097	+0.0049	0 | 0
MH patients	+0.0042a	+0.0065a	+0.0040a	3 | 0	+0.0045a	+0.0043a	+0.0032a	3 | 0
PRRC	−0.0003	−0.0003	−0.0004a	0 | 1	−0.0002	−0.0002	−0.0006a	0 | 1
Homeless	+0.0004	+0.0011a	+0.0007	1 | 0	+0.0012a	+0.0003	+0.0012a	2 | 0
SUD	+0.0085a	−0.0023	+0.0001	1 | 0	+0.0076a	+0.0039	+0.0033	1 | 0
trans’l work	−0.0011	+0.0005	+0.0008	0 | 0	−0.0007	−0.0001	−0.0002	0 | 0
sup empl	−0.0014	+0.0022	+0.0048a	1 | 0	+0.0006	−0.0011	+0.0015	0 | 0

Results based on OLS estimates, adjusted for age, marital status, sex, race, and Elixhauser comorbidities²⁶ (Appendix Table A1)

^aSignificant at the P <.05 level

OLS indicates ordinary least squares; PTSD, post-traumatic stress disorder; SMI, serious mental illness; SUD, substance use disorder; VHA, Veterans Health Administration

Table A7.

Risk-Adjusted OLS Coefficients for Treatment Continuity Measures Models

Performance measure (defined in Table 2)	Access satisfaction (defined in Table 1)				Encounter satisfaction (defined in Table 1)
	got needed app’t	got wanted app’t	got needed care	# positive | # negative	VHA rating	clinician rating	visit rating	# positive | # negative
num resid	+0.0004	+0.0017a	+0.0013a	2 | 0	+0.0013a	+0.0009	+0.0010a	2 | 0
num inpt	+0.0006	+0.0010a	+0.0009a	2 | 0	+0.0009	+0.0003	+0.0009a	1 | 0
f/u resid	+0.0287	+0.0402	+0.0410	0 | 0	+0.0159	+0.0327	−0.0042	0 | 0
Gap	−0.3596a	−0.5600a	−0.3295a	0 | 3	−0.3746a	−0.4477a	−0.2383a	0 | 3
SMI	+0.5971a	+0.7716a	+0.6239a	3 | 0	+0.5269a	+0.6379a	+0.5125a	3 | 0
PTSD	+0.3840a	+0.3442a	+0.2681a	3 | 0	+0.4020a	+0.3394a	+0.3361a	3 | 0
inpt detox	+0.1418a	+0.1856a	+0.1309a	3 | 0	+0.1175a	+0.0525	+0.1037a	2 | 0
outpt detox	+0.0295	+0.0428	+0.0318	0 | 0	+0.0229	−0.0276	−0.0060	0 | 0
SUD	+0.0552	+0.0947	+0.0500	0 | 0	+0.0543	+0.0765	+0.1124	0 | 0
Depression	+0.6519a	+0.5339a	+0.3861a	3 | 0	+0.5483a	+0.5038a	+0.4353a	3 | 0

Results based on OLS estimates, adjusted for age, marital status, sex, race, and Elixhauser comorbidities²⁶ (Appendix Table A1)

^aSignificant at the P < .05 level

OLS indicates ordinary least squares; PTSD, post-traumatic stress disorder; SMI, serious mental illness; SUD, substance use disorder; VHA, Veterans Health Administration