Access to Care for Youth in a State Mental Health System: A Simulated Patient Approach

Abstract

Objective

To examine access to psychiatric care for adolescents with depression in outpatient specialty clinics within a state mental health system, using a simulated patient approach.

Method

Trained callers posed as the mother of a 14-year-old female with depression, following a script. A stratified random sample (n = 264) of 340 state-licensed outpatient mental health clinics that serve youth was selected. Clinics were randomly assigned to season and insurance condition. We examined if access varied by season, clinic characteristics, and caller insurance type. Weighted logistic and linear mixed effects regression models were fitted to examine associations with appointment availability and wait times.

Results

Among clinics where a treatment appointment could be scheduled, appointment availability differed by season. Clinics who had participated in state-sponsored trainings targeting access were more available. Wait times for treatment appointments varied by season and region. Wait times in New York City were shorter than in some other regions. Although callers were 4.1 times more likely to be able to schedule a psychiatry appointment in the spring, wait times for psychiatry appointments were significantly longer in the spring than in the summer (49.9 vs. 36.7 days). Wait times for therapy appointments were significantly shorter in community than hospital clinics (19.1 days vs. 35.3 days).

Conclusion

Access to psychiatric care for youth with depression was found to be variable in a state system. State-sponsored trainings on strategies to reduce wait times appear to improve care access. The simulated patient approach has promise for monitoring the impact of healthcare policy reforms on care quality measures.

Introduction

Access to timely mental health care is an important issue given well-documented disparities in psychiatric care access and wide-ranging changes in health care delivery underway as a result of the Affordable Care Act (ACA). Ensuring that youth with mental health needs are offered timely access and access to quality services is especially important given the potential lifelong effects of childhood psychiatric problems and societal costs generated by untreated mental health needs.¹ Accountability expectations set forth by the ACA have increased attention to issues of equitable access to effective mental health care and establishment of measureable standards.^2-4 State mental health systems are experiencing massive changes consequent to the 2008 recession, followed by the Mental Health Parity Act, and more recently, the ACA.⁵ Altogether, the recession and these major policies have led states to be increasingly concerned about service costs, quality, and outcomes. As states and health plans are restructured to contain costs, ensuring appropriate access for the populations for whom they are responsible is of paramount importance. Disparities in access to specialty psychiatric care, especially for youth, have been well documented. Variations in insurance coverage and geographic location are among the most commonly reported contributors to differences in access, independent of population characteristics or level of need. ^6-10

Despite increased national attention to issues of access, few empirical studies have examined access to mental health care using rigorous research methodology. Defining access adequacy varies by state and health plan, and the most widely used methods to track access include complaint tracking, surveys of patients and providers, and data provided by health plans (such as numbers of providers in a geographical area). However, these estimates are limited: they are global and may not reflect actual access or address the experience of individuals who actually attempt to access services.¹¹ Actual experiences are critical because individuals who seek services vary greatly in their need for services, the persistence with which they seek them, and their ability to navigate complex health systems. Moreover, existing data are subject to numerous problems. Patient surveys may suffer from non-representative samples and recall bias. Social desirability may bias the data from providers or health plans. Rigorous and practical strategies to assess service access are needed to understand the impact of the changing health care landscape on youth psychiatric services.

One such rigorous methodology that has gained recent attention is the use of simulated patients or “mystery shoppers.” Mystery shopping originally referred to private investigators hired by banks and retail stores to assess for employee theft or fraud. It has evolved into “a form of research whereby individuals measure any type of customer service process by acting as actual or potential customers.”¹² Simulated patient is the term most often used in medical settings where actors are trained to play patients for the purpose of training and evaluation of medical services.¹³ In contrast to more conventional approaches, the simulated patient approach confers some benefits. Simulated patients/caregivers provide accurate, real-time estimates of appointment availability and wait times. When incorporated into a well-designed empirical study, these approaches allow for assessment of additional variables that may impact access, such as time of year, type of insurance, and caller persistence.^{8, 11, 14}

Three recent studies have used a simulated patient methodology to examine access to care for depression. In a study of adult patients, those presenting with depression were able to make an appointment significantly less often than patients calling for a medical complaint, and only 12% of calls for depression resulted in an appointment within 2 weeks (compared to 40% for a medical complaint).⁸ In two studies evaluating specialty psychiatric care for adolescents with depression, significant disparities in appointment availability and wait time by insurance status were evident, and over all less than 30% of appointments for routine medication management could be scheduled within 30 days.^{11, 14} However, neither study examined the range of available community mental health services nor examined clinic characteristics or regional differences that could influence appointment availability and wait times.

Through a partnership with the Office of Mental Health (OMH) of New York State (NYS), we designed a study to examine access to mental health care for adolescents with depression in all OMH licensed outpatient clinics serving youth and to test the usefulness of a simulated patient methodology for state systems. We used adolescent depression as a tracer condition because depression is common among adolescents, with up to 12% meeting full diagnostic criteria for depression.^15-17 Although the sequelae of depression may be life-threatening and effective treatments exist, data suggest that 60-80% of symptomatic adolescents do not receive appropriate care.¹⁸ Barriers to care are numerous, including scarcity of mental health providers, limited access to appropriate care, and attitudinal and practical barriers even when care is available.^19-21 This study measured appointment availability and wait times for psychiatry and therapy appointments, as well as availability of cognitive-behavioral therapy (CBT), an evidence-based psychotherapy for adolescents with depression. We also examined variation in appointment availability and wait time for appointments by clinic characteristics (e.g., affiliation, payer mix, client mix, geographic region, urbanicity), caller insurance type (Medicaid or private), and season contacted (spring or summer).

Method

Study Sample

The focus of this study was the 340 NYS outpatient mental health clinics licensed by OMH to serve children and adolescents that were in operation since October 2012 and at the time of data collection (spring/summer 2014). As hospitals and state-operated facilities represented a small percentage of the population (17% and 5%, respectively), these clinics, as well as all community clinics on Long Island (8.5%), were included in the study. Approximately 70% of the remaining 265 community-based clinics were selected via stratified random sampling by region (New York City [NYC], Hudson, Central, and Western) and urbanicity (i.e., metropolitan vs. non-metropolitan county). Clinics were randomly assigned to season (first call attempt in the spring [April/May] or summer [July/August]) and insurance condition (Medicaid/private insurance) (Figure 1). Because time of year likely influences demands for youth psychiatric services, we elected to call clinics during a more academically demanding and hence stressful spring season and during the summer vacation months. Private insurance was defined as coverage by Blue Cross Blue Shield (BCBS), as it held the largest market share in NYS at the time of data collection. For clinics that did not have a contract with BCBS, callers in the private insurance condition offered to self-pay. This study was a quality improvement project initiated by the NYS OMH and was exempt from review by the Institutional Review Board.

Figure 1.

Study recruitment.

Study Protocol

Following a standardized protocol and script, four trained research staff posed as a mother who had recently moved to the area and was seeking mental health services for her 14-year-old daughter. The adolescent, who had been prescribed antidepressant medication by her primary care doctor, had begun showing signs of functional decline. The caller attempted to schedule appointments with a psychiatrist and a therapist for her daughter, and inquired whether cognitive-behavioral therapy (CBT) was available. When necessary, the caller disclosed predetermined details about her daughter's previous success with CBT and information about her prescription (dose, one-month supply remaining, etc.).

Callers made up to five separate call attempts at each clinic to schedule an appointment when there was some initial contact with the clinic (e.g., the caller spoke to a receptionist but was waiting to hear back from intake personnel). Callers left detailed voicemail messages that included their name, phone number, and their primary reason for calling, and waited 48 hours before a subsequent call attempt if the clinic did not return their call. If the caller was unable to speak with anyone at the clinic, three separate call attempts were made.

Callers were trained both didactically and experientially to conduct calls and record information on data collection forms. For training purposes, calls were made to 11 clinics. For quality assurance and to ensure fidelity to the study protocol, callers were observed on the first 4 calls made as well as on a random sample of 20% of subsequent calls. Data collection forms were independently coded by two research staff members and reviewed by a third; discrepancies were reconciled via group discussion. Across four part-time research staff, all calls were completed between April and September.

Measures

Availability of services

Four binary variables summarized whether the caller was able to 1) discuss scheduling an appointment; 2) schedule an intake/screening for a psychiatry appointment; 3) schedule an intake/screening for a therapy appointment; 4) identify if CBT was available.

Psychiatry and therapy appointment wait-times

For clinics that provided an exact appointment date, wait-time was calculated as the difference between the call date and the date of the appointment. If an approximate time frame (e.g., “in 2-3 weeks”) or date (e.g. “first week of June”) was provided, wait time or appointment date was the average (e.g., 17 days or Wednesday 6/4/14, respectively). For clinics that did not provide a date or time frame but outlined the steps the family needed to complete prior to the appointment, the number of days to complete each step was determined a priori by the authors based on clinical experience (e.g., insurance pre-authorization = 2 days, intake appointment = 7 days, therapy appointment = 7 days). Thus, wait times included all the steps a clinic required a family to take prior to the treatment appointment, including time to the intake appointment.

Clinic Characteristics

Clinic characteristics of interest were derived from various sources, and have been described in detail elsewhere.^22-23 Briefly, clinics were characterized by OMH-defined administrative regions (Central, Hudson, Long Island, NYC, Western) and clinic types (hospital, state-operated facilities vs. community clinics). Data from the US Department of Health and Human Services Area Health Resources Files (AHRF) provided county rural–urban continuum codes that were used to categorize clinics by urbanicity (metropolitan versus non-metropolitan). Participation in state-sponsored training, which clinics volunteered for, were derived from attendance logs collected through the state's Clinical Technical Assistance Center. These trainings were categorized as evidence-based clinical trainings (e.g., motivational interviewing, trauma care, CBT) or business practice trainings (e.g., financial modeling tools and strategies to improve fiscal health).²² We also examined clinic participation in two training topics thought to be particularly relevant for the current study, as they addressed targeted clinic activities aimed at improving timely access to services; they included trainings on centralized scheduling and open access to reduce wait times. Clinical capacity was derived through the NYS OMH Consolidated Fiscal Report (CFR) system, which provided information on clinical operational structure, including the annual total clinical full-time equivalents. The NYS OMH Patient Characteristics Survey (PCS) provided information on the proportion of youth clients (<18 years of age), proportion of youth clients with a serious emotional disturbance, and the proportion of youth visits billed to Medicaid, either fee-for-service or managed care plans.

Data Analysis

To account for the unequal probability of selection of clinics, sample weights were created using the inverse of the probability of selection. Weighted means and standard errors, weighted percentiles (median, 25th and 75th percentiles), and weighted percentages were used to summarize normally distributed, skewed, and categorical variables, respectively. Logistic and linear mixed-effects regression models with a random effect for agency (to account for the within-agency correlation; 21% of agencies had more than one clinic associated with them) and sample weights were fitted to examine associations with appointment availability and wait times. Only clinics that offered appointments were included in modeling wait times. OMH facilities were excluded from the regression models, as clinic characteristics data were not collected for these facilities. Therapy and psychiatry appointment wait times were right-skewed and were log-transformed to achieve approximate normality. Multiple regression models included all study design variables (season, clinic type, OMH region, urbanicity and insurance type), regardless of their statistical significance. Clinic characteristics that were statistically significant (p<.05) after adjusting for study design variables were retained in the final model. The results of the logistic and linear models are summarized using adjusted odds ratios (aOR) and regression coefficients (β), respectively, and their 95% CIs. The data were analyzed using SAS version 9.3 (SAS Institute, Cary, NC) and Stata/SE version 14.0.²⁴

Results

Callers were able to discuss scheduling an appointment at 248 of the 264 clinics that were randomized (94%). The majority of these 248 clinics were located in NYC (43%) and in metropolitan areas (86%) (Table 1). Nearly two-thirds had participated in state-sponsored trainings on evidence-based practices (61%) and business practices (62%); 34% attended a centralized scheduling or open access training. The median number of clinical fulltime equivalents was 9.5. At half of the clinics, fewer than 30% of the clients served were <18 years old. The 16 clinics where callers were unable to discuss scheduling appointments were largely similar to the more accessible clinics (Table 1), with the exception that these 16 clinics had a larger proportion of youth visits billed to Medicaid compared to the other 248 more accessible clinics (p=.04).

Table 1. Characteristics of Clinics That Could (n=248) and Could Not (n=16) Be Reached.

	Clinics that Could Be Reached (n=248)		Clinics that Could Not Be Reached (n=16)
	n	Weighted % or Median (25th, 75th %tile)	n	Weighted % or Median (25th, 75th %tile)
Study Design Variables
Season of Initial Contact
Spring	127	51.1	9	57.9
Summer	121	48.9	7	42.1
Clinic Characteristics
Type of Clinic:
Hospital	53	16.3	4	19.3
State-operated Facility	16	4.9	1	4.8
Community Clinic	179	78.8	11	75.9
Geography/OMH Regions:
Central	35	13.8	1	4.8
Hudson	45	17.4	3	20.7
Long Island	30	12.7	2	13.8
New York City (NYC)	106	42.9	10	60.7
Western	32	13.2	0	0
Urbanicity:
Non-Metropolitann (Non-Metro)	33	13.6	1	4.8
Metropolitan (Metro)	215	86.4	15	95.2
Client characteristic:
Insurance:
Private	125	50.0	6	39.3
Medicaid	123	50.0	10	60.7
Covariates
Clinic Characteristics
Participation in state-sponsored trainings:
Evidence-based practices trainingsa	147	60.5	12	74.5
Business practices trainingsb	152	61.9	11	69.7
Centralized scheduling or open access trainings	83	33.9	6	41.4
Clinical Capacity:
Number of clinical full-time equivalents (FTEs)	204	9.5 (5.7, 16.1)	13	10.1 (7.0, 12.4)
Client Profile:
% of clients who are youth (age < 18 years)	223	29.8 (14.3, 61.4)	16	28.2 (4.2, 41.9)
% of youth visits billed to Medicaid	206	68.9 (55.1, 82.2)	15	73.4 (68.8, 98.1)
% of youth clients with a serious emotional disturbance	206	26.8 (14.1, 50.0)	15	45.9 (24.3, 52.9)

Note: Unweighted count (weighted %) shown for categorical variables; weighted median (25^th, 75^th percentile) shown for continuous variables. Details about definitions of evidence-based practices and business practices can be found in Chor et al., 2014.²² OMH = Office of Mental Health.

^aEvidence-based practices trainings include a variety of clinical topics such as trauma care, motivational interviewing, family engagement, cognitive-behavioral therapy (CBT), clinical support tools, and packaged curriculum for children with disruptive behavioral disorders.

^bBusiness practices trainings include a variety of topics such as financial modeling tools to develop effective business models, effective documentation, open access, and centralized scheduling.

Of the 248 clinics reached, nearly all (99%) clinics required at least one in-person appointment (e.g., intake/screening with a non-psychiatrist) prior to the psychiatry appointment. Callers were able to schedule an intake/screening for a psychiatry appointment at 63% of the clinics. Common reasons why callers could not schedule intake/screening for a psychiatry appointment were that clinics were not accepting new patients (14%) or that the adolescent was too young (9%). Callers were able to schedule a therapy appointment at 67% of the clinics, and CBT was available at 74% of those clinics. In the 155 clinics where callers were able to schedule a treatment appointment, an intake appointment was scheduled within 2 days for 20% of calls, 3-7 days for 34% of calls, 8-14 days for 21% of calls, 15-21 days for 11% of calls, 22-28 days for 4% of calls, and 4 or more weeks for 10% of calls.

Logistic mixed-effects models showed that after adjusting for study design variables, the odds of scheduling an intake for a psychiatry appointment were significantly higher in the spring than summer (aOR=4.1, 95% CI: 1.2, 14.5, p=.03) and among clinics that previously participated in a state-sponsored centralized scheduling/open access training (aOR=3.9, 95% CI: 1.1, 14.5, p=.04) (Table 2). Participation in a state-sponsored centralized scheduling or open access training was also associated with significantly higher odds of scheduling a therapy appointment (aOR=4.8, 95% CI: 1.1, 20.5, p=.04). None of the other variables were significantly associated with scheduling psychiatry or therapy appointments.

Table 2. Weighted Mixed Effects Logistic Regression: Adjusted Odds of Being Able to Schedule a Psychiatry or Therapy Intake Appointment at Community Clinics and Hospitals (N=232).

	Able to Schedule Psychiatry Intake Appointment			Able to Schedule Therapy Intake Appointment
	aOR	95% CI	p-val	aOR	95% CI	p-val
Study Design Variables
Season of Initial Contact
Spring vs. Summer	4.11	1.16, 14.53	.03*	3.91	0.93, 16.38	.06
Clinic Characteristics
Type of Clinic:
Community Clinic vs. Hospital	1.48	0.40, 5.46	.56	2.40	0.55, 10.49	.25
Geography/OMH Regions (reference=NYC):
Central	1.25	0.13, 11.67	.19	0.65	0.06, 7.57	.08
Hudson	8.96	0.91, 88.27		11.80	1.12, 124.48
Long Island	0.50	0.04, 5.60		0.33	0.05, 2.10
Western	1.50	0.22, 10.11		8.78	0.63, 121.81
Urbanicity:
Non-Metro vs. Metro	1.39	0.17, 11.49	.76	3.65	0.24, 54.83	.35
Client Characteristic
Insurance:
Medicaid vs. Private	1.21	0.38, 3.86	.74	1.65	0.34, 7.96	.53
Covariate
Clinic Characteristics
Participation in trainings on centralized scheduling or open access:
Yes vs. No	3.93	1.06, 14.50	.04*	4.77	1.11, 20.45	.04*

Note: Bolded text indicates significant comparisons. aOR = adjusted odds ratio; OMH = Office of Mental Health.

^*denotes significant p-value.

The median wait time for a psychiatry appointment (which includes time for completing the intake/screening appointment) was 42 days (interquartile range 31-63 days; range 9-205 days). Estimates from the multivariable linear mixed effects models showed that psychiatry appointment wait times were 36% longer in the spring than summer (49.9 days vs. 36.7 days; β=.310, SE=.122, p=.01) (Table 3). Compared to clinics in NYC, psychiatry appointment wait times were approximately 58% longer at clinics in Central NY (36.8 days vs. 58.4 days; β=.463, SE=.200, p=.02) and Long Island (36.8 vs. 57.9 days; β=.458, SE=.165, p=.005). Clinics that had participated in state-sponsored evidence-based practices trainings had significantly shorter psychiatry appointment wait times than those that had not (38.3 days vs. 55.5 days; β=-.371, SE=.165, p=.02).

Table 3. Weighted Mixed Effects Linear Regression: Adjusted Associations of Study Design Variables and Clinic Characteristics With Psychiatry and Therapy Appointment Wait Time^a at Community Clinics and Hospitals.

	Natural Log of the Number of Days Until the Psychiatry Appointment (n=139)			Natural Log of the Number of Days Until the Therapy Appointment (n=155)
Study Design Variables	β	SE	p-val	β	SE	p-val
Season of Initial Contact
Spring vs. Summer	.310	.122	.01	-.128	.193	.51
Clinic Characteristics
Type of Clinic:
Community Clinic vs. Hospital	-.083	.135	.54	-.619	.180	.001
Geography/OMH Regions (reference=NYC):
Central	.463	.200	.02	.443	.216	.03
Hudson	.169	.167		.003	.235
Long Island	.458	.165		.420	.218
Western	.325	.183		-.036	.304
Urbanicity:
Metro vs. Non-Metro	-.097	.196	.62	.230	.224	.31
Client Characteristic
Insurance:
Medicaid vs. Private	.125	.127	.33	-.162	.132	.22
Covariate
Clinic Characteristic
Participation in trainings on evidence-based practices:
Yes vs. No	-.371	.165	.02	n/a	n/a	n/a

Note: Bolded text indicates significant comparisons. N/a indicates that the variable was not statistically significant at p<.05 and was not included in the model.

^aEach outcome (number of days until psychiatry appointment; number of days until therapy appointment) was log transformed to achieve approximate normality.

^*denotes significant p-value.

The median wait time for a therapy appointment was 21 days (interquartile range=11-33 days; range 1-181 days). Multivariable linear mixed-effects models showed that therapy appointment wait times were significantly shorter at community clinics compared to hospital clinics (19.1 days vs. 35.3 days; β=-.619, SE=.180, p=.001). Therapy appointment wait times at clinics in central NY were 56% longer compared to clinics in NYC (30.9 days vs. 19.9 days; β=.443, SE=.216, p=.04) and were 61% longer compared to clinics in western NY (30.9 days vs. 19.1 days; β=.479, SE=.238, p=.05). None of the other variables were significantly related to therapy appointment wait times.

Discussion

This study documents disparities in access (defined in terms of availability and timeliness) to psychiatric care for youth with depression across a state mental health system. It is also a proof of concept testing the use of an innovative, rigorous, yet practical method for assessing an important component of service delivery. The collaboration between academic and state partners facilitated the feasibility of this work, which occurred over 15 months, from the time of study inception to report completion. Building on the already-developed study protocols, we estimate that this work could feasibly be replicated within 6 to 9 months, depending on the staffing resources, availability of necessary state data (list of clinics, type, contact information) and study design (e.g. calling across seasons vs. electing to focus only on one season). Such methods are particularly needed given the substantial reforms undertaken by states to integrate and coordinate behavioral and physical health care.²⁵

Callers were able to get an initial appointment at approximately two-thirds of clinics that are licensed by the state to treat youth with psychiatric problems. A positive finding is that the ability to schedule an intake for a psychiatry or therapy appointment did not differ by type of clinic, region of the state, urbanicity, or insurance status. Importantly, participation in state-offered trainings on centralized scheduling or open access was significantly associated with the clinic's ability to schedule an intake for a psychiatry and a therapy appointment. This finding highlights the importance of targeted state training efforts. The lack of insurance finding was unexpected, given prior studies that have shown differential impact of public insurance status on access.^{8, 10, 26} This finding likely reflects NYS' efforts to reduce disparities in access through enhanced Medicaid rates.²⁵

The effect of seasonality on access was noted only for psychiatry appointments. The ability to schedule an intake for a psychiatry appointment was 4.1 times higher in the spring than summer. Although callers in the spring were more likely to be able to schedule psychiatry appointments, wait times were significantly longer in the spring than in the summer. The reduced wait times for psychiatry appointments in the summer may reflect diminishing demand for appointments during the summer vacation months. Yet, ironically, the ability to schedule an intake for a psychiatry appointment in the summer months was much lower than in the spring. This phenomenon may reflect clinic workflow, where clinics stop scheduling psychiatry appointments in summer as waitlists from the spring begin to exceed capacity. Post hoc analyses support this hypothesis, where nearly twice as many clinics in the summer compared to the spring (22 vs. 12) informed the caller that the psychiatrist was not accepting new patients or there was a wait list to see a psychiatrist.

The workforce distribution across the state also appears to influence access and wait time. Consistent with the shortage of psychiatrists, wait times for a psychiatry appointment were twice as long as the wait time for a therapy appointment (42 vs. 21 days). In addition to the effect of seasonality on wait times, region and clinics' participation in state-sponsored evidence-based practice trainings were significant predictors. Interestingly, clinics that participated in state-supported evidence-based practices training had significantly shorter psychiatry appointment wait times, even though such training focused on psychosocial treatment strategies. In our prior work, we found that larger clinical staff capacity and having a higher proportion of youth clients predicted participation in state-supported evidence-based practice training.²³ Having a larger clinical staff capacity may explain the shorter psychiatry appointment wait times in these clinics. However, in our model, clinical staff capacity (i.e., clinical fulltime equivalent) was not significantly associated with psychiatry wait times. We speculate that in clinics that participated in state-supported evidence-based practices, non-physician clinicians may have greater clinical efficacy and hence are less reliant on psychiatry to address children's psychiatric problems, thus reducing the demands on psychiatry. Future work should test the hypothesized relationship between clinical efficacy and clinical efficiency.

Wait times for psychiatry appointments were significantly longer in central NY and Long Island compared to NYC. This may reflect the relative scarcity of psychiatrists in these areas. Wait times for therapy appointments were influenced by clinic affiliation and region of the state, suggesting that community-based clinics are better equipped to respond more quickly to therapy services. As with psychiatry wait times, central NY had longer therapy wait times. Given the differential access to psychiatric treatment across regions, state initiatives such as those that focus on supporting primary care providers with consultation and training on child mental health treatments may be particularly important in these regions where access is more challenging.^27-29

There are several limitations to this study. Due to resource constraints, we randomly assigned clinics to be contacted only once, either in the spring or summer. Because the same clinics were not surveyed twice, this makes it difficult to determine whether seasonal differences in access were actually accounted for by season or by some difference in clinic characteristics among the surveyed clinics. However, when we compared characteristics of clinics that were contacted in the spring vs. summer, we found no differences in the measured clinic characteristics. This provided some assurance that differences in the availability of appointments and wait times for appointments may be attributable to season rather than key clinic characteristics examined. Another study limitation is our focus on only one mental health condition (depression), whereas the most common presenting problems for access to services are disruptive behavior disorders. The scenario that was presented was of an adolescent girl; it is possible that caller responses might have varied had the age or sex of the youth been different. There may be differences between real and simulated families. Additionally, the study is limited to one state with a strong mental health system. Further, NYS has significantly restructured the financing of mental health services over the past several years, creating stressors to clinic capacity that may limit the generalizability of these findings outside the state. While we found a relationship between participation in state-supported training and access, the study as designed does not allow us to make causal attributions. It is possible that differences in access were related to some other characteristics of clinics that make them more likely to be both high training adopters and participants in state-sponsored training. Nevertheless, this represents the first study of its kind to demonstrate the use of a simulated patient or mystery shopper approach within a state system that applied a rigorous sampling frame and structured vignette to measure access to care for youth with depression. It thus represents an important first step in helping states align themselves with the goals of equitable access of ACA and offers a methodological tool to help states provide more timely and effective care.

Childhood psychiatric disorders are common and costly to the individual, their family, and to society. These problems are associated with significant long-term morbidity and increased costs to the healthcare system by exacerbating physical illness and care utilization. As states continue to implement significant health system changes, methods to systematically measure the impact of policies on service delivery will be increasingly critical. This study represents an important step in demonstrating the feasibility and usefulness of a simulated patient approach for assessing access to mental health care. Aligning the “mystery shopper” case scenario with quality indicators such as Healthcare Effectiveness Data and Information Set (HEDIS) measures (e.g., follow-up for attention-deficit/hyperactivity disorder medication and psychiatric hospitalization) could be an important next step in utilizing this methodology to assess care quality. Importantly, this study suggests that targeted trainings may influence treatment access in clinics and could be an important actionable tool for states to improve care quality for youth with psychiatric disorders more broadly.