The results of development and field testing of 8 new quality measures used to assess delivery of pediatric mental health care in the hospital setting (ED and inpatient).
Abstract
BACKGROUND AND OBJECTIVES:
Patients with a primary mental health condition account for nearly 10% of pediatric hospitalizations nationally, but little is known about the quality of care provided for them in hospital settings. Our objective was to develop and test medical record–based measures used to assess quality of pediatric mental health care in the emergency department (ED) and inpatient settings.
METHODS:
We drafted an evidence-based set of pediatric mental health care quality measures for the ED and inpatient settings. We used the modified Delphi method to prioritize measures; 2 ED and 6 inpatient measures were operationalized and field-tested in 2 community and 3 children’s hospitals. Eligible patients were 5 to 19 years old and diagnosed with psychosis, suicidality, or substance use from January 2012 to December 2013. We used bivariate and multivariate models to examine measure performance by patient characteristics and by hospital.
RESULTS:
Eight hundred and seventeen records were abstracted with primary diagnoses of suicidality (n = 446), psychosis (n = 321), and substance use (n = 50). Performance varied across measures. Among patients with suicidality, male patients (adjusted odds ratio: 0.27, P < .001) and African American patients (adjusted odds ratio: 0.31, P = .02) were less likely to have documentation of caregiver counseling on lethal means restriction. Among admitted suicidal patients, 27% had documentation of communication with an outside provider, with variation across hospitals (0%–38%; P < .001). There was low overall performance on screening for comorbid substance abuse in ED patients with psychosis (mean: 30.3).
CONCLUSIONS:
These new pediatric mental health care quality measures were used to identify sex and race disparities and substantial hospital variation. These measures may be useful for assessing and improving hospital-based pediatric mental health care quality.
What’s Known on This Subject:
Pediatric mental illness is a substantial public health issue with >4 million United States youth meeting mental health diagnostic criteria. High priority conditions are suicidality, psychosis, and substance use. There is a dearth of measures used to assess pediatric mental health care quality.
What This Study Adds:
New measures of pediatric mental health care quality are feasible to implement and demonstrate substantial variation across hospitals, with some measures varying by race and sex. These measures may be useful for assessing and improving hospital-based pediatric mental health care quality.
Pediatric mental illness is a substantial public health issue in both community and hospital settings. Approximately 20% of youth in the United States (>4 million) meet diagnostic criteria for a mental health disorder,1–3 and nearly 10% of hospitalizations in patients 3 to 17 years old were for primary mental health diagnoses in 2012.4 Inpatient and outpatient costs of treating these patients are estimated at $247 billion annually,2,3 and an increasing prevalence of mental health diagnoses and increasing service use in this population has been pointed to in evidence.5–8 Hospitalizations among pediatric patients with comorbid mental health diagnoses increased in children’s hospitals by 160% from 2005 to 2014,6 with comorbid mental health diagnoses leading to increased length of stay and cost.8 In recognition of this burden, recent federal health policies have identified pediatric mental health care as a key target area for quality measurement and improvement.9,10
In March 2011, the Centers for Medicare and Medicaid Services and the Agency for Healthcare Research and Quality partnered to fund 7 Centers of Excellence that constitute the Pediatric Quality Measures Program mandated by the 2009 Child Health Insurance Program Reauthorization Act.11,12 The charge to the Pediatric Quality Measures Program was to develop new quality of care measures and/or enhance existing measures for children’s health care across the age spectrum.11,12 The Center of Excellence on Quality of Care Measures for Children with Complex Needs (COE4CCN) was charged with developing measures to assess the quality of pediatric mental health care in both inpatient and emergency department (ED) settings. One pediatric quality measure of hospital-based mental health care had national endorsement at the time (outpatient follow-up after mental health hospitalization).13 Only 8 other measures for pediatric mental health care were nationally endorsed, and all were focused on outpatient care, with 5 of those focused on attention-deficit/hyperactivity disorder or developmental screening.13
Our objectives for this study were to develop a new evidence-based set of pediatric mental health care quality measures for use with medical records data and to field test the new measures in 5 nonpsychiatric hospitals providing mental health care to pediatric patients.
Methods
The National Quality Forum is a multistakeholder body tasked by the Centers for Medicare and Medicaid Services to review and endorse quality measures for potential Medicare and Medicaid use. Their criteria for endorsement, including whether the measure is high priority (important population or condition), evidence-based, valid, and has a demonstrated performance gap (provider low performance, variation, or disparities in performance across populations),14 were used to guide the study approach.
To develop and test these measures, we did the following: (1) determined common mental health diagnoses in pediatric visits to the ED and inpatient settings,4 (2) performed targeted evidence and clinical guideline reviews for the treatment and follow-up of the most prevalent conditions, (3) drafted pediatric mental health care quality measures on the basis of the evidence reviews, (4) convened a multistakeholder Delphi panel to prioritize the draft measures for further development, and (5) operationalized and field tested the Delphi panel–endorsed measures (Table 1) at 5 hospitals.
TABLE 1.
Measures of Pediatric Mental Health Quality in the Hospital Setting
| Measure Description | Measure Definition | Quality of Evidencea | Supporting Literature |
|---|---|---|---|
| Suicidality | |||
| Mental health assessment before ED discharge | Patients aged 5–19 years who present to the ED with dangerous self-harm or suicidality should have an assessment by a mental health professional before discharge from the ED | 5 | AACAP15 |
| Counseling to restrict access to lethal means before hospital discharge | Caregivers of patients aged 5–19 years who were admitted to the hospital for dangerous self-harm or suicidality should have documentation in the hospital record that they were counseled on how to restrict their child’s or adolescent’s access to potentially lethal means of suicide (eg, firearms, medications, car, etc) before discharge | 4 | Kruesi et al16 |
| 5 | AACAP15 | ||
| Hospital discharge communication with the outpatient provider | Patients aged 5–19 years admitted to the hospital for dangerous self-harm or suicidality should have documentation in the hospital record of discussion between the hospital provider and the patient’s outpatient provider regarding the plan for follow-up (discussion can be by phone or e-mail) | 5 | AACAP15 |
| Psychosis | |||
| Substance use comorbidity screening in the ED | Patients aged 5–19 years who present to the ED with psychotic symptoms should receive the following screening laboratory tests: urine drug screening, serum alcohol screening | 5 | AACAP, 200117 |
| 5 | Semper and McClellan18 | ||
| Timely inpatient consultation | Patients aged 5–19 years admitted to the hospital with psychotic symptoms should have a psychiatric consult (in person or via telepsychiatry) within 24 h of admission | 5 | AACAP17 |
| 2 | Gorrell, et al19 | ||
| Specialty input before starting antipsychotics | Patients aged 5–19 years admitted to the hospital with psychotic symptoms who are not currently taking antipsychotics should have a psychiatric consult (in person or via telepsychiatry) | 5 | AACAP17 |
| 2 | Gorrell, et al19 | ||
| Baseline metabolic screening before antipsychotics for psychosis | Patients aged 5–19 years admitted to the hospital with a diagnosis of psychotic disorder should have documentation in their medical record of a baseline metabolic assessment before starting any scheduled antipsychotic medication that includes the following: height, wt, blood pressure, pulse, blood glucose, total cholesterol, triglycerides, ECG | 5 | Hetrick et al20 |
| Substance use | |||
| Screening for mental health comorbidities in substance use hospitalizations | Patients aged 12–19 years admitted to the inpatient setting for substance use treatment should be screened for the following mental health comorbidities: depression, anxiety, history of abuse or witnessing violence | 5 | Bukstein et al21 |
| 4 | Deas-Nesmith et al22 | ||
| 4 | Langenbach et al23 | ||
AACAP, American Academy of Child and Adolescent Psychiatry.
Quality of Evidence Codes: (1) Randomized Controlled Trial, (2) cohort and outcome studies, (3) case-control studies, (4) case-series, (5) consensus, opinions, or “first principles” research.
Condition-Specific Quality Measure Development
On the basis of our previous findings that depression, psychosis, and substance use are the most common pediatric inpatient mental health diagnoses,4 we focused on suicidality, psychosis, and substance use for quality measure development. Suicidality may be present in depression or psychosis and has associated guidelines of care.15,24 Anxiety, a common outpatient diagnosis, was less common in the inpatient setting4 and was not chosen as a target condition for this quality measure development effort. We reviewed existing clinical practice guidelines and conducted targeted evidence reviews to identify best practices for the treatment, evaluation, and follow-up of pediatric suicidality, psychosis, and substance use. We used these reviews to guide condition-specific quality measure development. The validity and feasibility of the draft quality measures were then evaluated by a multistakeholder panel (psychology, psychiatry, family member, adolescent medicine, state Medicaid, hospitalist, ED) using the RAND–University of California, Los Angeles modified Delphi method25 (see Supplemental Information for descriptions of the literature reviews and the Delphi method). Measures rated favorably during that process were included for field testing (Table 1).
Measure Operationalization and Field Testing
Detailed measure specifications were used to develop an electronic medical record abstraction tool with automated scoring to ensure efficient, reliable, and feasible data collection (see Supplemental Information and online26).
Three children’s hospitals with inpatient psychiatric units participated in the field testing for ED and inpatient measures. They were tertiary care hospitals and not psychiatric specialty hospitals. Two community hospitals participated only in ED measures field testing because they did not have pediatric psychiatric inpatient units. The children’s hospitals were in different geographic regions of the country and had ∼13 000, ∼15 000, and ∼33 000 admissions in 2016, respectively; the 2 community hospitals were located in the same state but were operationally independent and had 8- and 12-bed pediatric units. All study procedures were approved by the participating institutions’ institutional review boards.
Eligible patients were 5 to 19 years old, and eligible adolescents for the Substance Use measure were 12 to 19 years old. Cases for the field test were selected by using International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) and Diagnostic and Statistical Manual, Fourth Edition, Text Revision codes for suicidality, psychosis, and substance use from each hospital’s administrative database of discharges between January 1, 2012, and December 31, 2013 (see Supplemental Tables 6 through 8 for ICD-9-CM codes). Transient psychosis (eg, confusional or delirious states) diagnoses were not included, unless drugs or alcohol were associated. Inpatients who were not discharged from the hospital were excluded from inpatient measures because the study team had no access to information about their subsequent care. All eligible patients from this time period were included in the final sample, with a goal of at least 200 patients per hospital over the 2-year time period.
After a 2-day training, 2 research staff nurses from each of the participating hospitals implemented the data abstraction tool. Each nurse abstracted half of their hospital’s medical record sample, with each chart abstraction taking ∼15 minutes. To assess interrater reliability, a randomly selected subsample of each nurse abstractor’s medical records was reabstracted by the other nurse at 2 of the children’s hospitals (other hospitals were not included because of limited funding). Prevalence and rater bias–adjusted κ statistics (PABAK) were calculated to examine reliability in assessing patient eligibility for each measure and reliability in measure scoring.27
For each condition, individual measure eligibility and scoring were reviewed to determine if the measure should be retained or dropped from the final quality measure set. Reasons for dropping measures fell into 1 of 4 broad categories: (1) eligibility for the measure was too rare for it to be a useful quality measure; (2) scores on the measure were extremely low (<25 on a 0–100 scale) across all hospitals indicating that, although the recommended process of care potentially represents high quality care, it had such low uptake that it should not yet be considered a standard of care; (3) interrater reliability was low, indicating a risk that performance variation might not be reliably and fairly measured; and (4) scores on the measure were extremely high (>90 on a 0–100 scale) across the hospitals, indicating there was little room for improvement. One exception to the last criterion was the decision to retain measures we hypothesized may demonstrate greater performance variability among a more representative sample of hospitals.
Assessing Disparities
For each measure, we assessed associations between performance and the following patient characteristics, drawn from administrative data from participating hospitals: sex, age, race and/or ethnicity, insurance type, and chronic disease status. To determine chronic disease status, we used the Pediatric Medical Complexity Algorithm (PMCA), which is used to categorize patients into 3 categories by using ICD-9-CM codes: patients with no chronic conditions, noncomplex chronic conditions, or complex chronic conditions.28 Data to assess PMCA were not available from the community hospitals’ administrative records.
Analysis
The detailed measure specifications26 were used to calculate quality measure scores. For individual-level binary measures (eg, mental health assessment in the ED), scores were 0 if absent (poor quality) and 100 if present (good quality). Subcomponents included in multicomponent measures (eg, height, blood pressure, blood glucose, etc, in the multicomponent quality measure, focused on metabolic assessment before initiating antipsychotics) were also scored by using this binary approach and then summarized to produce a mean composite score for the measure on a 0 to 100 scale. Hospital-level scores, summarizing both binary and multicomponent measures, ranged from 0 to 100, with higher scores indicating better quality.
Because some subcomponents in the multicomponent quality measures may be more challenging to pass than others (for instance, the subcomponent baseline electrocardiogram [ECG] was infrequently passed compared with the subcomponent of baseline weight documentation), we adjusted the overall measure score for each patient to account for the level of difficulty associated with passing each subcomponent. This “observed difficulty of delivery” (ODD) adjustment is performed by subtracting the grand mean population pass rate for each subcomponent from each patient’s score for that subcomponent.29,30 This allows a hospital’s performance to reflect success in harder-to-achieve subcomponents of the measure. The overall measure score for each patient is then calculated by averaging the ODD-adjusted subcomponent scores.
We assessed differences in measure performance by patient characteristics and by hospital using bivariate and multivariate regression analyses, using logistic regressions for dichotomous measures and linear regressions for continuous measures. To assess the statistical significance of hospital-level variation against the null hypothesis that all hospitals have the same mean measure scores, we used Fisher’s exact test for dichotomous measures (to avoid the questionable validity of the χ2 test if there are many expected cell counts <5) and analysis of variance for continuous measures. In analyses of hospital variation, we did not adjust for covariates because the measures are process measures rather than outcome measures (which are often risk adjusted). To assess differences by patient characteristics in multivariate analyses, we included predictor variables with a priori face validity (sex, age) and any additional variables with a statistically significant bivariate association with a given measure. Patients with missing data for 1 of the variables (Table 1) were excluded from bivariate analyses of that variable and from multivariable analyses. In the multivariate analyses, to assess associations with patient characteristics, we included a fixed-effect variable for hospitals to account for hospital-level systematic differences in care.
Results
Developing Condition-Specific Quality Measures
We drafted 21 measures on the basis of the literature and expert consensus guideline reviews: 8 for suicidality, 6 for psychosis, and 7 for substance use. In Delphi panel discussions, major themes and challenges that emerged included sparse evidence to inform the measures and potential difficulty in operationalizing measures (eg, defining documentation elements for counseling on lethal means restriction). Delphi scores indicated that 16 had sufficient face validity and feasibility to move forward to field testing. Of these 16, we report on 8 measures that underwent further testing (Table 1; see Supplemental Table 9 for measure-specific rationales for dropping measures from this measure set).
Field Testing
A total of 817 visits were analyzed across the 5 hospitals (n = 446 [55%] for suicidality, n = 321 [39%] for psychosis, n = 50 [6.0%] for substance use; 298 patients were seen in the ED only, n = 320 were seen in the ED and admitted to the hospital, and n = 199 were seen in the inpatient setting only, having been directly admitted). Most patients were teenagers (n = 745 [91%] 12–19-year-olds). There was some racial diversity, with 19% consisting of African American patients but more limited ethnic diversity (4% Hispanic). Insurance types were evenly distributed across private and public insurance. The majority of patients (86%) were seen at the children’s hospitals (Table 2).
TABLE 2.
Characteristics of Pediatric Patients With Mental Health Diagnoses and Hospitals for Measure Field Testing (N = 817 Patients)
| Characteristics | n | % |
|---|---|---|
| Patient characteristics | ||
| Sex | ||
| Male | 374 | 45.8 |
| Female | 438 | 53.6 |
| Missing | 5 | 0.6 |
| Age (y) | ||
| 5–11 | 72 | 8.8 |
| 12–15 | 412 | 50.4 |
| 16–19 | 333 | 40.8 |
| Condition | ||
| Suicidality | 446 | 54.6 |
| Psychosis | 321 | 39.3 |
| Substance use | 50 | 6.1 |
| Race and/or ethnicity | ||
| White | 488 | 59.7 |
| African American | 157 | 19.2 |
| Hispanic | 34 | 4.2 |
| Other | 116 | 14.2 |
| Missing | 22 | 2.7 |
| Insurance status | ||
| Private | 393 | 48.1 |
| Public or uninsured | 419 | 51.3 |
| Missing | 5 | 0.6 |
| PMCAa | ||
| Nonchronic | 147 | 20.9 |
| Noncomplex chronic | 357 | 50.6 |
| Complex chronic | 201 | 28.5 |
| Hospitals | ||
| Children’s hospital A | 301 | 36.8 |
| Children’s hospital B | 404 | 49.4 |
| Children’s hospital C | 87 | 10.6 |
| Community hospital D (ED only) | 12 | 1.5 |
| Community hospital E (ED only) | 13 | 1.6 |
Not all percentages add to 100% because of rounding.
PMCA data available for 2 children’s hospitals only (n = 705).
Overall performance on the 8 measures is summarized in Table 3. Performance ranged from a low of 27% for discharge communication with the outpatient provider before discharge for inpatients with suicidality to a high of 95% for mental health assessment for patients with suicidality before discharge from the ED. Performance was relatively low for the multicomponent “baseline metabolic testing before starting antipsychotic medications” for patients with psychosis (mean: 69.6), which was driven by 4 of the 8 elements: obtaining blood glucose (61.7%), cholesterol (48.7%), triglycerides (48.7%), and an ECG (20.0%).
TABLE 3.
Measure Performance Overall and Variation by Patient Characteristics and by Hospital, Bivariate Analyses (N = 817)
| Suicidality (n = 446) | Psychosis (n = 321) | Substance Use (n = 50) | ||||||
|---|---|---|---|---|---|---|---|---|
| Mental Health Assessment (ED), % | Counseling on Lethal Means Restriction (Inpatient), % | Discharge Communication With Outpatient Provider (Inpatient), % | Substance Use Screening (ED)a, Mean (SD) | Timely Consultation (Inpatient), % | Specialty Input Before Starting Medication (Inpatient), % | Baseline Metabolic Testing Before Medications (Inpatient)b, Mean (SD) | Assessing for Comorbid Mental Health (Inpatient)b, Mean (SD) | |
| n = 235 | n = 215 | n = 177 | n = 208 | n = 252 | n = 116 | n = 115 | n = 50 | |
| Overall performance | 95 | 83 | 27 | 30.3 (33.6) | 85 | 92 | 69.6 (19.3) | 86.7 (26.9) |
| Patient characteristics | ||||||||
| Sex | ||||||||
| Male | 94 | 71c | 30 | 32.3 (36.2) | 85 | 92 | 69.2 (19.1) | 77.8 (33.6)d |
| Female (ref) | 96 | 89 | 26 | 26.9 (29.7) | 85 | 93 | 70.3 (20.0) | 94.9 (15.5) |
| Age, y | ||||||||
| 5–11 | 95 | 93 | 33 | —e | 90 | 100f | 60.4 (21.5) | —e |
| 12–15 (ref) | 97 | 83 | 21 | 31.7 (32.2) | 84 | 87 | 73.6 (16.4) | 87.0 (28.0) |
| 16–19 | 93 | 82 | 34 | 29.0 (35.1) | 84 | 96 | 66.4 (21.1) | 86.4 (26.6) |
| Race and/or ethnicity | ||||||||
| White (ref) | 95 | 86 | 28 | 27.5 (32.9) | 81 | 96 | 70.2 (18.9) | 88.6 (25.5) |
| Hispanic | 94 | 75 | 14 | 16.7 (28.9) | 83 | —e | —e | 100.0 (0.0) |
| African American | 97 | 69d | 32 | 25.8 (28.4) | 88 | 92 | 65.0 (20.3) | 75.0 (38.8) |
| Other | 95 | 88 | 24 | 44.4 (40.0)d | 91 | 89 | 72.2 (17.5) | 83.3 (23.6) |
| Insurance | ||||||||
| Private (ref) | 96 | 88 | 28 | 32.8 (36.4) | 85 | 93 | 70.6 (18.7) | 87.5 (30.8) |
| Public or uninsured | 94 | 79 | 26 | 28.2 (31.8) | 84 | 92 | 69.0 (19.8) | 85.9 (23.4) |
| PMCAg | ||||||||
| Nonchronic (ref) | 100f | 75 | 14 | 19.3 (26.9) | 83 | 92 | 67.3 (24.0) | 66.7 (37.3) |
| Noncomplex chronic | 95h | 80 | 29 | 29.3 (26.7) | 86 | 86 | 70.7 (16.8) | 81.8 (31.1) |
| Complex chronic | 88 | 89 | 36 | 17.8 (24.1) | 80 | 96 | 70.7 (18.0) | 83.3 (33.3) |
| Hospitali | P = .15 | P = .70 | P = .002 | P < .001 | P = .001 | P = .23 | P = .31 | P = .01 |
| Children’s hospital A | 95 | 81 | 38 | 26.5 (30.7) | 95 | 95 | 66.1 (21.5) | 88.9 (17.2) |
| Children’s hospital B (ref) | 96 | 84 | 23 | 20.6 (24.7) | 76 | 88 | 72.2 (17.2) | 72.2 (36.6) |
| Children’s hospital C | —e | 87 | 0 | 87.5 (28.9) | 94 | 100f | 68.4 (20.6) | 96.2 (14.4) |
| Community hospital D (ED only) | —j | —e | —e | 44. (30.1) | —e | —e | —e | —e |
| Community hospital E (ED only | —e | —e | —e | 61.5 (41.6) | —e | —e | —e | —e |
—, not applicable.
ODD adjusted and restricted to ages 12–19 y as specified.
ODD adjusted.
P < .01.
P < .05.
No eligible patients for this group.
Not included in hypothesis testing because of no variation in category (performance either 100 or 0).
PMCA only available for 2 children’s hospitals.
Used as a reference group for hypothesis testing for this measure.
P values in this line indicate results of omnibus testing for variation in scores across hospitals.
Not included because of only 3 eligible cases.
For the 8 measures retained in the set, interrater reliability scores ranged from almost perfect to substantial27 on 2 levels: the patient’s eligibility for the measure (PABAK = 0.99) and the child’s score for that measure (PABAK = 0.76).
In bivariate analyses, performance did not vary substantially across patient characteristics, with a few notable exceptions. In the inpatient setting, caregivers of male patients and African American patients with suicidality were less likely to have received counseling before discharge about lethal means restriction (71.0% for male patients versus 89.0% for female patients [P = .001]; 68.8% for African American patients compared with 85.7% for non-Hispanic white patients [P = .03]). Male patients admitted with substance use were less likely to have been assessed for comorbid mental health diagnoses (mean: 77.8 vs 94.9 for female patients, [P = .02]). When assessing psychosis in the ED, those of other race were more likely to be screened for comorbid substance use (44.4 vs 27.5 for non-Hispanic white patients [P = .02]). Neither medical complexity nor insurance status was associated with performance on any measure (Table 3).
There was statistically significant variation across hospitals for the following: (1) whether patients hospitalized with psychosis were screened for comorbid substance use, (2) whether patients hospitalized with psychosis received timely psychiatric evaluation, and (3) whether patients hospitalized with substance use were screened for comorbid mental health conditions.
In multivariate analyses used to assess disparities by patient characteristics, differential performance persisted for male patients compared with female patients (odds ratio [OR]: 0.27 [95% confidence interval (CI): 0.12 to 0.58, P < .001]) and African American patients compared with white patients (OR: 0.31 [95% CI: 0.12 to 0.83, P = .02]) on counseling parents of those with suicidality on lethal means restriction, and differential performance also persisted for male patients on screening those admitted for substance use for other mental health conditions (coefficient: −20.0 [95% CI: −34.2 to −5.8, P = .007]). In addition, patients aged 16 to 19 years with suicidality were more likely to have documentation of communication between the inpatient and outpatient provider, compared with 12- to 15-year-olds (OR: 2.21 [95% CI: 1.05 to 4.65, P = .04]) (Table 4).
TABLE 4.
Assessment of Measure Disparities Across Patient Characteristics, Multivariate Analyses (N = 817)
| Suicidality | Psychosis | Substance Use | ||||||
|---|---|---|---|---|---|---|---|---|
| Mental Health Assessment (ED), OR (95% CI) | Counseling on Lethal Means Restriction (Inpatient), OR (95% CI) | Discharge Communication With Outpatient Provider (Inpatient), OR (95% CI) | Substance Use Screening (ED)a, Coefficient (95% CI) | Timely Consultation (Inpatient), OR (95% CI) | Specialty Input Before Starting Medication (Inpatient), OR (95% CI) | Baseline Metabolic Testing Before Medications (Inpatient)b, Coefficient (95% CI) | Assessing for Comorbid Mental Health (Inpatient)b, Coefficient (95% CI) | |
| n = 235 | n = 211 | n = 156 | n = 201 | n = 251 | n = 92 | n = 115 | n = 50 | |
| Patient characteristics | ||||||||
| Sex | ||||||||
| Malec | 0.59 (0.16 to 2.19) | 0.27 (0.12 to 0.58)d | 1.42 (0.66 to 3.05) | 4.36 (−3.83 to 12.54) | 0.83 (0.39 to 1.75) | 0.84 (0.19 to 3.72) | −0.80 (−8.30 to 6.70) | −19.98 (−34.18 to −5.77)e |
| Age, y | ||||||||
| 5–11c | 0.58 (0.10 to 3.37) | 3.99 (0.45 to 35.58) | 2.22 (0.55 to 8.94) | —f | 1.93 (0.40 to 9.31) | —g | −14.01 (−30.49 to 2.48) | —f |
| 16–19c | 0.45 (0.11 to 1.87) | 0.80 (0.36 to 1.79) | 2.21 (1.05 to 4.65)h | −5.95 (−14.08 to 2.19) | 0.63 (0.29 to 1.35) | 2.99 (0.53 to 16.85) | −5.62 (−13.41 to 2.16) | 6.81 (−7.87 to 21.49) |
| Race | ||||||||
| Hispanicc | —f | 0.36 (0.06 to 2.13) | —f | −1.32 (−33.89 to 31.26) | —f | —f | —f | —f |
| African Americanc | —f | 0.31 (0.12 to 0.83)h | —f | 0.52 (−8.50 to 9.53) | —f | —f | —f | —f |
| Otherc | —f | 1.26 (0.33 to 4.83) | —f | 8.90 (−4.10 to 21.90) | —f | —f | —f | —f |
Models all also included a fixed effect variable for hospitals to account for hospital-level systematic differences in care. —, not applicable.
ODD adjusted and restricted to ages 12–19 y as specified.
ODD adjusted.
Reference groups: girls, ages 12–15 y, white.
P < .001.
P < .01.
Variable not included for this measure because relationships were not statistically significant in the bivariate analysis.
Not included in hypothesis testing because of no variation in category (performance either 100 or 0).
P < .05.
Discussion
We present the results from developing and testing a new set of medical record–based pediatric mental health care quality measures in the hospital setting, focusing on suicidality, psychosis, and substance use. We developed evidence-based measures with face validity, clear performance gaps, demonstrable variation across providers, and disparities between populations. We discuss key findings in these areas below and their implications, limitations, and next steps.
Performance Gap Assessment
Most of the measures had performance lower than 90% (or 90 for nonbinary measures), implying that there is room for improvement in these evidence-based care processes. Two measures had particularly low performance: discharge communication with outpatient providers for patients with suicidality (27.1%) and substance use screening for patients with psychosis in the ED (mean: 30.3). This low performance overall, regardless of whether there is site-to-site variation, indicates the potential for substantial increases with quality improvement (QI) efforts.
There was relatively low performance (mean: 69.6) on performing baseline metabolic testing before starting a new antipsychotic medication for patients admitted for psychosis. In this multicomponent measure, there were 4 elements with particularly low performance rates: obtaining glucose, cholesterol, triglycerides, and an ECG. Youth treated with atypical antipsychotics are known to have increased risk of metabolic syndrome, arrhythmias, and severe weight gain.31,32 Higher performance on these measure subelements will potentially improve our ability to track and address downstream effects of these medications on cardiovascular health.
Mental health assessment in the ED for patients with suicidality was high (95.7%), without substantial site-to-site variation, but it is possible that the participating children’s hospitals had better access to pediatric psychiatric consultative services than community hospitals serving youth. With >70% of children’s hospitalizations occurring in community hospitals nationwide,33 it will be important to assess performance at other community-based EDs, where mental health resources may be more limited.
Finally, there were 3 measures not retained because of low performance, despite support from the Delphi panel and in the literature: alcohol abuse or dependence formal screening for patients presenting for interpersonal violence in the ED (performance of 0 for all hospitals) and for patients presenting with suicidality in the ED (performance of 0 for all hospitals) and face-to-face counseling and referral for patients who screen positive for substance abuse in the ED (performance of 12%–25% of patients across hospitals). These represent clear gaps in quality and could be areas for enhanced psychiatric consultation and improved collaborative care models.
Performance Variation Across Hospitals
For 3 measures (Substance Use Screening for Patients With Psychosis in the ED, Timely Mental Health Consultation for inpatients With Psychosis, and Assessing for Comorbid Mental Health Diagnoses for inpatients With Substance Use), performance varied across sites. This implies practice variations, with a potential for identifiable best practices from high performers.34,35 Future researchers can assess whether QI collaborations can facilitate improved performance and decrease variation across sites.
Disparities by Patient Characteristics
For most measures, performance did not vary in multivariate models by patient characteristics, indicating a lack of strong evidence of disparities in care across specific patient populations. However, for male patients and for African American patients with suicidality, there were lower odds of counseling caregivers on lethal means restriction compared with female patients and white patients. In contrast to our findings, in a previous study by Kruesi et al,16 they did not find differences in this counseling by sex or race, although they only compared white to non-white patients and had a smaller sample size of 100, limiting their power to detect such differences.
Data from other sites, gathered in further testing of this measure, could be used to better characterize this potential disparity and suggest QI efforts to address it. Performance on this measure is particularly important in light of recent work by Runyan et al,36 who found that an ED-based counseling intervention led to substantial improvements in families securing guns (increasing from 67% to 100%) and medications (increasing from 10% to 76%) and work by Scott et al,37 who showed that 43.5% of families with children who had a history of self-harm risk factors had household firearms, with 11.6% of those stored unlocked and loaded.
In our study, we also found that male patients admitted for substance use were less likely to be screened for comorbid mental health diagnoses, compared with female patients. This practice variation may reflect a common perception that female patients have a higher incidence of mental health diagnoses than male patients. However, this screening is a universal recommendation from the American Academy of Child and Adolescent Psychiatry.21 Given the high co-occurrence of substance use and comorbid mental health diagnoses,22,23 with our findings, we suggest that measuring and reporting on this quality measure could meaningfully address this disparity.
Our findings should be interpreted in light of several limitations. In this first field testing of these measures, we assessed performance in a limited number of hospitals. Having established feasibility of implementation, subsequent testing in more children’s and community hospitals will better characterize generalizability and variations across a larger set of hospitals. Also, the number of patients admitted for substance use was low. This likely reflects that children’s hospitals often do not have inpatient substance use services and that patients with substance use are generally admitted to specialized rehabilitation centers. We did not test for predictive validity, which should be done in future studies of these measures to assess whether better performance on them predicts decreased subsequent hospital-based use and costs of care. Finally, although they represent a national multistakeholder consensus and have undergone external review,38 the National Quality Forum endorsement criteria have not been tested to assess whether they lead to measures that support improved health outcomes in the population.
Conclusions
We present the results of the development and testing of a new set of medical record–based measures to assess pediatric mental health care quality in the hospital setting. We focused on high priority populations, those with suicidality, psychosis, and substance use, and identified measures with a demonstrated performance gap and variations in performance across hospitals or disparities across patient populations. In our findings, it is suggested that these measures may be useful for assessing and improving hospital-based pediatric mental health care quality for a vulnerable and high-priority population.
Glossary
- CI
confidence interval
- COE4CCN
Center of Excellence on Quality of Care Measures for Children with Complex Needs
- ECG
electrocardiogram
- ED
emergency department
- ICD-9-CM
International Classification of Diseases, Ninth Revision, Clinical Modification
- ODD
observed difficulty of delivery
- OR
odds ratio
- PABAK
prevalence and rater bias–adjusted κ statistics
- PMCA
Pediatric Medical Complexity Algorithm
- QI
quality improvement
Footnotes
Dr Bardach contributed to study design, interpretation of results, and drafted the initial manuscript; Mr Burkhart and Dr Parast conducted the statistical analyses, assisted with interpretation of results, and reviewed and revised the manuscript; Drs Richardson, Murphy, and Gidengil contributed to study design, interpretation of results, and reviewed and revised the manuscript; Ms Roth designed the data collection instruments, trained data abstractors, provided abstractor guidance during data collection at all 4 sites, and critically reviewed the manuscript; Dr Marmet contributed to data collection design, coordinated and supervised data collection at 1 of the 4 sites, and critically reviewed the manuscript; Drs Britto and Mangione-Smith contributed to study conceptualization and design, interpretation of results, and critically reviewed and revised the manuscript; and all authors approved the final manuscript as submitted.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Funded by a cooperative agreement with the Agency for Healthcare Research and Quality and Centers for Medicare and Medicaid Services, grant U18HS020506, part of the Children’s Health Insurance Program Reauthorization Act Pediatric Quality Measures Program. Dr Bardach was funded by the National Institute of Child Health and Human Development (K23HD065836). Funded by the National Institutes of Health (NIH).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
References
- 1.Shaffer D, Fisher P, Dulcan MK, et al. . The NIMH Diagnostic Interview Schedule for Children version 2.3 (DISC-2.3): description, acceptability, prevalence rates, and performance in the MECA study. Methods for the Epidemiology of Child and Adolescent Mental Disorders Study. J Am Acad Child Adolesc Psychiatry. 1996;35(7):865–877 [DOI] [PubMed] [Google Scholar]
- 2.Costly mental disorders affect millions of US children and teens: news from the centers for disease control and prevention. JAMA. 2013;310(1):23 [Google Scholar]
- 3.Perou R, Bitsko RH, Blumberg SJ, et al. ; Centers for Disease Control and Prevention (CDC) . Mental health surveillance among children–United States, 2005-2011. MMWR Suppl. 2013;62(2):1–35 [PubMed] [Google Scholar]
- 4.Bardach NS, Coker TR, Zima BT, et al. . Common and costly hospitalizations for pediatric mental health disorders. Pediatrics. 2014;133(4):602–609 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Health Care Cost Institute Children’s health care spending report: 2007–2010. 2012. Available at: www.healthcostinstitute.org/files/HCCI_CHCSR20072010.pdf. Accessed May 29, 2013
- 6.Zima BT, Rodean J, Hall M, Bardach NS, Coker TR, Berry JG. Psychiatric disorders and trends in resource use in pediatric hospitals. Pediatrics. 2016;138(5):e20160909. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Pfuntner A, Wier LM, Stocks C Most frequent conditions in US hospitals, 2010. HCUP Statistical Brief #148. 2013. Available at: www.hcup-us.ahrq.gov/reports/statbriefs/sb148.pdf. Accessed September 20, 2017
- 8.Doupnik SK, Lawlor J, Zima BT, et al. . Mental health conditions and medical and surgical hospital utilization. Pediatrics. 2016;138(6):e20162416. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Pallone F. Children’s Health Insurance Program Reauthorization Act of 2009. Office of the Legislative Council 111th Congress. Public Law No. 111–113
- 10.Strokoff SL. Patient Protection and Affordable Care Act of 2010. Office of the Legislative Council 111th Congress. Public Law No. 111–148: 288–291
- 11.Dougherty D, Schiff J, Mangione-Smith R. The Children’s Health Insurance Program Reauthorization Act quality measures initiatives: moving forward to improve measurement, care, and child and adolescent outcomes. Acad Pediatr. 2011;11(suppl 3):S1–S10 [DOI] [PubMed] [Google Scholar]
- 12.Byron SC, Gardner W, Kleinman LC, et al. . Developing measures for pediatric quality: methods and experiences of the CHIPRA pediatric quality measures program grantees. Acad Pediatr. 2014;14(suppl 5):S27–S32 [DOI] [PubMed] [Google Scholar]
- 13.Zima BT, Murphy JM, Scholle SH, et al. . National quality measures for child mental health care: background, progress, and next steps. Pediatrics. 2013;131(suppl 1):S38–S49 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.NQF Measure evaluation criteria and guidance for evaluating measures for endorsement. 2015. Available at: http://www.qualityforum.org/Measuring_Performance/Submitting_Standards/Measure_Evaluation_Criteria.aspx. Accessed September 2, 2015
- 15.American Academy of Child and Adolescent Psychiatry Practice parameter for the assessment and treatment of children and adolescents with suicidal behavior. American Academy of Child and Adolescent Psychiatry. J Am Acad Child Adolesc Psychiatry. 2001;40(suppl 7):S24–S51 [DOI] [PubMed] [Google Scholar]
- 16.Kruesi MJ, Grossman J, Pennington JM, Woodward PJ, Duda D, Hirsch JG. Suicide and violence prevention: parent education in the emergency department. J Am Acad Child Adolesc Psychiatry. 1999;38(3):250–255 [DOI] [PubMed] [Google Scholar]
- 17.McClellan J, Werry J, Bernet W, et al. ; American Academy of Child and Adolescent Psychiatry . Practice parameter for the assessment and treatment of children and adolescents with schizophrenia. J Am Acad Child Adolesc Psychiatry. 2001;40(suppl 7):S4–S2311434484 [Google Scholar]
- 18.Semper TF, McClellan JM. The psychotic child. Child Adolesc Psychiatr Clin N Am. 2003;12(4):679–691 [DOI] [PubMed] [Google Scholar]
- 19.Gorrell J, Cornish A, Tennant C, et al. . Changes in early psychosis service provision: a file audit. Aust N Z J Psychiatry. 2004;38(9):687–693 [DOI] [PubMed] [Google Scholar]
- 20.Hetrick S, Alvarez-Jiménez M, Parker A, et al. . Promoting physical health in youth mental health services: ensuring routine monitoring of weight and metabolic indices in a first episode psychosis clinic. Australas Psychiatry. 2010;18(5):451–455 [DOI] [PubMed] [Google Scholar]
- 21.Bukstein OG, Bernet W, Arnold V, et al. ; Work Group on Quality Issues . Practice parameter for the assessment and treatment of children and adolescents with substance use disorders. J Am Acad Child Adolesc Psychiatry. 2005;44(6):609–621 [DOI] [PubMed] [Google Scholar]
- 22.Deas-Nesmith D, Campbell S, Brady KT. Substance use disorders in an adolescent inpatient psychiatric population. J Natl Med Assoc. 1998;90(4):233–238 [PMC free article] [PubMed] [Google Scholar]
- 23.Langenbach T, Spönlein A, Overfeld E, et al. . Axis I comorbidity in adolescent inpatients referred for treatment of substance use disorders. Child Adolesc Psychiatry Ment Health. 2010;4:25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Birmaher B, Brent D, Bernet W, et al. ; AACAP Work Group on Quality Issues . Practice parameter for the assessment and treatment of children and adolescents with depressive disorders. J Am Acad Child Adolesc Psychiatry. 2007;46(11):1503–1526 [DOI] [PubMed] [Google Scholar]
- 25.McCormick KA, Moore SR, Siegel RA, eds. Clinical Practice Guideline Development: Methodology Perspectives. Rockville, MD: U.S. Department of Health and Human Services, Public Health Service, Agency for Health Care Policy and Research; 1994 [Google Scholar]
- 26.Seattle Children’s Research Institute Mangione-Smith lab measurement tools. Available at: www.seattlechildrens.org/research/child-health-behavior-and-development/mangione-smith-lab/measurement-tools/. Accessed February 16, 2017
- 27.Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005;85(3):257–268 [PubMed] [Google Scholar]
- 28.Simon TD, Cawthon ML, Stanford S, et al. ; Center of Excellence on Quality of Care Measures for Children With Complex Needs (COE4CCN) Medical Complexity Working Group . Pediatric medical complexity algorithm: a new method to stratify children by medical complexity. Pediatrics. 2014;133(6). Available at: www.pediatrics.org/cgi/content/full/133/6/e1647 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Min LC, Wenger NS, Fung C, et al. . Multimorbidity is associated with better quality of care among vulnerable elders. Med Care. 2007;45(6):480–488 [DOI] [PubMed] [Google Scholar]
- 30.Reid RO, Friedberg MW, Adams JL, McGlynn EA, Mehrotra A. Associations between physician characteristics and quality of care. Arch Intern Med. 2010;170(16):1442–1449 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Riordan HJ, Antonini P, Murphy MF. Atypical antipsychotics and metabolic syndrome in patients with schizophrenia: risk factors, monitoring, and healthcare implications. Am Health Drug Benefits. 2011;4(5):292–302 [PMC free article] [PubMed] [Google Scholar]
- 32.Newcomer JW. Second-generation (atypical) antipsychotics and metabolic effects: a comprehensive literature review. CNS Drugs. 2005;19(suppl 1):1–93 [DOI] [PubMed] [Google Scholar]
- 33.Leyenaar JK, Ralston SL, Shieh MS, Pekow PS, Mangione-Smith R, Lindenauer PK. Epidemiology of pediatric hospitalizations at general hospitals and freestanding children’s hospitals in the United States. J Hosp Med. 2016;11(11):743–749 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Bradley EH, Curry LA, Ramanadhan S, Rowe L, Nembhard IM, Krumholz HM. Research in action: using positive deviance to improve quality of health care. Implement Sci. 2009;4:25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Curry LA, Spatz E, Cherlin E, et al. . What distinguishes top-performing hospitals in acute myocardial infarction mortality rates? A qualitative study. Ann Intern Med. 2011;154(6):384–390 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Runyan CW, Becker A, Brandspigel S, Barber C, Trudeau A, Novins D. Lethal means counseling for parents of youth seeking emergency care for suicidality. West J Emerg Med. 2016;17(1):8–14 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Scott J, Azrael D, Miller M. Firearm storage in homes with children with self-harm risk factors. Pediatrics. 2018;141(3):e20172660. [DOI] [PubMed] [Google Scholar]
- 38.Gold M, Conwell L, Stewart K, Nysenbaum J, Peterson S Assessment of the National Quality Forum’s consensus development process. 2010. Available at: https://www.mathematica-mpr.com/download-media?MediaItemId=%7B2F3BFC49-38BD-4880-B8C4-032134BDF316%7D. Accessed February 23, 2018