Validation of New Care Coordination Quality Measures for Children with Medical Complexity

Abstract

Objectives:

To validate new caregiver-reported quality measures assessing care coordination services for children with medical complexity (CMC).

Methods:

A cross sectional analysis of the associations between 20 newly developed Family Experiences with Coordination of Care (FECC) quality measures and three validation measures among 1209 caregivers who responded to a telephone or mailed survey between August-November 2013 in Minnesota and Washington. Validation measures included an access composite, a provider rating item, and a care coordination outcome measure, all derived from Consumer Assessments of Healthcare Providers and Systems (CAHPS^©) survey items. Multivariate regression was used to examine associations between the three validation measures and each of the 20 FECC quality measures.

Results:

Nineteen of the twenty FECC quality measures were significantly and positively associated with one or more of the validation measures. The components of care coordination demonstrating the strongest positive association with provider ratings included (1) having a care coordinator who was knowledgeable, supportive and advocated for the child’s needs (β =26.4; 95% Confidence Interval [CI] 20.0,32.8, scaled to reflect change associated with a 0 to 100 change in the FECC measure score) and (2) receiving a written visit summary that was useful and easy to understand (β =22.0; 95% CI 17.1,27.0).

Conclusions:

Nineteen newly developed FECC quality measures demonstrated convergent validity with previously validated CAHPS measures. These new measures are valid for assessing the quality of care coordination services provided to CMC and may be useful for evaluating new models of care focused on improving these services.

What’s New:

We demonstrate the validity of newly developed caregiver-reported quality measures to assess care coordination services provided to children with medical complexity(CMC). These measures may be useful for assessing the quality of care coordination for CMC and identifying areas for improvement.

Introduction

Children with medical complexity (CMC) account for approximately one-third of children’s healthcare expenditures, though they only comprise 1-5% of the pediatric population.^1,2 CMC often receive care from multiple providers; for example, Cohen et al (2012) found that the median number of distinct physicians providing outpatient care was 13 in a large sample of CMC over 2 years.² Given their high utilization among multiple providers, CMC would likely benefit from care coordination - the “deliberate organization of patient care activities between two or more participants (including the patient) involved in a patient's care to facilitate the appropriate delivery of health care services.”³ The development of quality measures to assess this aspect of care for CMC- arguably the children who stand to gain the most from coordinated care- has been identified as a need by the Centers for Medicare and Medicaid Services (CMS), Agency for Healthcare Research and Quality (AHRQ), and the National Quality Forum (NQF)^4-6

Importantly, prior care coordination quality measure development efforts have focused on the broader population of children with special healthcare needs (CSHCN), defined by the federal Maternal and Child Health Bureau as children who “have or are at increased risk for a chronic physical, developmental, behavioral, or emotional condition and who also require health and related services of a type or amount beyond that required by children generally”^7-10 However, specifically developing measures for CMC, who are a subset of CSHCN (estimated to be 28% of CSHCN¹¹), would facilitate quality improvement efforts targeting the care and services they are more likely to need. For example, in addition to often receiving care from multiple physicians, CMC tend to have high hospital and emergency department utilization.¹ CMC represent the most vulnerable children within this broader population and are those most likely to suffer serious consequences resulting from the receipt of poor quality of care.

With funding from AHRQ and CMS, the Center of Excellence on Quality of Care Measures for Children with Complex Needs (COE4CCN) developed twenty new caregiver-reported Family Experiences with Coordination of Care (FECC) measures to assess the quality of care coordination for CMC.¹² Measure development was guided by a conceptual framework that identified multiple steps in the care delivery process at risk for failures in care coordination.¹² Our objective with this study was to validate the newly developed FECC quality measures. We hypothesized that better performance on the FECC quality measures would be associated with higher scores on established measures of provider ratings, access to care, and care coordination outcomes.

Methods

A consensus definition of CMC was developed within the COE4CCN as a child with either (1) significant chronic conditions in two or more body systems, (2) a progressive condition that is associated with deteriorating health with a decreased life expectancy in adulthood, (3) continuous dependence on technology for at least 6 months, or (4) progressive or metastatic malignancies that affect life function.¹¹ This study was approved by the Institutional Review Boards at the RAND Corporation, Seattle Children’s Research Institute, and the Washington (WA) and Minnesota (MN) state Medicaid programs.

Measure Development

Details concerning the development of the FECC quality measures are described elsewhere¹²; we briefly summarize the development process here. The FECC quality measures for CMC examined in this paper were developed by the COE4CCN.¹² Briefly, the study team identified topics for evidence review based on a conceptual framework, summarized the evidence, wrote draft quality measure statements, and convened a multi-stakeholder Delphi panel to prioritize the draft measures. The Delphi panel evaluated and scored each measure on content validity and feasibility, on a scale from 1(low) to 9(high); two rounds of independent panel member scoring were conducted with a group discussion in between. Table 1 lists the 20 FECC quality measures endorsed by the Delphi panel.

Table 1.

Developed FECC Quality of Care Measures

CARE COORDINATION SERVICES
Short measure description	Measure full text
Has care coordinator	Caregivers of children with complex needs should report that their child has a designated care coordinator.
Access to care coordinator	Caregivers of children with complex needs who report that their child has a designated care coordinator should report that they know how to access their care coordinator.
Care coordinator helped to obtain community services	Caregivers of children with complex needs who report having a designated care coordinator and who require community services should also report that their care coordinator helped their child to obtain needed community services in the last year.
Care coordinator contact in the last 3 months	Caregivers of children with complex needs who report having a care coordinator should also report that their care coordinator has contacted them (via face-to-face contact, telephone, email, or written correspondence) or attempted to contact them at least once in the last 3 months.
Care coordinator asked about concerns and health changes	Caregivers of children with complex needs who report having a care coordinator and who report that their care coordinator has contacted them in the last 3 months should also report that their care coordinator asked them about: (1) Caregiver concerns, and (2) Health changes of the child
Care coordinator asked about progress towards goals	Caregivers of children with complex needs who report the following: having a designated care coordinator, having a copy of a written shared care plan for their child, and having been contacted by their care coordinator in the last 3 months should also report that their care coordinator asked them about progress towards goals documented in the patient's shared care plan
Care coordinator assisted with specialist service referrals	Caregivers of children with complex needs who report having a care coordinator for their child should also report that the care coordinator assists them with specialty service referrals by ensuring that the appointment with the specialty service provider occurs
Care coordinator was knowledgeable, supportive and advocated for child’s needs	Caregivers of children with complex needs who report having a care coordinator should also report that their care coordinator: (1) Was knowledgeable about their child’s health, (2) Supported the caregiver, and (3) Advocated for the needs of the child
Caregiver has access to medical interpreter when needed	Caregivers of children with complex needs or children with complex needs who self-identify as having a preference for conducting medical visits in a language other than English should have access at all visits to a professional medical interpreter (live or telephonic).
MESSAGING
Short measure description	Measure full text
Appropriate written visit summary content	Caregivers who report receiving a written visit summary during the last 12 months from their child’s main provider’s office should report that it contained the following elements: (1) Current problem list, (2) Current medication list, (3) Drug allergies, (4) Specialists involved in the child’s care, (5) Planned follow-up, (6) What to do for problems related to outpatient visit
Written visit summary was useful and easy to understand	Caregivers who reported ever receiving a written visit summary in the last 12 months from their child’s main provider’s office should report that the summary: (1) Was easy to understand, and (2) Was useful
Invited to join hospital rounds	Caregivers should report having been invited to join in hospital rounds during their child’s last hospitalization
Appropriate written hospitalization summary content	Caregivers should report receiving a written visit summary of their child’s last hospitalization at the time of discharge, and they should report the summary contained the following elements: (1) Problem list at time of discharge, (2) Medication list at time of discharge, (3) Drug allergies, (4) Specialists involved during the hospitalization, (5) Planned follow-up, and (6) What to do for problems related to the hospitalization
Written hospitalization summary was easy to understand	Caregivers who received a written summary of their child’s hospitalization at discharge should report that the information contained in the visit summary was easy to understand
Caregiver has access to electronic health record	Caregivers of children with complex needs should report having access to an electronic health record to look up information about their child’s visits and health care.
Electronic health record has immunization and medication information	Caregivers of children with complex needs who report having access to an electronic health record should also report that it includes the following health information: (1) Immunization record and (2) List of child’s medications
Health care provider communicated with school staff about child’s condition	Caregivers of children with complex needs who report their child’s condition causes difficulty learning, understanding, or paying attention in class should also report that one of their child’s health care providers (i.e., primary care physician, specialist physician, care coordinator, nurse practitioner, nurse, social worker, etc.) communicated with school staff at least once a year about the educational impacts of the child’s condition.
PROTOCOLS/PLANS
Short measure description	Measure full text
Child has shared care plan	Caregivers of children with complex needs should report that their child’s primary care provider created a shared care plan for their child.
Child has written transition plan	Caregivers of children (age 15 years or older) with complex needs should report that their child’s main provider created a written transition plan for their child.
Child has emergency care plan	Caregivers of children with complex needs should report that their child’s main provider created an emergency care plan for their child.

FECC Quality Measures

Details regarding the specific survey items that are used to score each quality measure and the scoring specifications are provided elsewhere.^12,13 We categorized the 20 measures into three content areas: care coordination services (8 measures), messaging (9 measures), and protocols/plans (3 measures). These categories do not represent survey sub-scales or domains; they are strictly an organizational structure for the 20 FECC quality measures. To assess performance on these quality measures, the survey items ask caregivers about care from their child’s main healthcare provider in the last 12 months, getting help to manage the child’s care, getting summaries of the child’s visits and hospitalizations, getting information in between visits, and care plans.

All of the FECC quality measures are scored on a 0-100 scale such that higher scores indicate better care; this scoring approach is consistent with the NQF-endorsed scoring of the measures (for the subset that were NQF endorsed in June 2016⁴) and is used for other established patient and caregiver experience measures in other settings, e.g., composite measures derived from the Clinician and Group (CG) Consumer Assessment of Healthcare Providers and Systems (CAHPS^®) Child 12-month Survey.¹⁴ Specifically, for dichotomous quality measures, a score of 100 indicates that the child had a recommended structure of care in place and/or received a recommended process of care and a score of 0 indicates that the child did not have a recommended structure in place and/or did not receive a recommended process of care. For example, for the measure assessing whether the child has a care coordinator, caregivers of children who visited more than one doctor’s office or used more than one kind of health care service in the past 12 months (eligible population) are asked: “Did anyone in the main provider’s office help you to manage your child’s care or treatment from different doctors or care providers?” and their response is scored such that “Yes”=100 and “No” = 0. For measures with ordinal response options, responses are linearly transformed to scores on a 0-100 scale. For example, for the measure assessing access to a medical interpreter when needed, eligible caregivers are asked “When you needed a professional interpreter during a visit to the main provider, how often was an interpreter available?” and their response is scored such that “Never”=0, “Sometimes” = 33.3”, “Usually” = 66.7, and “Always” = 100. Measures with multiple components were scored as the mean of the component item scores. For example, for the measure assessing whether a received written visit summary was useful and easy to understand, eligible caregivers are asked whether (1) the summary was useful with response options scored as “Never” = 0, “Sometimes”=50, “Always” = 100”, and (2) the summary was easy to understand with the same response options; the overall score for the measure was the average of these two scored components.

Survey Sample and Administration

We began by identifying CMC enrolled in Medicaid in Minnesota or Washington according to the Pediatric Medical Complexity Algorithm (PMCA).¹¹ The PMCA uses International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes to classify children into three levels of medical complexity: complex chronic disease (e.g. seizure disorder, developmental delay, and chronic lung disease), non-complex chronic disease (e.g. asthma), and no chronic disease (e.g. febrile seizures).¹¹ Children were eligible for inclusion if they were: (a) classified as having complex chronic disease based on the PMCA; (b) aged 0-17 years; (c) had at least two Medicaid eligibility months between October through December 2012 (the three months prior to obtaining the sample); and (d) had at least four visits to a health care provider in 2012. Note that children classified as having complex chronic disease based on the PMCA meet the consensus definition of CMC described above.¹¹ Children were excluded if: (a) the child had died; (b) the listed caregiver of the child was <18 years old; or (c) the caregiver’s primary language was other than English or Spanish. We then identified the caregivers of eligible children and sampled 3000 (1500 in each state). The survey was administered between August and November 2013 with caregivers randomized to either telephone-only or mixed mode (mail with telephone follow-up) modes of administration. The survey was available in English and Spanish.

Survey Instrument

The field test survey instrument (provided in the Supplementary Material) was composed of 103 items: 45 of these items were used to calculate the 20 FECC quality measures, 9 items assessed the validation measures described below, 8 items collected information on child and caregiver characteristics including their age, race/ethnicity, caregiver education, caregiver preferred language for medical care and the caregiver’s relationship to the child, and the remaining items were related to other aspects of the study. At the beginning of the survey the caregiver was asked to identify the child’s main provider, defined as “the doctor, physician assistant, nurse or other healthcare provider who knows the most about your child’s health, and who is in charge of your child's care overall.” Survey questions about the main provider referred to this provider and/or practice.

Validation Measures

Two validation measures were derived from the CG CAHPS^® Child 12-month Survey and the third validation measure was adapted from CAHPS^® supplemental items.¹⁵ All three validation measures were scored on a 0-100 scale using linear mean scoring, a scoring approach typically used for CAHPS measures.^14,15

The first validation measure, which assessed provider ratings, was obtained with the following survey item: “Using any number from 0 to 10, where 0 is the worst provider possible and 10 is the best provider possible, what number would you use to rate the main provider?” Provider rating responses were linearly transformed to a 0-100 scale.

The second validation measure assessed access to care and was constructed using a modified version of the CG CAHPS^® Getting Timely Appointments, Care, and Information composite¹⁶ composed of the following items:

• In the last 12 months, when you phoned the main provider's office to get an appointment for care your child needed right away, how often did you get an appointment as soon as your child needed?

• In the last 12 months, when you phoned the main provider's office during regular office hours, how often did you get an answer to your medical question that same day?

• In the last 12 months, when you made an appointment for a check-up or routine care for your child with the main provider, how often did you get an appointment as soon as your child needed?

• In the last 12 months, how often did your child see the main provider within 15 minutes of his or her appointment time?

A screening question was used for each of the first three questions to ensure that the questions were asked only of those caregivers who had tried to get a time-sensitive appointment, ask a question or make a regular appointment. Since the eligibility criteria included only children with at least four visits in 2012, a screening question was not required for the fourth item. Responses were scored on a 0-100 scale (Never = 0, Sometimes = 33.3, Usually = 66.7, Always = 100); all caregivers who answered at least one of the four questions included in the composite received a score calculated as the mean of the non-missing responses to the four questions.

The third validation measure, adapted from CAHPS^® supplemental items, was used to assess a care coordination outcome: “Overall, how often did the main provider’s office give you the help you needed to manage your child's care or treatment from different doctors or care providers in the last 12 months?”. Responses were scored on a 0-100 scale (Never = 0, Sometimes = 33.3, Usually = 66.7, Always = 100) with higher scores indicating better care coordination outcomes. Caregivers were only asked this question if they endorsed that their child had a designated care coordinator from the main provider’s office and had seen more than one health care provider or needed more than one kind of health care service during the last 12 months.

Analyses

We used linear regression models to examine the association between FECC quality measure scores (independent variable) and the three CAHPS^® validation measures (dependent variable). Models were adjusted for the following child and caregiver characteristics: state of residence, mode of survey administration, child age, caregiver gender, caregiver age, caregiver race/ethnicity, caregiver education, and caregiver relationship to child. Regression models employed complete case analysis; missing data were not imputed. Each quality measure was evaluated separately. Note that the sample size for each validation regression differs due to FECC measure eligibility and validation measure eligibility. For example, only caregivers reporting their child had a care coordinator were asked additional FECC measure questions about the care coordinator and the validation question about how often they received care coordination.

Results

From the 3000 sampled caregivers, 33 (1%) were deemed ineligible (i.e. met exclusion criteria or did not meet inclusion criteria for the sample) and 1209 responded to the survey (600 in WA, 609 in MN) for an overall response rate among eligible caregivers of 41%.Characteristics of the survey respondents and their children are shown in Table 2 and further results can be found elsewhere.¹² The vast majority (89%) of respondents were the child’s mother; more than half of respondents were non-Hispanic white.

Table 2.

FECC Survey Respondent Characteristics

	Respondents, N=1209 N(%)
Child characteristics
Child age
< 2 years	127 (11%)
2-5 years	270 (22%)
6-10 years	357 (30%)
11-13 years	207 (17%)
14-17 years	248 (21%)
Child race/ethnicity
Non-Hispanic White	585 (48%)
Hispanic	308 (26%)
African American	94 (8%)
Other	195 (16%)
Missing	27 (2%)
Caregiver characteristics
Gender
Female	1150 (95%)
Male	35 (3%)
Missing	24 (2%)
Caregiver relationship to child
Parent	1108 (92%)
Grandparent	42 (4%)
Aunt or uncle	5 (<1%)
Other relative	1 (<1%)
Legal guardian	21 (2%)
Other or Missing	32 (3%)
Caregiver age
18-24	60 (5%)
25-34	433 (36%)
35-44	417 (35%)
45-54	150 (12%)
55-64	41 (3%)
65-74	9 (<1%)
75+	3 (<1%)
Other, Unknown, not answered	96 (8%)
Caregiver race/ethnicity
Non-Hispanic White	722 (60%)
Hispanic	250 (21%)
African American	92 (8%)
Other	119 (10%)
Missing	26 (2%)
Caregiver education (highest level completed)
8th grade or less	70 (6%)
High school	435 (36%)
Some college or 4-year college	639 (53%)
degree	38 (3%)
More than 4-year college degree	27 (2%)
Not answered or don’t know
Caregiver English language proficiency
Speaks very well	972 (80%)
Speaks well	78 (6%)
Does not speak well	82 (7%)
Does not speak at all	52 (4%)
Not answered	25 (2%)
Language of survey completion
English	1048 (87%)
Spanish	161 (13%)
Mode of survey completion
Mail	434 (36%)
Telephone only	544 (45%)
Telephone following mailing	230 (19%)
State of residence
WA	600 (50%)
MN	609 (50%)

We briefly summarize the FECC measure results which are presented fully in Gidengil et al. (2017).¹² Seventy-two percent of caregivers reported that their child had a care coordinator. The measure with the highest score was the measure assessing access to the care coordinator: 96% of caregivers who reported that their child had a care coordinator reported knowing how to access the care coordinator. The lowest-performing measure was the measure assessing whether eligible adolescents had a written transition care plan: only 10% had such a plan.

Among eligible respondents, the average provider rating was 89.0 (standard deviation = 15.9) and the average access to care composite score was 78.0 (standard deviation = 20.9). The average score on the validation measure assessing caregiver reports of receiving needed help managing their child’s care was 82.2 (standard deviation = 26.4).

Higher scores on 19 of the FECC measures were positively associated with higher provider ratings, better access to care scores, and higher scores on the measure assessing caregiver reports of receiving needed help managing their child’s care (Table 3). Adjusted regression β-coefficients in Table 3 were scaled to reflect the change in the validation metric associated with a 0 to 100 change in the FECC quality measure score. For example, for the measure assessing whether the caregiver had access to a medical interpreter when needed, a score of 100 versus a score of 0 on this FECC measure was associated with a 27 point increase in the access to care composite score.

Table 3.

Validation of FECC quality measures

	Provider Rating (0-100)		Access Composite (0-100)		How often provider’s office gave needed help with coordinating care (0-100)
FECC Quality Measures	N	β1 (95%CI)	N	β1 (95%CI)	N	β1 (95%CI)
Care Coordination Services    Has care coordinator2	773	5.92 (3.46, 8.38)***	776	6.91 (3.72, 10.1)***	--3	--3
Access to care coordinator	560	7.72 (1.69, 13.74)*	561	10.43 (2.33, 18.53)*	560	33.8 (23.21, 44.39)***
Care coordinator helped to obtain community services2	251	5.55 (1.68, 9.43)**	251	6.24 (1.56, 10.91)**	250	7 (0.93, 13.07)*
Care coordinator contact in the last 3 months	554	4.69 (2.29, 7.1)***	555	6.72 (3.55, 9.88)***	554	11.36 (7.1, 15.61)***
Care coordinator asked about concerns and health changes2	246	14.3 (8.49, 20.11)***	246	28.84 (20.22, 37.45)***	246	35.44 (24.66, 46.23)***
Care coordinator asked about progress towards goals	101	9.56 (3.09, 16.02)**	101	7.91 (−1.48, 17.29)	101	11.12 (1.03, 21.22)*
Care coordinator assisted with specialist service referrals2	419	7.78 (4.87, 10.69)***	420	5.2 (1.19, 9.2)*	419	11.82 (6.75, 16.9)***
Care coordinator was knowledgeable, supportive and advocated for child’s needs2	517	26.36 (19.98, 32.75)***	517	21.56 (12.59, 30.54)***	516	56.38 (45.24, 67.53)***
Caregiver has access to medical interpreter when needed2	114	4.53 (−3.34, 12.41)	113	26.98 (7.82, 46.14)**	63	25.34 (−10.18, 60.86)
Messaging    Appropriate written visit summary content2	650	14.53 (9.54, 19.51)***	651	26.41 (18.91, 33.92)***	380	40.1 (28.3, 51.91)***
Written visit summary was useful and easy to understand	726	22.04 (17.07, 27.01)***	728	31.4 (23.85, 38.95)***	423	33.21 (20.92, 45.49)***
Invited to join hospital rounds	238	−0.42 (−4.5, 3.66)	240	1.15 (−3.75, 6.04)	149	5.71 (−2.72, 14.15)
Appropriate written hospitalization summary content	221	18.09 (5.08, 31.09)**	222	21.83 (6.6, 37.05)**	142	55.71 (25.48, 85.93)***
Written hospitalization summary was easy to understand	222	18.93 (8.32, 29.55)***	223	10.17 (−2.32, 22.66)	143	28.93 (8.03, 49.83)**
Caregiver has access to electronic health record	1090	3.47 (1.51, 5.43)***	1090	3.46 (0.84, 6.07)**	559	11.64 (7.2, 16.07)***
Electronic health record has immunization and medication information	323	6.77 (2.78, 10.76)***	323	5.63 (0.35, 10.91)*	180	4.07 (−3.59, 11.73)
Health care provider communicated with school staff about child’s condition	606	5.38 (2.49, 8.26)***	606	6.14 (2.55, 9.73)***	330	7.14(1.19, 13.08)*
Protocols/Plans    Child has shared care plan2	1003	6.54 (4.53, 8.55)***	1005	6.74 (4.09, 9.38)***	522	11.25 (6.65, 15.84)***
Child has written transition plan	162	8.01 (−1.94, 17.95)	162	18.92 (7.32, 30.53)**	85	5.53 (−15.11, 26.17)
Child has emergency care plan	1046	5.81 (3.42, 8.21)***	1048	6.8 (3.67, 9.93)***	538	9.76 (4.49, 15.02)***

Notes: Adjusted regression coefficients (β) have been scaled to reflect the change in the validation metric associated with a 0 to 100 change in the FECC measure score; FECC = Family Experiences with Coordination of Care. The sample size (N) for each quality measure varies due to variable eligibility for each measure. For example, only children >15 years old were eligible for the written transition plan quality measure.

^*p<0.05,

^**p<0.01,

^***p<0.001

¹Adjusted regression coefficients were obtained using a linear regression model adjusting for state, mode of survey administration (randomized to mixed mode or telephone only), child age, caregiver gender, caregiver age, caregiver race/ethnicity, caregiver education, and caregiver relationship to child. All quality measures and validation measures were on a 0-100 scale.

²Endorsed by the National Quality Forum (NQF) in June 2016

³Regressions not possible for this FECC measure because all caregivers eligible for these care coordination outcomes reported having a care coordinator (all have a score = 100 for this FECC measure: “has care coordinator”)

The FECC quality measures with the largest positive associations with provider ratings included having a care coordinator who was knowledgeable, supportive and advocated for the child’s needs (β =26.36, 95% CI: 19.98, 32.75, p<0.001), receiving a written visit summary that was useful and easy to understand (β =22.04, 95%CI: 17.07, 27.01, p<0.001), and receiving a written hospitalization summary that was easy to understand (β =18.93, 95%CI: 8.32, 29.55, p<0.00l).

The FECC quality measures with the largest positive associations with the access to care composite included having a care coordinator who asked about concerns and health changes (β = 28.84, 95% CI: 20.22, 37.45, p<0.001), having access to a medical interpreter when needed (β = 26.98, 95% CI: 7.82,46.14, p<0.01), receiving appropriate written visit summary content (β = 26.41, 95%CI: 18.91, 33.92, p<0.001), and receiving a written visit summary that was useful and easy to understand (β =31.4, 95% CI: 23.85, 38.95, p<0.001).

In terms of reports of receiving needed help managing the child’s care, the FECC quality measures with the largest positive associations with this validation measure were having a care coordinator who was knowledgeable, supportive and advocated for the child’s needs (β =56.38, 95% CI: 45.24, 67.53, p<0.001) and receiving a written hospitalization summary with appropriate content (β =55.71, 95%CI: 25.48, 85.93, p<0.001).

The only FECC quality measure that was not associated with any of the validation measures was the measure assessing whether the caregiver was invited to join hospital rounds.

Discussion

In this study examining the convergent validity of the newly developed FECC quality measures for CMC, we found that 19 of the 20 measures had significant positive associations with reports of provider ratings, access to care, and receiving needed help managing care, as assessed by three modified CAHPS^© measures. These results are consistent with our hypothesis that receipt of higher quality care coordination as assessed by the FECC measures would be associated with higher scores on the CAHPS^© measures.

These results demonstrate the convergent validity between the FECC quality measures developed by the COE4CCN and well-established measures of caregiver experience. These new measures add considerable depth to our ability to measure the quality of key components of care coordination for CMC. Improved care coordination is a key element of the medical home model¹⁷ and previous research has shown that children cared for in medical homes may have better health outcomes, more efficient utilization of services, and improved patient and family experiences with care.^18,19 Our results suggest that organizations wishing to evaluate their medical home initiatives or other programs for CMC might consider assessing the FECC measures most strongly associated with provider ratings, timely access to needed care, and caregiver reports of receiving needed help with managing their child’s care. In particular, the FECC measures assessing whether a child’s care coordinator was knowledgeable, supportive and advocated for the child’s needs, whether the caregiver received appropriate written visit summary content, and whether the caregiver received a written hospitalization summary that was easy to understand were measures that were strongly associated with two or more of the validation outcomes. For health care organizations focused on improving models of care for CMC, implementing this subset of FECC measures may facilitate tracking performance over time as well as informing ongoing improvement efforts.

Prior quality measure developers focusing on the broader population of children with special healthcare needs have identified and implemented useful measures of care coordination for these children.^8-10 For example, the National Survey of Children with Special Health Care Needs contains a Care Coordination domain with items measuring frequency of care coordination services, satisfaction with care coordination, perceptions of how the child’s health care providers communicate about the child’s care, and perceptions of communication between doctors and other health providers with the child’s early intervention program, school, child care providers, or vocational rehab program.⁹ While such measures are undoubtedly applicable to CMC, the FECC quality measures more specifically focus on the unique needs of CMC, a subset of children with special healthcare needs that would suffer most from poor care coordination. For example, the FECC quality measures assess the caregiver’s access to the care coordinator, whether the care coordinator asked about caregiver concerns and health changes of the child, the existence of a shared care plan, and the existence of an emergency care plan - aspects of care that are particularly important when a child has one or more complex chronic conditions.

Our study has several limitations. First, the associations noted in our cross-sectional survey cannot be interpreted as causal. Ideally, we would have been able to test whether the implementation of a new care coordination program was associated with changes in scores on the new FECC quality measures over time. This longitudinal design was beyond the scope of our charge for the current study. The validity of our measures in a cross-sectional study offers promise for testing them in longitudinal interventions or natural experiments where the responsiveness of the measures to quality improvement efforts and/or new care models for CMC could be tested further. Second, our results are based on survey data and are thus restricted to survey respondents; unfortunately, limited information was available on nonrespondents to the survey. However, our survey response rate of 41% is similar to response rates observed in other mailed/telephone health surveys and is higher than response rates typically observed in pediatric surveys.^20-22 Lastly, it may be difficult to generalize from our caregiver sample to caregivers for all CMC. Caregivers/CMC were selected from only two states (WA and MN) and from a Medicaid population. Caregivers in these states may not be representative of caregivers in other states and thus, testing of these observed associations in other states with more diverse populations would be beneficial. We also required four visits to a health care provider in the past year and thus the results may not be as meaningful for CMC with less intensive patterns of utilization. However, inclusion of CMC with higher levels of health care utilization was deliberate in order to ensure that the FECC quality measures were relevant for the caregivers responding to the survey.

Conclusion

This study demonstrates the convergent validity of 19 newly developed FECC quality measures with well-established measures of caregiver experience. As a result, this work offers several new measures that can be used to validly assess the quality of care coordination services provided to CMC and may be useful for evaluating new health care models focused on improving services for this vulnerable population.