Development and Validation of the Adolescent Assessment of Preparation for Transition (ADAPT): A Novel Patient Experience Measure

Abstract

Purpose

Significant gaps exist in health care transition (HCT) preparation that can impact care and outcomes in young adults with chronic illness. No quality measure exists to directly assess adolescent experiences of HCT preparation. Our objective was to develop an adolescent-reported measure of the quality of HCT preparation received from pediatric health care providers.

Methods

The ADolescent Assessment of Preparation for Transition (ADAPT) is a 26-item mailed survey designed for completion by 16- and 17-year-old adolescents with a chronic health condition. Adolescents from 3 samples (2 large Medicaid insurance plans [n=3000 each] and 1 large tertiary care pediatric hospital [n=623]) were mailed the survey. An iterative developmental process included focus groups and cognitive interviews, and validity was assessed using confirmatory factor analysis and ordinal reliability coefficients.

Results

Reliability and validity was evaluated for three pre-specified composite measures: (1) Counseling on Transition Self-Management; (2) Counseling on Prescription Medication; (3) Transfer Planning. Across the 3 samples, all but one measure had good internal consistency (ordinal reliability coefficient ≥ 0.7). Confirmatory factor analysis using tetrachoric correlation coefficients was stable across samples and supported the construct validity of the first 2 composite measures.

Conclusions

ADAPT is a reliable, validated instrument measuring the quality of HCT preparation experiences reported by adolescents with chronic disease. ADAPT will enable clinical programs and health care delivery systems to assess the quality of HCT preparation and provide targets for improvement in adolescent counseling related to transition.

Introduction

The process during which adolescents and young adults move from pediatric-focused to adult-focused health care delivery systems, referred to as health care transition (HCT), has the central goal of providing adolescents with uninterrupted, high-quality, and developmentally appropriate care.¹ Lack of effective HCT may contribute to fragmentation of health care and increased risk for adverse health outcomes, particularly for youth with chronic health conditions.²

There is consensus that initial preparation for HCT should start in early adolescence and involve individualized counseling, planning, and skills development.^1,3 However, national data demonstrate a lack of attention to implementing recommendations for HCT preparation, with little improvement over the last several years. In the 2001 National Survey of Children with Special Health Care Needs (NS-CSHCN), a minority of parents reported having discussed transition with their child’s physician,² and only 30% had a plan for addressing transition needs.⁴ In both the 2005–2006 and 2009–2010 NS-CSHCN, this percentage improved, but remained below 50%.^5,6 Almost all assessments of HCT preparation are from parent or caregiver reports. An exception is the 2007 Survey of Adolescent Transition and Health (SATH), which targeted young adults age 19–23 years. Even in this survey, less than half of respondents reported receiving counseling around transition.⁷

Preparation for HCT should include the acquisition of self-care skills, development of a transition plan, and promotion of increased responsibility for chronic condition management.³ Examples of self-management and self-advocacy skills include scheduling medical appointments, obtaining medications and prescription refills, having one-on-one conversations with medical providers, being familiar with their medical history, understanding health insurance coverage, and feeling empowered to manage medical conditions. The Maternal and Child Health Bureau has reported a core transition services quality measure based on parent-reported data from 5 questions in the NS-CSHCN.⁸ Although many of these skills have been incorporated into patient-reported transition readiness scales and skills checklists such as the Transition Readiness Assessment Questionnaire (TRAQ),^9–12 a patient-reported experience measure on HCT preparation does not exist. Measures of patient experience are an important component of healthcare quality. Patient experience quality measures capture information on what patients value, what they directly observe, and can capture data for which patients are generally the best source¹³

The Agency for Healthcare Research and Quality and the Centers for Medicare and Medicaid Services established the Pediatric Quality Measurement Program (PQMP) to enhance and develop evidence-based pediatric quality measures including measures on transition from child- to adult-focused care.¹⁴ Because HCT preparation is primarily a series of interactions with clinicians, obtaining reports from youth directly about their experience is critical to understanding the current quality of health care delivery during HCT for this population. As part of the PQMP, we developed the ADolescent Assessment of Preparation for Transition (ADAPT) survey to serve as a reliable, validated, patient-reported instrument to measure the quality of experiences with HCT preparation in populations of adolescents with chronic health conditions.

Methods

Measure Development

We developed the ADAPT survey to measure adolescent-reported experiences of HCT preparation. An initial list of key concepts and possible survey questions were created following a series of interviews with experts in the field, review of existing measures of transition readiness and preparation, and review of prior parent-reported survey questions on HCT.^{9, 15} Subsequently, eleven focus groups were conducted in 3 US cities to explore parent and youth experiences with various processes of care involved in HCT, including self-management education, care coordination, and communication with health care providers around transition planning and transfers of care. Focus groups consisting of either adolescents with a chronic health condition, young adults with a chronic health condition, or parents of such youth, were conducted in English and Spanish and included a diverse spectrum of youth and parents with regard to sex, race, ethnicity, and type of chronic health condition. We synthesized focus group findings to inform the draft survey, with questions written at or below a 6^th grade reading level. We then conducted 26 cognitive interviews over 4 rounds to assess whether the intended respondents, 16- to 17-year-old adolescents with chronic health conditions, understood each of the survey items. The cognitive interview protocol contained candidate questions from the draft survey followed by pre-specified cognitive probes to evaluate the understandability of specific words and phrases and to clarify participant thought processes in answering the questions and selecting response options. Participants were also given the opportunity to suggest alternative language for specific items. Following the cognitive interviews, a final survey was developed for field testing.

Field testing of the final ADAPT survey was conducted with samples of health plan members with chronic illness and patients receiving care in a variety of pediatric clinical programs (primary care and specialty programs) at a large, freestanding children’s hospital. The survey was mailed to the parent or guardian with a cover letter asking to give the questionnaire to the adolescent to complete based on his or her own experience. Youth respondents were asked to identify a “main health care provider” and were directed to answer all subsequent questions in relation to their interaction with that provider over the previous 12 months. A postage-paid return envelope was included. A second survey packet was sent after about 30 days to non-respondents. Respondents received a $10 gift card for completing the survey. Each participating institution’s Institutional Review Board approved the study.

The final survey includes 26-items, with 12 items specifically measuring the quality of care received in 3 major areas: (1) Counseling on Transition Self-Management; (2) Counseling on Prescription Medication; and (3) Transfer Planning (Table 1). Each set of questions assesses the extent to which the components of preparation for a specific aspect of health care transition occurred, as reported by the adolescent. The survey also includes questions on each respondent’s frequency of visits to the identified main health care provider, self-reported health status, and key demographic characteristics (age, sex, education, race/ethnicity).

Table 1.

ADAPT items by composite scale

ADAPT Composite||
Counseling on Transition Self-management	Counseling on Prescription Medication	Transfer Planning
Q4. Did you talk with this provider without your parent or guardian in the room? Q5. Did you and this provider talk about you being more in charge of your health? Q6.* Did you and this provider talk about you scheduling your own appointments with this provider instead of your parent or guardian? Q7.* How often did you schedule your own appointments with this provider? Q8. Did you and this provider talk about how your health insurance might change as you get older?	Q9.** Did you take any prescription medicine? Q10. How often did you and this provider talk about all of your prescription medicines at each visit? Q11.** Were you prescribed any medicine to take every day for at least a month? Q12. Did you and this provider talk about remembering to take your medicines? Q13. Did you and this provider talk about you refilling your own prescriptions instead of your parent or guardian?	Q14.** Does this provider treat mostly children and teens? Q15. Did you and this provider talk about whether you may need to change to a new provider who treats mostly adults? Q16. Did this provider ask if you had any questions or concerns about changing to a new provider who treats mostly adults? Q17. Did you and this provider talk about a specific plan for changing to a new provider who treats mostly adults? Q18. Did this provider give you this plan in writing?

^||All questions begin with the phrase, “In the last 12 months,…” All questions except question 10 and 14 have response choices: “Yes” or “No”. Question 10 has the response choices “Never”, “Sometimes”, “Usually”, “Always”. Question 14 has the response choices “Yes”, “No”, “Don’t Know”.

^*For the CFA, a “Yes” response to either question 6 or 7 was combined as a “Yes” response to this item (Q67)

^**Screener questions; If the respondent answered “No” then directed to skip to next section of the survey.

Field Test Participants

Two health plans, AmeriHealth Caritas in Pennsylvania (AHCP), a Medicaid managed care health plan serving individuals across two regions in Pennsylvania (Health Plan #1), and Texas Children’s Health Plan (TCHP), a pediatric-focused Medicaid health plan serving individuals in Texas (Health Plan #2), each distributed mailed surveys to a simple random sample of parents of 16–17 year old youth identified as having a chronic disease. Chronic disease status was determined using the Pediatric Medical Complexity Algorithm (PMCA).¹⁶ This publicly available algorithm uses diagnosis codes in health plan claims to identify youth with either complex chronic disease (C-CD) or noncomplex chronic disease (NC-CD). Youth with C-CD have a physical, mental or developmental condition that can be expected to last at least a year and that affects 2 or more body systems, a malignancy, or a progressive condition associated with decreased life expectancy in adulthood. Youth with NC-CD have a chronic condition affecting one body system that lasts at least 1 year, but is generally not progressive, can vary widely in severity, and thus results in highly variable health care utilization.¹⁶

Each survey was pre-populated with the name of the specialist provider that the youth most commonly saw during the preceding year. For the two health plans, a choice of up to 3 healthcare providers was pre-populated in the survey based on the most common providers (either primary care or specialty care) in healthcare claims over the preceding year, and the respondent was asked to check one name as a “main provider”. For each survey, the youth was given the option of selecting a different “main healthcare provider” if the pre-populated name(s) was not correct. The survey directs the respondent to answer the remaining questions thinking about the main provider that they selected.

Surveys were sent to 1,500 families of youth with C-CD and 1,500 families of youth with NC-CD in each health plan. There were 248 and 231 undeliverable surveys in each health plan sample, respectively. For AHCP recipients, surveys were sent in English with an option for the family to contact the health plan to receive a survey in Spanish. In Texas, both English and Spanish language surveys were sent to each recipient. We received 1,339 surveys (780 of 2734 from AHCP [18 completed in Spanish] and 575 of 2752 from TCHP [28 completed in Spanish] for a final response rate of 28% and 21%, respectively.

For the clinical program field test, we mailed surveys to parents of 623 Boston Children’s Hospital (BCH) outpatients age 16–17 years old identified as receiving care in one of 10 different clinical programs (Endocrinology, Gastroenterology, Hematology – Sickle Cell and Hemophilia, Immunology, Metabolism, Nephrology, Primary Care, Pulmonology, and Spina Bifida). The families were identified by clinicians or clinical coordinators of each of the participating clinics. A total of 293 of 617 of these surveys were returned (response rate 47%); 6 surveys were undeliverable. There was a higher response rate among females in the clinical program sample (52% vs. 44%, p=0.02) but not in the two health plans.

There was a higher response rate among 17–18 year olds only in health plan 1 (28% vs. 24%, p=0.04) but not in the other two sites. There were significant differences in response by race/ethnicity only in the clinical program site driven by a lower response rate among blacks. There was no statistically significant difference in response rate based on chronic condition complexity category. Overall, therefore, there does not appear to be a systematic bias in response based on these demographic factors.

Statistical Analyses

Descriptive statistics (counts and percentages) for the respondent characteristic categorical variables (age, sex, race/ethnicity, level of education, insurance status, and chronic disease classification) were assessed.

Confirmatory Factor Analysis (CFA) was used for measure validation because the 3 survey constructs were pre-defined. In addition, since the items in these constructs had dichotomous responses, tetrachoric correlation coefficients were determined to be most appropriate for assessing the pairwise correlations among the items.^17,18 CFA was performed only for Counseling on Transition Self-Management and Counseling on Prescription Medication. CFA could not be reliably conducted for the Transfer Planning construct. Very few respondents in each sample (7% Health Plan #1, 6% Health Plan #2, 10% Clinical Program) answered “Yes” to question 15 (“Did you and this provider talk about whether you may need to change to a new provider who treats mostly adults?”), which serves as the screener for the remaining items for the construct, so the sample size was inadequate for this analysis.

The CFA evaluates whether the hypothesized latent two construct structure (2-factor model) represented the collected ADAPT data. A separate model was created for each of the three sites to ensure similarity of the 2-factor models across the samples. The hypothesized independent association of each question with its construct was tested based on the loading factor estimates using a two-tailed t-test. The association between the constructs was also tested with a t-test. Several indices were used to assess the goodness of fit of the two-factor model including the chi-square goodness of fit test, the Standardized Root mean Square Residual (RMSEA), the Comparative Fit Index (CFI) and the Tucker Lewis Index (TLI). These fit statistics allow for an overall interpretation of the adequacy of pre-defined sets of survey questions used in the constructs.

Of the available statistical indicators for internal consistency, the ordinal reliability coefficient (ordinal alpha), which uses a tetrachoric correlation matrix, was determined to be most appropriate for items with dichotomous responses. For questions with few response categories, the ordinal indicator more accurately estimates reliability compared to the more commonly used Cronbach’s alpha.¹⁹

Mplus (Statistical Analysis With Latent Variables) software was used to conduct the CFA for each site and SAS 9.3 was used for other analyses.

Results

Respondent characteristics for the three field test samples are presented in Table 2. There were more female than male respondents in each of the three samples; 40–45% of respondents in each sample were 16 years old, with the remaining respondents 17 years old. The samples were diverse in race/ethnicity. Among the clinical program respondents, 29% were insured by Medicaid, as were all respondents in the two health plan samples. In both health plans, there were about equal numbers of respondents by youth with C-CD or NC-CD consistent with the sampling approach.

Table 2.

ADAPT Survey Respondent Characteristics

	Health Plan #1	Health Plan #2	Clinical Program
N	780	575	293
Variable	n (%)	n (%)	n (%)
Sex
Female	397 (51.1)	323 (56.4)	157 (53.6)
Male	380 (48.9)	250 (43.6)	136 (46.4)
Age (years)*
16	350 (45.1)	229 (40.1)	124 (42.3)
17	426 (54.9)	342 (59.9)	169 (57.7)
Race/Ethnicity
Hispanic or Latino	119 (16.0)	331 (59.0)	19 (6.5)
American Indian or Alaskan Native	2 (0.3)	2 (0.4)	0 (0.0)
Asian/Pacific Islander	34 (4.6)	32 (5.7)	7 (2.4)
Black, Non-Hispanic	178 (24.0)	101 (18.0)	14 (4.8)
White, Non-Hispanic	386 (52.0)	87 (15.5)	244 (83.3)
Other	0 (0.0)	0 (0.0)	1 (0.3)
Multiple	23 (3.1)	8 (1.5)	8 (2.7)
Education
9th grade or less	112 (14.6)	57 (9.9)	21 (7.2)
10th grade	284 (37.1)	188 (32.7)	114 (39.0)
11th grade	299 (39.0)	222 (38.6)	136 (46.6)
12th grade or some college	71 (9.3)	108 (18.8)	21 (7.2)
Health Insurance**
Private	0	0	207 (70.6)
Public	780 (100.0)	575 (100.0)	86 (29.4)
Health Condition Category***
Complex Chronic	386 (49.5)	285 (49.7)	N/A
Non-Complex Chronic	394 (50.5)	288 (50.3)

^*A few were completed by 15 year olds (n=7 in health plan #2) or 18 year olds (n=24 in the clinical program, n=44 in health plan #1, and n=63 in health plan #2)

^**All health plan respondents are enrolled in Medicaid

^***Participants from the clinical programs were identified as receiving specialty care for a chronic health condition. The sample from the health plans was identified using the Pediatric Medical Complexity Algorithm,¹⁶ which results in 2 distinct categories of health condition.

The goal of the CFA was to test the construct validity of the survey using a two-factor structure including: (1) Counseling on Transition Self-management (4 items – 2 levels) and (2) Counseling on Prescription Medication (3 items – 2 levels; Table 1). The standardized solutions for the two-factor models measuring independence of each construct are included in Table 3. Results from these analyses supported the hypothesis that the individual items within each of the constructs are associated with one another. In each sample, the p-values of the loading factor estimates within each construct demonstrate that items are associated strongly with their hypothesized construct. In addition, the association between the two constructs in all 3 samples is also significant (p<.001).

Table 3.

Confirmatory factor analysis to evaluate the fit of a 2-factor model to the ADAPT data across samples

Health Plan #1 Model
Variable	Factor Loading Estimate	S.E.	Two-tailed T-test	P-value
Counseling on Transition Self-Management
Q4	.332	.075	4.442	<.001
Q5	.480	.076	6.306	<.001
Q67	.694	.093	7.489	<.001
Q8	.551	.114	4.809	<.001
Counseling on Prescription Medication
Q10	.600	.080	7.503	<.001
Q12	.673	.084	7.968	<.001
Q13	.576	.089	6.471	<.001

Health Plan #2 Model
Variable	Factor Loading Estimate	S.E.	Two-tailed T-test	P-value
Counseling on Transition Self-Management
Q4	.527	.092	5.702	<.001
Q5	.753	.1	7.515	<.001
Q67	.447	.105	4.269	<.001
Q8	.469	.113	4.152	<.001
Counseling on Prescription Medication
Q10	.594	.11	5.414	<.001
Q12	.643	.11	5.851	<.001
Q13	.408	.119	3.428	.001

Clinical Program Model
Variable	Factor Loading Estimate	S.E.	Two-tailed T-test	P-value
Counseling on Transition Self-Management
Q4	.516	.097	5.300	<.001
Q5	.594	.090	6.615	<.001
Q67	.561	.112	5.027	<.001
Q8	.665	.130	5.128	<.001
Counseling on Prescription Medication
Q10	.165	.108	1.527	.127
Q12	.463	.112	4.14	<.001
Q13	.826	.16	5.163	<.001

Model fit statistics for each of the three samples are presented in Table 4. Overall, the majority of fit statistics evaluated in each of the three samples suggest adequacy of the fit of a two-factor model. In two samples, the p-value of the chi-square test of fit was <.05, indicating that the observed covariance matrix is statistically significantly different from the expected matrix predicted by the hypothesized model; however, the chi-squared test is sensitive to sample size and therefore is not the only test of fit considered. In general, the other fit statistics measured in the CFA suggest that the models are adequate across the three samples (Tables 3, 4), providing further confirmation that questions grouped together in the ADAPT survey on conceptual grounds are empirically related.

Table 4.

Goodness of fit measures for CFA

	Health Plan #1	Health Plan #2	Clinical Program
Chi-square test of fit p-value	<0.001	0.244	0.013
Root mean squared error of approximation RMSEA (90% CI)	0.081 (0.061, 0.103)	0.026 (0, 0.062)	0.064 (0.028, 0.098)
Comparative Fit Index (CFI)	0.792	0.974	0.892
Tucker Lewis Index (TLI)	0.664	0.958	0.826

The ordinal alpha, a measure of internal consistency reliability, provides reliability results for all three constructs in the three samples (Table 5). In general, internal consistency reliability of 0.7 or greater is desirable. All constructs in all samples have an internal consistency of 0.70–0.99, with the exception of a single construct in one site (Counseling on Prescription Medication, clinical program sample). In general, the similarity of the ordinal alpha across sites also supports the internal consistency reliability in indicating the consistency of the responses to the questions intended to measure the same construct.

Table 5.

Internal Consistency Reliability for ADAPT Survey Composites by Sample

	Health Plan #1	Health Plan #2	Clinical Program
	Ordinal alpha
Counseling on Transition Self-Management	.70	.78	.79
Counseling on Prescription Medication	.75	.74	.57
Transfer Planning	.99	.99	.99

Discussion

Consensus guidelines for health care transition (HCT) emphasize the importance of preparation throughout adolescence that incorporates facilitation of self-management and purposeful transfer planning.¹ Because effective HCT preparation consists of specific interactions with health care providers, obtaining reports from youth directly about their experiences is essential. The ADolescent Assessment of Preparation for Transition (ADAPT) survey fills a critical gap in measuring the quality of adolescent health care by assessing adolescent experiences of HCT preparation, particularly related to adolescent-provider communication and counseling. Our analysis of data from three large, diverse populations provides evidence that the ADAPT survey provides a reliable and valid measure of three key HCT preparation domains: chronic condition self-management, prescription medications, and planning for transfer of care.

Other surveys of transition preparation have been either skills assessments or based on parent reports of HCT preparation. Available surveys of young adults following transfer of care have generally included a small set of questions regarding counseling on transition preparation.⁷ Although youth-reported HCT readiness instruments such as the TRAQ exist, they primarily evaluate self-reported skills and do not assess whether counseling on such skills was received within the health care system.¹² Unlike the TRAQ, ADAPT directly assesses adolescent experiences of HCT preparation during an age range in which these preparatory activities should be occurring as part of routine chronic disease management. ADAPT incorporates several domains that have been recommended as indicators of high quality health care for adolescents including time alone in consultations during clinical encounters, self-management, and transfer to adult health services.^{3, 20} The survey questions focus on patient experience, particularly communication with a health care provider identified by the respondent as their “main provider”. Although health care teams could include other members including social workers who might serve in care coordination roles for transition, we felt that the focus of the survey on the experiences with a main health care provider was important since such counseling at the patient-provider level is an essential component of high quality adolescent care. Because consensus recommendations for HCT preparation identify 14–15 years as the ideal age to initiate the development of a patient-specific transition plan,³ querying adolescents at 16–17 years captures them at a time by which some transition preparation should have occurred. The applicability of the measure to younger adolescents or young adults would require further study.

Our data has several limitations. In the health plan field tests, approximately 15% of surveys were undeliverable, perhaps to families with lower socio-economic status limiting access. The response rates in the health plan field tests were low, but are similar to reported response rates on mailed surveys to adolescents conducted by other healthcare organizations.²¹ The ADAPT measures also has some inherent limitations. First, it does not evaluate all of the important aspects of HCT preparation. For example, adolescents may not consistently assess whether a practice level transition policy exists. As with any patient-reported measure, patients may not recall a specific discussion or may recall erroneously one that did not occur. ADAPT is not appropriate for assessment of HCT preparation for adolescents who have cognitive or developmental delay and specific items in this survey would not apply to this population. The quality of HCT preparation in this population would need to be evaluated via parent-reported surveys or via modifications to the ADAPT after assessing whether an individual respondent had developmental delay. ^2,5,22 In addition, factor analysis for the Transfer Planning domain of the survey could not be completed due to a small sample size; such validation could be accomplished in the future with increased use of the survey. Nonetheless, the items assessed in that domain are included in all consensus-based recommendations for transition preparation and thus have appropriate face validity.³

In summary, ADAPT is a reliable, validated instrument measuring the quality of HCT preparation experiences reported by adolescents with chronic disease. The ADAPT survey was extensively tested to ensure that the measures are those for which adolescents are the best source of information. The instrument is designed to be brief and understandable in order to promote the best response rate possible from diverse samples. ADAPT fills an important gap in pediatric quality measurement and is meant to be used as a patient experience measure to evaluate and compare the quality of transition preparation among clinical programs or larger health systems. It will enable health care delivery systems to assess the quality of HCT preparation, and identify targets for improvement in care for adolescents.

Implications and Contribution.

The ADolescent Assessment of Preparation for Transition (ADAPT) is a 26-item survey with good psychometric properties that fills a quality measurement gap in key areas of preparation for transition from pediatric to adult-focused health care. ADAPT enables health systems to assess transition preparation quality and provide targets for improvement.

Abbreviations

HCT

health care transition

CSHCN

children with special health care needs