Development of a Novel Adult Congenital Heart Disease–Specific Patient‐Reported Outcome Metric

Abstract

Background

Patient‐reported outcome metrics (PROs) quantify important outcomes in clinical trials and can be sensitive measures of patient experience in clinical practice. Currently, there is no validated disease‐specific PRO for adults with congenital heart disease (ACHD).

Methods and Results

We conducted a preliminary psychometric validation of a novel ACHD PRO. ACHD patients were recruited prospectively from 2 institutions and completed a series of questionnaires, a physician health assessment, and a 6‐minute walk test. Participants returned to complete the same questionnaires and assessment 3 months±2 weeks later. We tested the internal consistency and test–retest reliability by comparing responses among clinically stable patients at the 2 study visits. We assessed convergent and divergent validity by comparison of ACHD PRO responses to existing validated questionnaires. We assessed responsiveness by comparison with patient‐reported clinical change. One hundred three patients completed 1 study visit and 81 completed both. The ACHD PRO demonstrated good internal consistency in each of its 5 domains (Cronbach's α: 0.87; 0.74; 0.74; 0.90; and 0.89, respectively) and in the overall summary score (0.92). Test–retest reliability was good with an intraclass correlation ≥0.73 for all domains and 0.78 for the Summary Score. The ACHD PRO accurately assessed domain concepts based on comparison with validated standards. Preliminary estimates of responsiveness suggest sensitivity to clinical status.

Conclusions

These studies provide initial support for the validity and reliability of the ACHD PRO. Further studies are needed to assess its sensitivity to changes in clinical status.

Nonstandard Abbreviations and Acronyms

ACHD

adult congenital heart disease

KCCQ

Kansas City Cardiomyopathy Questionnaire

ICC

 intraclass correlation

NYHA FC

  New York Heart Association functional status classification

PRO

patient‐reported outcome metric

QOL

quality of life

Clinical Perspective

What Is New?

• We present the development of the first disease‐specific patient‐reported health‐status metric for adults with congenital heart disease.

What Are the Clinical Implications?

• This patient‐reported outcome metric may be used to facilitate patient‐centered care in clinical practice, and as a meaningful outcome in clinical trials.

There is increasing awareness of the need to develop tools that directly measure patients’ health status; their symptoms, physical and psychological function, and quality of life (QOL). While numerous tools have been developed for a range of diseases,1, 2 there remains no reliable and valid disease‐specific patient‐reported outcome (PRO) measure for patients with adult congenital heart disease (ACHD). The absence of an ACHD‐specific PRO is an important gap, given the growing prevalence of long‐term survivors with ACHD.3 Once available, such a tool could support further study into treatments for ACHD and foster a more patient‐centered focus in clinical practice and care quality assessment.4, 5

Despite its potential usefulness, developing a PRO for ACHD poses several unique challenges. Most specifically, the ACHD population is very heterogeneous with highly variable anatomy and physiology. There is therefore the possibility that each patient will report unique clinical manifestations and experiences. Accordingly, in prior work we directly assessed this potential heterogeneity and found that patient experiences during periods of deteriorating cardiac status are similar among ACHD patients regardless of anatomy, prior surgical procedures, or geographical location.6 This work supported the principle that a single instrument could encompass the diverse breadth of patients with ACHD. We have now extended this initial work and have developed a PRO for patients with ACHD, adhering to the US Food and Drug Administration's guidance for creating PROs.7 This report describes the preliminary psychometric validation and reliability estimates of the tool we have developed in a clinical population of ACHD patients. It also describes the process through which items were eliminated to decrease response burden and minimize the time required for patients to complete the questionnaire.

Methods

We conducted a prospective study of patients at 2 different institutions, Baylor University Medical Center in Dallas, Texas and Children's National Medical Center in Washington, DC. This study was approved by the institutional review boards at each of these institutions and the University of Texas Southwestern Medical Center and was conducted in accordance with the Helsinki declaration and the International Conference on Harmonization Good Clinical Practice guidelines. Each participant provided written informed consent to participate in the study. We will make the data, methods used in the analysis, and materials used to conduct this research available to any researcher for purposes of reproducing the results. The data that support the findings of this study will be made available from the corresponding author upon reasonable request.

Study Population

Consecutive patients meeting inclusion criteria and seen in the outpatient clinics of each participating institution were approached at the time of regular outpatient visits by local site principal investigators, or their surrogates, and offered enrollment. The inclusion criteria included age >18 years, a diagnosis of ACHD confirmed by chart review at the enrolling institution, regular follow‐up in the local ACHD clinic, mental capability to reliably complete the study questionnaires as documented by having independent decision‐making capacity, ability to provide independent informed consent, and English language fluency. Exclusion criteria included pregnancy and hospitalization within the 6 weeks prior to enrollment.

ACHD PRO Development

The ACHD PRO was designed according to US Food and Drug Administration recommendations.3 Briefly, after verifying that a single PRO could be used to assess symptoms in patients across the anatomical breadth of ACHD,6 focus groups were held in 2 different geographical locations (1 in Saint Louis, Missouri and 1 in Boston, Massachusetts). Sessions were audio recorded with participants’ consent, transcribed, and reviewed by a multidisciplinary team including a qualitative methodologist. These focus groups included 1 provider (A.C. in Saint Louis and A.S. in Boston) who asked open‐ended questions, beginning with “Describe your experiences when your congenital heart disease is getting worse.” The focus groups included 13 patients in Saint Louis and 4 in Boston. We achieved saturation for concepts surrounding clinical deterioration at both focus groups based on 2 lines of evidence. First, the same concepts were elicited and repeated in each session in 2 geographically separated sessions. Second, the concepts elicited on open‐ended questioning were identical to those identified on the patient and provider surveys we had previously conducted and published.6

Upon conceptual saturation, we organized transcribed patient statements into 5 domains, based on the aspects of health status to which the elicited concepts referred. These domains were the following: physical limitations, symptoms, anxiety or depression (psychological burden), arrhythmia, and QOL. We then developed question items from the transcribed statements for each domain, using as closely as possible the exact language patients used in the focus groups to describe these concepts. Additional items were recommended by the qualitative methodologist on the team based on her analysis of the transcript. Multiple questions for each domain were created to provide options for selecting the best items based upon subsequent patient feedback. We then conducted a second focus group session in Saint Louis, Missouri (10 individuals) where the preliminary questionnaire was presented for cognitive debriefing. Based on this second focus group, certain items were discarded and others reworded in response to patients’ feedback. We then conducted a final cognitive debriefing focus group in Saint Louis, Missouri (8 individuals) to ensure we had achieved optimal clarity and comprehensibility, which was confirmed by the participants. The PRO resulting from this process includes 33 total items: 5 in the physical limitations domain, 5 in the symptoms domain, 3 in the arrhythmia domain, 7 in the QOL domain, and 13 in the psychological burden domain. This tool, named the ACHD PRO, was evaluated in the present study (Figure 1).

Figure 1. Graphic depiction of the iterative stages in patient‐reported outcome metric development as recommended by the US Food and Drug Administration.

The present work is a part of the third step in development according to this process. ACHD indicates adult congenital heart disease; PRO, patient‐reported outcome metric.

Other PROs Utilized in the ACHD PRO Validation

To assess the convergent validity of the ACHD PRO domains, we compared each domain with existing validated tools measuring similar concepts. Each comparison is outlined in Table 1. The Kansas City Cardiomyopathy Questionnaire (KCCQ) is an extensively validated and sensitive health status measure in patients with heart failure, and scales of this tool were correlated with the physical limitation, symptom, and QOL scales of the ACHD PRO.1 The 8‐item Patient Health Questionnaire is a well‐validated and widely used tool to assess depressive symptoms.8 This and the 7‐item Generalized Anxiety Disorder Questionnaire were used to validate the psychological burden scale of the ACHD PRO.9 The Rand 36‐item Short Form Health Survey is a widely used tool for evaluating perceived health status with a well‐established psychometric profile,10 and its physical function domain and general health scales were used to assess the convergent validity of the ACHD PRO Physical Limitations and Quality of Life domains, respectively. In addition, the physician‐assigned New York Heart Association (NYHA) scale and 6‐minute walk test were used to further establish the convergent validity of the ACHD PRO Physical Limitation and Symptom scales.

Table 1.

Validation Standards

ACHD PRO Domain	Comparison Standard
Physical limitations	SF‐36 Physical Function domain, KCCQ Physical Limitation scale, NYHA FC, 6 MWT
Symptoms	KCCQ Symptom score, NYHA FC
Arrhythmia	No valid scale exists
Quality of life	KCCQ Quality of Life scores, SF‐36 General Health score
Psychological Burden	PHQ‐8 and GAD‐7

6MWT indicates 6‐minute walk test; ACHD PRO, adult congenital heart disease patient‐reported outcome metric; GAD‐7, 7‐item Generalized Anxiety Disorder questionnaire; KCCQ, Kansas City Cardiomyopathy Questionnaire; NYHA FC, New York Heart Association functional class; PHQ‐8, 8‐item Patient Health Questionnaire; and SF‐36, Rand 36‐item Short Form.

Study Protocol

The study required visit 1 at the time of study enrollment and a second visit 3 months±2 weeks later. At the time of both visit 1 and visit 2, patients were administered the ACHD PRO (the study PRO), Rand 36‐item Short Form Health Survey, KCCQ, 8‐item Patient Health Questionnaire, and 7‐item Generalized Anxiety Disorder Questionnaire. Physicians assigned an NYHA class at each visit, without referring to the patients’ PROs. At visit 1, patients also completed a 6‐minute walk test. At visit 2, both patients and providers completed a 15‐item Likert scale assessment to evaluate clinical changes and help establish clinically important thresholds of changed clinical status as previously described.11 At visit 1, the following baseline clinical variables were collected: age, sex, ethnicity, household income, employment status, education, cardiac diagnosis, cardiovascular surgeries, presence of a pacemaker or implantable defibrillator, and medical diagnoses. At visit 2 the following clinical variables were collected: admissions between visits, admission diagnosis if admitted, death, and interim heart transplant or mechanical circulatory support. A retrospective chart review was also performed by study staff if needed to complete missing clinical information.

All questionnaires were administered in the outpatient setting at the time of either a regular clinical office visit or at a time arranged by the research team. In cases where patients were unable to be physically present for visit 2, questionnaires were completed at home and clinical assessment by a physician was done via Skype (n=4).

Instructions on questionnaire completion were limited to those printed on the questionnaires themselves. Participants were advised that they were free to skip any item on any questionnaire that made them uncomfortable. Responses to questionnaire items and all abstracted data were entered into REDCap electronic data report forms.

Descriptive Analyses of the ACHD PRO

For the 103 patients participating in the study, we report the means, standard deviations, and ranges using the scoring strategy described below. 6

Psychometric Analysis

All measures used in the present analysis were scored according to the developers’ instructions.1, 12, 13, 14 The ACHD PRO scales were scored by assigning a point for each Likert category from the worst to the best functioning, subtracting 1, dividing by the range, and multiplying by 100. This converts each scale to a 0 to 100 range with higher scores indicating better function, fewer symptoms, and better QOL. The summary scale was calculated by averaging all individual domain scales, with each domain given equal weight. The ACHD PRO questionnaire can be found in Figure S1 and domain items in Table S1. When ≥25% of item responses in any given domain were missing (eg, a participant failed to respond to 2 items of a 7‐item scale), then the scale was not scored. Similarly, when >1 domain score was missing, the summary score was not computed.

Reliability and convergent validity Pearson coefficients were computed by analyzing complete cases (ie, patients missing scores were excluded from each analysis) and by analyzing all cases with multiple imputation of missing values. For multiple imputation, 10 complete data sets were generated using PROC MI and analyses were conducted on each complete data sets. The results were then pooled using PROC MIANALYZE in SAS. The 2 analysis strategies yielded similar results. Because all patients’ data are included, multiple imputation potentially increases the generalizability of findings. Consequently, reliability and validity coefficients reported used multiply imputed data.

For assessment of test–retest reliability, clinical stability was defined as the following: unchanged NYHA FC, lack of hospitalization, and stable clinical status based on treating cardiologist's and patient's report as assessed by a 1‐item question with 15 possible responses (Figure S2). For participants who experienced a change in clinical status between visit 1 and visit 2 based on self‐reported change in clinical status, we investigated the relationship between self‐reported clinical status on the 15‐item Likert scale and ACHD PRO summary scale as an exploratory analysis.

To estimate internal consistency reliability, we computed Cronbach's α coefficient.15 We estimated test–retest reliability using Pearson and intraclass correlations (ICC). Consistent with recommendations for PRO measures,16 we computed each ICC in a 2‐way mixed‐effect (random patient effect, fixed time effect) ANOVA model with interaction for the absolute agreement between single scores. To analyze data from all available cases, we computed Pearson correlations after multiple imputation of missing data, and ICC using maximum likelihood estimation.17

Item Reduction

For the longer ACHD PRO scales with higher internal consistency, we explored reducing the number of items while preserving acceptable reliability and validity. Item‐level analyses were conducted using baseline data and then the shortened scales were tested at follow‐up. First, the change in internal consistency was considered when particular items were dropped from the scale. Second, the magnitude of correlations of each scale item with conceptually convergent scales was evaluated. Finally, the reliability and validity of the original and shortened scales were compared using follow‐up data.

All analyses were conducted using SAS 9.3 (SAS Institute, Cary, NC), except for ICC computed using R.18

Results

Study Population

Table 2 shows descriptive statistics for the sample. Participants were primarily younger to middle‐aged adults, about half were women, the majority were white, and most were employed. Most participants had a NYHA functional classification of I and none were NYHA Class IV. There was a broad diversity of congenital heart lesions represented. Other medical diagnoses are listed in Table S2.

Table 2.

Participant Characteristics

Variable	N (Completed Assessment)	Mean/%	SD
Baseline assessment
Age, y	103	35.92	13.09
Female sex	103	50%
Ethnicity	97
Asian/Pacific Islander		2%
African/African American/black		12%
Hispanic		13%
Native American/American Indian		1%
Multiple ethnicities		1%
White/Caucasian		70%
Employment status	100
Unemployed		11%
Employed part‐time		12%
Employed full‐time		64%
Disabled		10%
Retired		3%
Household income ≥$50 000a	46	46%
ICD	103	19%
PPM	103	12%
Cardiac lesion	103
ALCAPA		1%
ASD		3%
ASD/VSD		3%
AVCD		2%
AVCD/TAPVR		1%
BAV		3%
BAV/CoA		1%
CoA		3%
Congenital MR		1%
Cor triatriatum		1%
Coronary anomaly		2%
DOLV		1%
DORV		1%
DTGA		16%
Atrial switch		13%
Arterial switch		3%
Ebstein's		2%
Eisenmenger		1%
Fontan		5%
Interrupted aortic arch		1%
LTGA		3%
PA/IVS		2%
PA/VSD		3%
PS		9%
PS/aortic hypoplasia		1%
PS/ASD		2%
PS/PAPVR		1%
PS/VSD		1%
Shone's		1%
Sinus venosus		1%
TOF		20%
VSD		5%
VSD/CoA		2%
VSD/DCRV		2%
VSD/ruptured sinus of Valsalva aneurysm		1%
Lesion complexity
Low	25	24%
Medium	41	40%
High	37	36%
6‐MWT distance, m	98	437.05	88.25
NYHA functional class	102
I		78%
II		16%
III		7%
IV		0%
Follow‐up assessment
NYHA functional class	80
I		81%
II		15%
III		4%
IV		0%
Physician‐rated clinical status change	80
Worse		15%
Same		64%
Better		21%
Patient‐rated clinical status change	80
Worse		15%
Same		62%
Better		22%

ALCAPA indicates anomalous left coronary artery from the pulmonary artery; ASD, atrial septal defect; AVCD, atrioventricular canal defect; BAV, bicuspid aortic valve; CoA, coarctation of the aorta; DCRV, double chambered right ventricle; DOLV, dual outlet left ventricle; DORV, dual outlet right ventricle; DTGA, d‐transposition of the great arteries; ICD, internal cardiac defibrillator; IVS, intact ventricular septum; LTGA, l‐transposition of the great arteries; MR, mitral regurgitation; 6‐MWT, 6‐minute walk test; NYHA, New York Heart Association; PA, pulmonary atresia; PAPVR, partially anomalous pulmonary venous return; PPM, implanted permanent cardiac pacemaker; PS, pulmonary stenosis; TAPVR, totally anomalous pulmonary venous return; TOF, tetralogy of Fallot; and VSD, ventricular septal defect.

^aIn US dollars.

Of 103 patients enrolled in the study, 21 did not participate in the follow‐up visit within the designated time window and 1 died, leaving 81 as candidates for assessment of test–retest reliability. Of these 81 candidates, 64 had stable NYHA FC but were not clinically stable based on 15‐item Likert scale assessment. Thirty‐eight patients had stable NYHA FC and were clinically stable in the opinion of both the provider and the patient based on clinical status scale responses.

Construct Validity

Table 3 shows correlations between the ACHD PRO questionnaire domains at baseline. The 5 scales were moderately to highly intercorrelated (median r=0.68, range=0.58–0.82), suggesting that they capture overlapping information. As expected, each domain scale correlated highly with the summary score (median r=0.87, range=0.82–0.92), which is the average of the individual domain scores.

Table 3.

Correlations Among the ACHD PRO Questionnaire Domains at Baseline

Scale	1	2	3	4	5
1. Physical limitations	···
2. Symptoms	0.80 [0.72, 0.86] (<0.001)	···
3. Arrhythmia	0.66 [0.53, 0.75] (<0.001)	0.69 [0.57, 0.78] (<0.001)	···
4. Quality of life	0.81 [0.73, 0.87] (<0.001)	0.68 [0.56, 0.80] (<0.001)	0.58 [0.43, 0.70] (<0.001)	···
5. Psychological burden	0.71 [0.59, 0.79] (<0.001)	0.61 [0.47, 0.72] (<0.001)	0.65 [0.52, 0.75] (<0.001)	0.82 [0.74, 0.87] (<0.001)	···
6. Summary score	0.92 [0.88, 0.95] (<0.001)	0.87 [0.81, 0.91] (<0.001)	0.82 [0.74, 0.87] (<0.001)	0.89 [0.84, 0.93] (<0.001)	0.86 [0.80, 0.90] (<0.001)

N=103, with multiple imputation of missing data. For each correlation, 95% CI appear in brackets and probability values appear in parentheses. Correlations computed without imputation of missing data were within +/− 0.03 of the values shown. ACHD PRO indicates adult congenital heart disease patient‐reported outcome metric.

Table 4 shows the correlations at baseline for each ACHD PRO domain and the summary score with their designated reference standards. Correlation between the ACHD PRO domain and each predefined reference standard were strong (median absolute correlation=0.76, range 0.53–0.82), with the exception of moderate correlations between the ACHD PRO physical limitations domain and 6‐minute walk test (0.29) and the ACHD PRO symptoms domain and NYHA functional class (−0.46). Correlations with the Patient Health Questionnaire depression and anxiety scales were negative because these scales measure distress, whereas correlations with the Rand 36‐item Short Form Health Survey and the KCCQ scales were generally positive because these scales measure favorable health.

Table 4.

Correlations of the ACHD PRO Domain and Summary Scores With Patients’ Functional Classification and Walking Distance, as Well as Other Patient‐Report Measures, at Baseline

ACHD PRO Questionnaire Scale	Reference Questionnaire
	SF‐36 Physical Function Domain	KCCQ Physical Limitation Scale	NYHA FC	6 MWT
Physical limitations	0.79 [0.71, 0.85] (<0.001)	0.78 [0.70, 0.85] (<0.001)	−0.53 [−0.66, −0.37] (<0.001)	0.29 [0.09, 0.46] (0.005)

	KCCQ Symptom Score	NYHA FC
Symptoms	0.73 [0.62, 0.81] (<0.001)	−0.46 [−0.60, −0.29] (<0.001)

	KCCQ Quality of Life Score	SF‐36 General Health Score
Quality of Life	0.78 [0.69, 0.85] (<0.001)	0.58 [0.44, 0.70] (<0.001)

	PHQ‐8	GAD‐7
Psychological burden	−0.74 [−0.82, −0.64] (<0.001)	−0.67 [−0.77, −0.54] (<0.001)

	KCCQ Functional Status Summary	KCCQ Clinical Summary
Summary Score	0.78 [0.69, 0.85] (<0.001)	0.82 [0.74, 0.88] (<0.001)

N=103, with multiple imputation of missing data. For each correlation, 95% CIs appear in brackets and probability values appear in parentheses. Tabled values are Spearman (NYHA functional class) or Pearson (all others) correlations. The KCCQ functional status summary combines physical limitations and symptoms scales, and the KCCQ clinical summary combines physical limitations, symptoms, quality of life, and social limitations. Arrhythmia domain is not included because there was no standard for comparison. Correlations computed without imputation of missing data were within +/− 0.01 of the values shown. 6MWT indicates 6‐minute walk test; ACHD PRO, adult congenital heart disease patient‐reported outcome metric; GAD‐7, 7‐item Generalized Anxiety Disorder questionnaire; KCCQ, Kansas City Cardiomyopathy Questionnaire; NYHA FC, New York Heart Association functional class; PHQ‐8, 8‐item Patient Health Questionnaire; and SF‐36, Rand 36‐item Short Form.

Internal and Test–Retest Reliability

Table 5 shows descriptive statistics for the ACHD PRO scales at baseline and at 3‐month follow‐up. Higher scores indicate better perceived health on a 0 to 100 scale. Cronbach's α internal consistency at baseline was high for the ACHD PRO scales assessing physical limitations (0.87), symptoms (0.74), arrhythmia (0.74), QOL (0.90), psychological burden (0.89), and the overall summary score (0.92).

Table 5.

Descriptive Statistics for the ACHD PRO Scales at Baseline and the 3‐Month Follow‐Up

Scale	Baseline Assessment			Follow‐Up Assessment			Retest Reliability
	N	Mean	SD	N	Mean	SD	r	ICC
ACHD PRO Scales
Physical limitations	103	69.18	23.07	81	71.53	18.97	0.66 [0.48, 0.78] (<0.001)	0.79 [0.63, 0.88] (<0.001)
Symptoms	102	80.11	19.64	81	79.58	17.38	0.53 [0.33, 0.68] (<0.001)	0.83 [0.70, 0.91] (<0.001)
Arrhythmia	101	82.33	20.5	81	82.96	18.47	0.54 [0.33, 0.70] (<0.001)	0.80 [0.64, 0.89] (<0.001)
Quality of life	103	71.25	21.46	81	73.25	19.51	0.57 [0.39, 0.72] (<0.001)	0.74 [0.55, 0.85] (<0.001)
Psychological burden	101	82.22	17.21	81	83.48	16.94	0.68 [0.54, 0.79] (<0.001)	0.84 [0.71, 0.91] (<0.001)
Summary score	103	76.91	17.82	81	78.16	15.56	0.69 [0.53, 0.80] (<0.001)	0.84 [0.71, 0.91] (<0.001)
PHQ Scales
PHQ‐8 Depression	103	4.66	5.14	81	4.74	4.72	0.54 [0.36, 0.68] (<0.001)	0.64 [0.41, 0.80] (<0.001)
GAD‐7 Anxiety	102	4.51	5.05	81	4.14	4.85	0.48 [0.27, 0.65] (<0.001)	0.57 [0.31, 0.75] (<0.001)
Heath Survey SF‐36
Physical functioning	103	76.28	26.26	81	73.82	27.03	0.74 [0.62, 0.83] (<0.001)	0.92 [0.86, 0.96] (<0.001)
Role limits–physical	103	72.73	37.56	81	74.38	39.13	0.65 [0.48, 0.78] (<0.001)	0.72 [0.52, 0.84] (<0.001)
Role limits–emotional	101	72.94	37.03	81	68.72	41.62	0.38 [0.19, 0.54] (<0.001)	0.60 [0.35, 0.77] (<0.001)
Energy/fatigue	102	53.97	23.8	80	56.6	22.62	0.61 [0.45, 0.73] (<0.001)	0.80 [0.65, 0.89] (<0.001)
Emotional well‐being	102	73.37	16.98	80	74.1	16.8	0.60 [0.43, 0.73] (<0.001)	0.73 [0.54, 0.85] (<0.001)
Social functioning	94	81.12	23.53	76	82.07	25.03	0.55 [0.35, 0.70] (<0.001)	0.89 [0.79, 0.94] (<0.001)
Pain	102	78.43	24.28	80	79.31	22.74	0.52 [0.33, 0.67] (<0.001)	0.64 [0.40, 0.79] (<0.001)
General health	102	60.29	20.27	81	59.83	22.92	0.59 [0.41, 0.72] (<0.001)	0.82 [0.67, 0.90] (<0.001)
Health change	103	60.19	26.76	81	62.35	25.96	0.47 [0.26, 0.63] (<0.001)	0.55 [0.29, 0.74] (<0.001)
KCCQ Scales
Physical limitations	103	86.35	20.31	80	84.55	21.12	0.66 [0.53, 0.77] (<0.001)	0.83 [0.69, 0.91] (<0.001)
Symptoms	103	83.18	21.79	81	82.26	19.64	0.61 [0.47, 0.73] (<0.001)	0.83 [0.69, 0.91] (<0.001)
Symptom stability	103	71.46	32.73	81	65.19	31.75	0.39 [0.17, 0.56] (0.001)	0.57 [0.31, 0.75] (<0.001)
Self‐efficacy	98	84.31	19.27	79	81.01	19.29	0.44 [0.25, 0.60] (<0.001)	0.60 [0.35, 0.77] (<0.001)
Quality of life	102	79.82	25.11	81	81.43	19.52	0.42 [0.18, 0.61] (0.001)	0.73 [0.54, 0.85] (<0.001)
Social limitations	90	84.44	25.04	74	85.59	21.59	0.43 [0.23, 0.59] (<0.001)	0.94 [0.88, 0.97] (<0.001)
Functional status	103	84.76	20.28	80	83.29	19.01	0.68 [0.55, 0.77] (<0.001)	0.86 [0.74, 0.92] (<0.001)
Clinical summary	102	83.72	21.36	80	83.50	17.16	0.64 [0.49, 0.74] (<0.001)	0.91 [0.84, 0.95] (<0.001)

Higher scale scores mark lower symptom burden, from 0 to 100. The follow‐up was mean=13.3 weeks (range 10.7–15.9) after baseline. Internal consistency estimated at baseline. Retest r estimated using all cases with multiple imputation of missing data. Retest ICC estimated using the 46.9% of cases with no hospitalizations, change in New York Heart Association functional class, patient‐reported clinical status, or physician‐reported clinical status. For each r or ICC, 95% CIs appear in brackets and probability values appear in parentheses. Correlations computed without multiple imputation (r) or use of maximum likelihood estimation (ICC) to account for missing data were within +/− 0.13 of the values shown. ACHD PRO indicates adult congenital heart disease patient‐reported outcome metric; ICC, intraclass correlation; PHQ, Patient Health Questionnaire; PHQ ‐ 8, 8‐item Patient Health Questionnaire; GAD ‐ 7, 7‐item Generalized Anxiety Disorder questionnaire; KCCQ, Kansas City Cardiomyopathy Questionnaire; and SF‐36, Rand 36‐item Short Form.

The follow‐up assessment occurred an average of 13.3 (SD=1.3; range 10.7–15.9) weeks after baseline. Test–retest reliability for the ACHD PRO scales was moderately high (median ICC=0.82, range 0.74–0.84) and similar to that of the other PROs evaluated (median ICC=0.73, range 0.55–0.94), as shown in Table 5.

Sensitivity to Clinical Status

Among those participants who reported a change in clinical status as assessed on the 15‐item Likert scale, 12 reported worsening clinical status, while 18 reported improved clinical status. There was a weak but significant correlation between patient‐reported clinical status and ACHD PRO score change between visit 1 and visit 2 (ρ=0.24, 95% CI [0.02, 0.44], P=0.031; Figure 2).

Figure 2. Relationship between patient‐reported change in clinical status as assessed by 15‐item Likert scale and ACHD PRO Summary Score; ρ=0.24, P=0.031.

ACHD PRO indicates adult congenital heart disease patient‐reported outcome metric.

Item Reduction

For the QOL scale, dropping item 30 (“Made me concerned that I might not have a successful career because of my health”), and for the psychological burden scale, dropping items 23 to 25 (all concerning difficulty sleeping: “Felt like my thoughts were racing,” “Felt afraid that I might not wake up when I fall asleep,” “Wondered what my heart was doing”), had minimal impact on reliability and validity at baseline. Table 6 shows the internal consistency and convergent validity correlations of the QOL and psychological burden scales at follow‐up. The original/longer and shortened versions of the scales had similar reliability and validity coefficients.

Table 6.

Internal Consistency and Convergent Validity Correlations of the Quality of Life and Psychological Burden Scales for Abbreviated Domains

	Quality of Life Scale		Psychological Burden Scale
	7 Items	6 Items	13 Items	10 Items
Cronbach's α internal consistency	0.89	0.89	0.92	0.92
Correlation with
SF‐36 General health	0.62 [0.46, 0.74] (<0.001)	0.61 [0.45, 0.73] (<0.001)	···	···
KCCQ Quality of life	0.57 [0.40, 0.70] (<0.001)	0.56 [0.39, 0.69] (<0.001)	···	···
PHQ‐8 Depression	···	···	−0.66 [−0.76, −0.51] (<0.001)	−0.62 [−0.74, −0.47] (<0.001)
GAD‐7 Anxiety	···	···	−0.58 [−0.71, −0.41] (<0.001)	−0.56 [−0.70, −0.39] (<0.001)

N=81, with multiple imputation of missing data. For each correlation, 95% CIs appear in brackets and probability values appear in parentheses. Correlations computed without imputation of missing data were within +/− 0.01 of the values shown. GAD‐7 indicates 7‐item Generalized Anxiety Disorder questionnaire; KCCQ, Kansas City Cardiomyopathy Questionnaire; PHQ‐8, 8‐item Patient Health Questionnaire; and SF‐36, Rand 36‐item Short Form.

Discussion

As health care strives to become more patient centered, directly measuring health status, symptoms, function, and QOL from patients’ perspectives is becoming increasingly important. In the present study, we present the development and initial psychometric validation of a novel ACHD‐specific health status PRO, the ACHD PRO. We demonstrate that in a diverse population of ACHD patients, the ACHD PRO had excellent validity as compared with other validated measures and high internal and test–retest reliability. These data are an important next step in developing a clinically useful PRO for ACHD, which can potentially benefit both research and clinical practice in the field.

The ACHD PRO demonstrates good initial evidence of its construct validity. Although there is significant overlap between ACHD PRO domains, when compared with external standards, the domains largely demonstrated moderate‐to‐strong correlations with the concepts of interest. These correlations suggest that the items developed for the ACHD PRO effectively measure the concepts for which they were developed based on comparison with existing established tools specific for those concepts. As an example, the questions in the ACHD PRO designed to assess depression had a very strong correlation with responses to the 8‐item Patient Health Questionnaire, an established and validated screening metric for depression. Notable deviations from this trend included correlation between the ACHD PRO physical limitations domain score and 6‐minute walk test and the ACHD PRO symptoms domain score and NYHA functional class. In the case of the former, the present findings concur with previous data demonstrating a generally poor correlation between self‐assessed exercise capacity and objectively measured values of maximal oxygen consumption.19 In the case of the latter, we suspect that NYHA was a suboptimal reference standard for the symptom domain in the ACHD population. We used NYHA as a standard for symptoms based on previous experience with the KCCQ. The KCCQ, however, is designed specifically for individuals with heart failure in which condition the majority of disease‐related symptoms are attributable to physical incapacity to perform activities. In ACHD, activity limitation was one among many disease‐related symptoms elicited in focus groups. Items in the ACHD PRO symptoms domain reflect this fact and address specifically dizziness, headache, sleepiness, and swelling. It is therefore not entirely unexpected that correlation with NYHA was somewhat low for this domain.

The reliability of the ACHD PRO compares favorably with that of existing widely used disease‐specific metrics for chronic heart disease. As part of assessing the validity of the ACHD PRO, subjects simultaneously completed the KCCQ (a well‐validated and widely used health status questionnaire for assessing individuals with chronic heart failure) and performance was similar. In addition, the internal and test–retest reliability were similar to those reported in the initial validations of the KCCQ1 and the Minnesota Living with Heart Failure Questionnaire.20, 21

Although the present study was not intended to evaluate sensitivity to clinical status, a number of participants experienced a change in clinical status between visits 1 and 2. We found that ACHD PRO summary score correlated significantly with patient‐reported change in clinical status, although the correlation was modest. This finding is encouraging for the clinical utility of the ACHD PRO; however, more data are required to investigate not only a relationship with patient‐reported clinical status but also with clinical event rates. These topics are the subjects of an ongoing study that we hope will provide this essential information.

Initial psychometric validation of the first ACHD‐specific PRO is a significant step forward in objectively quantifying outcomes in an important and underserved patient population. The population of ACHD patients represents a relatively new and growing group of individuals with chronic heart disease22, 23 subject to high lifetime rates of hospitalization24 and compromised life expectancy.25 To confront the challenges posed by ACHD, providers use therapeutic interventions that are based largely on expert opinion and physiologic intuition26, 27 because of the difficulty in conducting clinical trials in a small heterogeneous population with comparatively low annual rates of hospitalization and death.28 Given these impediments, the use of PROs as uniquely sensitive outcomes to define treatment response may permit researchers to address the profound evidence gap in caring for these patients.3

In addition, quality of life is a valid and fundamentally important outcome in and of itself.4 In many cases, it may be more important to patients with chronic heart disease than mortality.29, 30 The ability of PROs to simultaneously assess QOL and patient‐reported health status permits their use both as an independent outcome and as a surrogate for hospitalization and mortality.31, 32

While there are existing PROs for each of the domains assessed by the ACHD PRO, these metrics were developed and validated in non‐ACHD populations. We believe that the ACHD patient‐centric development of the ACHD PRO will make it a superior measure of health status in the psychologically unique ACHD population. In future research we plan to investigate its comparative performance in the clinical setting.

Limitations

Our data should be interpreted in the context of the following potential limitations: First, generalizability of the internal consistency and to a greater extent test–retest reliability testing may have been hampered by small patient numbers, although numbers are comparable to those in validations of other existing metrics.1 Assessment of domain validity is complicated by the absence of standards for the tested concepts in the relatively unique ACHD population. While we recognize this limitation, we used widely accepted and broadly used standards for comparison. There is no standard for assessment of arrhythmia, and validation of this domain specifically will need to be the subject of further research in the future. In addition, the QOL domain was compared with specific domains of existing questionnaires designed for different populations in the present study. While this is a limitation, “validity” is an ongoing process and future use of the ACHD PRO will hopefully further confirm that the measure is accurately capturing QOL specifically for patients with ACHD. Finally, the present validation was not intended to demonstrate sensitivity to changes in clinical status, only the capacity of the ACHD PRO to reliably assess clinical status in the indicated domains at one time‐point. Despite the preliminary responsiveness shown on the patients’ global assessment of change, future research will be required to further define the capacity of the ACHD PRO to reflect meaningful changes in clinical status across the breadth of ACHD lesions.

Conclusions

The ACHD PRO is a promising health status PRO for patients with ACHD and we have provided initial data to support its reliability and validity. With future research into the sensitivity of the metric to changes in clinical status, it may prove to be an important outcome tool for use in clinical trials and patient care.

Sources of Funding

This work was supported by the Cardiovascular Research Review Committee, grant # 51535, Baylor Heart and Vascular Institute, Dallas, TX.

Disclosures

None.

Supporting information

Tables S1–S2 Figures S1–S2

(J Am Heart Assoc. 2020;9:e015730 DOI: 10.1161/JAHA.119.015730.)