Psychometric findings and normative values for the CLEFT-Q based on 2434 children and young adult patients with cleft lip and/or palate from 12 countries

Abstract

BACKGROUND:

Patients with cleft lip and/or palate can undergo numerous procedures to improve appearance, speech, dentition and hearing. We developed a cleft-specific patient-reported outcome instrument to facilitate rigorous international measurement and benchmarking.

METHODS:

Data were collected from patients aged 8–29 years with cleft lip and/or palate at 30 hospitals in 12 countries between October 2014 and November 2016. Rasch measurement theory analysis was used to refine the scales and to examine reliability and validity. Normative CLEFT-Q values were computed for age, sex and cleft type.

RESULTS:

Analysis led to the refinement of an eating and drinking checklist and 12 scales measuring appearance (of the face, nose, nostrils, teeth, lips, jaws and cleft lip scar), health-related quality of life (psychological, social, school, speech distress) and speech function. All scales met the requirements of the Rasch model. Analysis to explore differential item functioning by age, sex and country provided evidence to support the use of a common scoring algorithm for each scale for international use. Lower (worse) scores on CLEFT-Q scales were associated with having a speech problem, being unhappy with facial appearance, and needing future cleft-related treatments, providing evidence of construct validity. Normative values for age, sex and cleft type showed poorer outcomes associated with older age, female sex and having a visible cleft.

INTERPRETATION:

The CLEFT-Q represents a rigorously developed instrument that can be used internationally to collect and compare evidence-based outcomes data from patients aged 8–29 years of age with cleft lip and/or palate.

Cleft lip and/or palate is the most common congenital craniofacial abnormality, with 7.9 clefts per 10 000 live births annually.¹ This condition presents with an abnormal connection between the mouth and the nose that can be visible on the lip, or invisible, affecting the palate alone. Patients can undergo numerous procedures to improve appearance, speech, dentition and hearing. Within and between countries, there is substantial variation in how surgeries are carried out.^2–4 For example, the landmark Eurocleft study^2,5–8 identified that the 201 participating centres used 194 different protocols for treatment of just 1 cleft subtype.

The goal of treatment for cleft lip and/or palate is to improve appearance, speech and psychosocial function, but measurement of outcomes has used mainly objective measures.^2–8 Inclusion of the patient perspective using a cleft-specific patient-reported outcome instrument could increase understanding of patient concerns. Patient-reported outcome instruments measure outcomes that matter to patients, from their perspective.⁹ Such tools are increasingly used worldwide to inform patient care, as quality metrics, as audit tools and in research.^10–16 Some countries (e.g., the United Kingdom, Netherlands, Sweden) measure patient-reported outcomes at the national level to compare providers or in clinical registries.^10–14

To address the call for a cleft-specific patient-reported outcome instrument,¹⁷ our team developed the CLEFT-Q. Our goal was to develop a common metric that could be used internationally and that works the same across age. The optimal design for an internationally applicable instrument would be for development to take place cross-culturally from the outset. Such an instrument could allow for rigorous measurement of the burden of cleft lip and/or palate in both high-resource and low-resource settings. Ensuring an instrument works the same for patients of different ages requires a mixed-methods approach with extensive qualitative research to identify common concerns for scale development, followed by advanced statistical techniques to identify any items biased by age. Full protocol details have been published.¹⁸ Content was developed from interviews with 138 patients from 6 countries.¹⁹ Findings were used to develop a conceptual framework comprising 3 domains; i.e., appearance, health-related quality of life and facial function. Cognitive interviews with 69 patients from 6 countries, and feedback from an international sample of 44 experts, provided input used to refine 13 scales and establish their content validity.²⁰ The scales were then translated and culturally adapted into 5 languages^21,22 following international guidelines.^23,24

The aim of this paper is to describe reliability and validity findings based on a set of statistical and graphical tests, including that of construct validity, whereby CLEFT-Q scores were hypothesized to be worse for participants who needed (v. did not need) cleft-related treatments, who were less (v. more) happy with their appearance and who had a more (v. less) severe speech problem. We also present CLEFT-Q normative values for age, sex and cleft type.

Methods

Study participants

Participants were aged 8–29 years with cleft lip, cleft palate, cleft lip and palate or cleft lip and alveolus, and able to complete the CLEFT-Q independently. Participants were recruited from craniofacial clinics in 30 hospitals located in 12 countries.

Data collection

Participants were recruited between October 2014 and November 2016. Data were collected either face to face or by mail using electronic devices (tablets) or paper and pencil (booklets) based on each site’s logistics and preference. The appearance scales asked respondents how much (not at all, a little, quite a bit, very much) they like the appearance of their face, nose, nostrils, teeth, lips, jaws and cleft lip scar, and to answer thinking of how they look now. Health-related quality of life (psychological, school, social, speech distress) and facial function (speech, and eating and drinking) scales ask respondents how often (never, sometimes, often, always) a set of statements apply to them, and to answer based on the past week. Participants completed only relevant scales (e.g., jaws, for participants aged 12–29 yr; cleft lip scar, for anyone with cleft lip; and school, for participants aged 8–18 yr and attending school). Site staff completed a clinical form that asked about cleft type, speech and current or future treatment needs. We entered data into REDCap,²⁵ hosted at McMaster University (Canada).

Statistical analysis

We used SPSS (IBM SPSS Statistics, Version 24, IBM Corp) and RUMM2030²⁶ software for data analysis. In Rasch analysis (described in detail elsewhere²⁷), individuals are placed along a scale based on the probability that they will answer a set of items in a certain manner.²⁸

We based decisions about items on a set of statistical and graphical tests.^27,29,30 We amended the sample to 500 for tests of fit statistics.²⁷ We conducted the analysis using the entire sample, except for the 2 speech scales (participants without a speech problem were excluded) and eating and drinking (participants with cleft lip only were excluded).

Thresholds for item response options

We examined thresholds between response options (e.g., “not at all” and “a little”) to determine if a scale’s response categories were ordered, meaning that a “1” on a 4-point scale must sit lower in the continuum than a “2,” and so on.³¹ On detecting disordered thresholds, we recoded the scores.

Item fit statistics

Three fit indicators were examined: log residuals (item–person interaction), Χ² values (item–trait interaction) and item characteristic curves.²² Ideal fit residuals are between −2.5 and +2.5 with Χ² values nonsignificant after Bonferroni adjustment. We interpreted fit statistics together and in relation to clinical usefulness.

Dependency

We inspected residual correlations between pairs of items to identify any that were greater than 0.30, as high residual correlations can inflate reliability. For any residual correlations greater than 0.30, we carried out a subtest to determine impact on reliability.²⁹

Targeting

We examined item locations to determine whether they were evenly spread over a reasonable range that matched the range of the construct reported by the sample.²⁷

Differential item functioning

We used this test to determine if individuals in subgroups (i.e., country, sex, age [8–10, 11–13, 14–17, 18–29 yr]) responded differently to items despite the same measured trait level. We chose random samples to create equal-sized subgroups, with a proviso that there had to be a minimum of 200 to provide 50 for 4 class intervals.³⁰ The number of countries in the analysis by scale was as follows: jaws = 2; cleft lip scar and school = 3; teeth and psychological = 5; face, lips, nose, nostrils and social = 6. For age, we divided the sample into 4 groups to ensure there were at least 50 participants per class interval in the differential item functioning analysis. We conducted the analyses with and without adjusting the overall sample to 500.²⁷ Items with Χ² values significant after Bonferroni adjustment were split on the variable that evidenced differential item functioning, and the new and original person locations were correlated to examine any impact on scoring.²⁷

Person separation index

This statistics measures error associated with the measurement of people in a sample. Higher values show greater reliability.³²

In addition to Rasch analyses, we computed Cronbach α.³³ To examine construct validity,³⁴ Rasch logit scores were transformed into 0 (worse) to 100 (best). We hypothesized that participants who may need (v. not need) cleft-related treatments, or were receiving (v. not receiving) psychological therapy to cope with cleft lip and/or palate, would report lower scores on the scales designed to evaluate these treatments. We asked participants: “How happy are you about how you look (your appearance)?” Four options (very unhappy, a little unhappy, a little happy, very happy) were provided. We hypothesized that unhappy participants (v. happy) would report lower scores on appearance and health-related quality of life scales. Third, we hypothesized that participants with a mild or moderate-to-severe speech problem (v. no problem) would report lower scores on speech and scales. Mean CLEFT-Q scores by age, sex and cleft type provided normative values.

As we found data for subgroups to be normally distributed, we used parametric (t test or one-way analysis of variance [ANOVA]) tests to examine mean differences. We used p values < 0.05 to identify statistical significance.

Ethics approval

Ethics board approval was obtained at 30 hospitals in 12 countries; the first board to grant approval was the Hamilton Integrated Research Ethics Board in Canada.

Results

Table 1 shows sample characteristics. The proportion of participants with a syndrome and/or other craniofacial anomaly was 10.9%.

Table 1:

Characteristics of participants in the CLEFT-Q study

Characteristic	No. of participants (%) n = 2434
Country
Canada	624 (25.6)
United States	362 (14.9)
England	339 (14.0)
India	232 (9.5)
Colombia	210 (8.6)
Netherlands	206 (8.5)
Ireland	100 (4.1)
Sweden	100 (4.1)
Spain	93 (3.8)
Chile	89 (3.7)
Turkey	54 (2.2)
Australia	25 (1.0)
Language
English	1450 (59.6)
Spanish	392 (16.1)
Hindi	232 (9.5)
Dutch	206 (8.5)
Swedish	100 (4.1)
Turkish	54 (2.2)
Age, yr
8–9	426 (17.5)
10–11	411 (16.9)
12–13	372 (15.3)
14–15	385 (15.8)
16–17	293 (12.0)
18–20	300 (12.3)
≥ 21	246 (10.1)
Missing	1 (0.1)
Sex
Male	1351 (55.5)
Female	1081 (44.4)
Missing	2 (0.1)
Student
Yes	2149 (88.3)
No	274 (11.3)
Missing	11 (0.4)
Type of schooling
School regular	1834 (85.3)
School home	35 (1.6)
College or university	248 (11.5)
Other	24 (1.1)
Missing	8 (0.4)
Cleft type
Cleft lip and palate	1399 (57.5)
Cleft palate	568 (23.3)
Cleft lip	263 (10.8)
Cleft lip and alveolus	204 (8.4)
Current speech problem
No	1271 (52.2)
Mild	808 (33.2)
Moderate	270 (11.1)
Severe	32 (1.3)

Participants (n = 2434) in our convenience sample were recruited with different approaches: 23 sites used face-to-face recruitment and 7 sites included a mailout. Nonparticipants were tracked by 13 sites: 8 (using face-to-face) recruited 1096 patients and had 144 decline, 88.4% response rate; and 5 (using mailout) recruited 422 patients and had 872 decline, 32.6% response rate.

Psychometric findings

Rasch analysis provided evidence of reliability and validity for 12 of 13 scales. The eating and drinking scale proved an exception. Eight items had disordered thresholds. After we rescored across the 2 middle response options, the item fit statistics were acceptable, but overall model fit was significant (p = 0.02) and the person separation index was low (< 0.60). Used as a problem checklist rather than a scale, the proportion of participants to report a problem was as follows: biting into some foods (42%), some foods hard to chew (37.4%), food or drink going up the nose (36.6%), having to eat slowly (36.1%), having to take small bites (32.7%), food getting stuck in hole in the mouth (28.1%), having to avoid certain foods (25.8%), trouble drinking through a straw (18.6%), and food falling out of the mouth (14%).

Rasch analysis reduced the CLEFT-Q by 35 items to 110. Thresholds were disordered for 4 items in speech distress and 2 items in speech function. When we rescored all items across the 2 middle response options, both scales had ordered thresholds. Subsequent Rasch analysis used the rescored data.

All 110 items had nonsignificant Χ² p values after Bonferroni adjustment (Appendix 1, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.170289/-/DC1). Item fit was within ± 2.5 for 58/110 items. The item residual correlations were greater than 0.30 for 2 pairs of items (r = 0.43 speech distress; r = 0.32 speech function). Subtests showed that the correlated items had marginal impact on scale reliability (< 0.01 drop in the person separation index).

Appendix 2 (available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.170289/-/DC1) shows the distribution of person measurement and item locations for 3 scales to illustrate targeting. These figures show that most participants scored inside the range for which the scales provide measurement. Participants who scored outside the range of the scales tended to be those who “very much” liked how their face looks overall (Figures 1A and 1C), or did not have a speech problem (Figures 1B and 1D).

Figure 1:

A) Mean scores for CLEFT-Q appearance and health-related quality of life scales based on answer to question: “How happy are you about how you look (your appearance)?” B) Mean scores for CLEFT-Q speech and health-related quality of life scales by current speech problem. Note: Mod = moderate, psych = psychological.

Appendix 1 shows findings for differential item functioning, which was evident for 23 items by country, 25 items by age group and 10 items for sex. When we amended the sample to 500, differential item functioning was evident for 9 items by country and 4 items by age group. When we split items that were significant in the unadjusted analysis by the relevant patient characteristic, Pearson correlations between the original and split person locations showed that differential item functioning had a negligible impact, with all correlations ≥ 0.99.

All 12 scales satisfied the requirements of the Rasch model, with nonsignificant Χ² values (Appendix 3, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.170289/-/DC1). Person separation index values were ≥ 0.85 for 10/12 scales (with extremes) and 9/12 (without extremes). Cronbach α values ranged from 0.89 to 0.96. Scale-level missing data ranged from 9% to 2.2%. The Flesch–Kincaid readability statistics³⁵ for CLEFT-Q items were 0 to 5.2, with 115/119 items at grade 3 reading level or lower.

The number of participants judged to need cleft-related treatments was as follows: orthodontic (n = 860, 35.3%), rhinoplasty (n = 743, 30.5%), speech therapy or surgery (n = 491, 20.2%), orthognathic surgery (n = 401, 16.5%), lip revision (n = 380, 15.6%) and alveolar bone graft (n = 253, 10.4%). The mean scores for the CLEFT-Q scales designed to evaluate outcomes for these treatments were lower for participants who needed (v. did not need) treatment (p < 0.001 on all independent samples t tests). The mean differences in scores were as follows: orthodontic: teeth scale −7.9 (1.0; lower [worse] for those who needed treatment); rhinoplasty: nose scale −11.7 (1.0), nostrils scale −15.9 (1.3); speech therapy or surgery: speech distress scale −14.5 (1.1); speech function scale −15.0 (1.1); orthognathic surgery: jaws scale −15.3 (1.8); lip scar revision: lips scale −11.9 (1.3), cleft lip scar scale −10.4 (1.7); and alveolar bone graft: teeth scale −7.2 (1.6). Participants (n = 276) who were receiving psychological therapy (v. no therapy) reported lower scores on the school (−4.3 [1.6], p < 0.01) and social (−4.0 [1.2], p < 0.01) scales.

Participants who were unhappy (n = 381) versus happy (n = 1663) with how they looked reported lower scores (p < 0.001 on independent samples t tests) on all appearance and health-related quality of life scales (Figure 1A). Mean scores for health-related quality of life and speech scales (Figure 1B) were highest for the group without a speech problem and lowest for the group with a moderate or severe speech problem (p < 0.001 on ANOVA).

Normative values

Appendix 4 (available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.170289/-/DC1) shows the normative values. By age group, mean scores for 10/12 scales (exceptions were teeth and school) differed significantly. For most appearance scales, psychological and social scores were higher for younger participants and lower for older participants. By sex, differences in mean scores were significant for lips, cleft scar, jaws, psychological, social and speech function; male participants reported better scores on 5 of the 6 scales. By cleft type (Figure 2), the difference in mean scores was significant for 11/12 scales (the exception was cleft lip scar). Participants with an invisible cleft reported higher (better) scores for face, nose, nostrils and lips scales, and participants with no palate involvement reported higher scores for teeth and jaws. On all scales, the cleft lip and palate group had the lowest mean scores.

Figure 2:

Mean scores for CLEFT-Q by cleft type. Note: Psych = psychological.

Interpretation

The CLEFT-Q represents a rigorously developed patient-reported outcome instrument for patients aged 8 to 29 years. The psychometric analyses showed that 12/13 scales evidenced reliability and validity and measured a clinical hierarchy for each concept. Importantly, the age, sex and language differential item functioning analysis provided evidence for the use of a common scoring algorithm for each scale. The eating and drinking scale represented an exception. For this set of items, there was no clinical hierarchy and the items instead form a problem checklist. Findings of a poor functioning scale highlight the usefulness of modern psychometric methods for diagnosing problems within scales. As patient-reported outcome instruments are increasingly used worldwide to inform clinical practice, comparative effectiveness research and quality improvement initiatives,^10–16 it is crucial that only clinically meaningful and scientifically sound instruments are used.

Comparisons with other studies

Research shows that the overall impact of cleft lip and/or palate on psychosocial adjustment is low.³⁶ Our study provides findings about factors associated with poorer outcomes including cleft-related treatments, age, sex and cleft type. The finding of lower scores in those who need cleft-related surgery is in line with a study of 1200 youth with cleft lip and/or palate, which showed that surgical candidates reported lower health-related quality of life than nonsurgical candidates.³⁷ Our findings are also in agreement with Ranganathan and colleagues,³⁸ who found lower health-related quality of life in 71 children who wanted cleft revision surgery.

Limitations

Patients with syndromes and other craniofacial anomalies were underrepresented in our sample,³⁹ and patients younger than 8 years were excluded. Guidelines suggest 8 years of age is the lower limit for valid and reliable collection of patient-reported outcomes data.^40,41 To broaden the criteria to include these underrepresented groups would have required parent-report, which was outside the scope of our study. We did not use parents as proxies for patients who were not able to self-report, as they could have created error in the field-test data set.

The response rates were 88.4% for the face-to-face recruitment of nonparticipants in the study, and 32.6% for the mailout recruitment. The characteristics of the nonparticipants are not known.

Although the REDCap survey allowed participants to skip items, resulting in 2.2%–9.0% scale-wise missing data, an advantage of Rasch analysis is the ability to score scales with missing data. Our study did not examine test–retest reliability or responsiveness, nor how scales correlate with other instruments. These topics are the focus of the next phase of our research.

Finally, our large sample means that small differences in mean scores were statistically significant. Future research to understand the clinical meaning of CLEFT-Q scores is needed.

Conclusion

Recently, generic instruments such as PROMIS, KIDSCREEN and the Pediatric Quality of Life Inventory (PedsQL) were used to measure outcomes in cleft lip and/or palate.^42–45 Because cleft-related treatments can substantially change appearance and speech, using scales that lack content validity for cleft lip and/or palate could make it appear that interventions do not improve outcomes, when in fact the scales might not be asking the right questions. The CLEFT-Q provides clinicians and researchers with a rigorously developed instrument they can use to measure clefts and the impact of treatment. The CLEFT-Q is made available free of charge to all nonprofit users once they have signed a licensing agreement.

The science of psychometrics and the sophistication of electronic health records systems have advanced such that it is now possible for providers of health care to collect patient-reported data over the Internet or on a smart device with data used for multiple purposes.⁴⁶ Integration of the CLEFT-Q into electronic health records for use in patient care, research and global benchmarking, such as the International Consortium for Health Outcomes Measurement’s cleft standard set initiative,⁴⁷ could transform health services and outcomes for patients with cleft lip and/or palate around the world.