The measurement properties of tests and tools used in cystic fibrosis studies: a systematic review

Abstract

There is no consensus on how best to measure responses to interventions among children and adults with cystic fibrosis (CF). We have systematically reviewed and summarised the characteristics and measurement properties of tests and tools that have been used to capture outcomes in studies among people with CF, including their reliability, validity and responsiveness. This review is intended to guide researchers when selecting tests or tools for measuring treatment effects in CF trials. A consensus set of these tests and tools could improve consistency in how outcomes are captured and thereby facilitate comparisons and synthesis of evidence across studies.

Short abstract

A diverse range of tests and tools were found, these varied with respect to their measurement properties. There was inconsistency in the selection of tests and tools to measure the same/similar outcomes across studies. Consensus is required. https://bit.ly/3nw2uoE

Introduction

Research is conducted to generate evidence of the efficacy and safety of interventions to inform best clinical practice and thereby improve outcomes for patients. When designing studies, it is necessary to establish which outcomes must be evaluated to meet the study objectives and how these outcomes will be measured and analysed as end-points [1]. Tests or tools may be required for outcome measurement. To improve consistency and facilitate synthesis of evidence across studies, there is a need to establish a consensus set of these tests and tools for measuring outcomes in studies in people with cystic fibrosis (CF). These must be responsive to changes in the outcome of interest and capture outcomes with sufficient validity, reliability and precision [2, 3]. This is necessary so that results can be interpreted with confidence and be used to support the translation of evidence into practice.

There is no existing consensus on the selection of tests or tools for measuring outcomes. Selection is challenging for a number of reasons. First, tests and tools may lack appropriation validation, and hence their quality might be uncertain. Secondly, criteria to facilitate interpretation of the results of the test or tool may not exist. Thirdly, logistic or economic constraints may restrict the use of some. Finally, although initiatives to improve and standardise the use of patient-reported outcome measures (PROMs) have been established [4, 5], there is no standardised approach for evaluating and selecting optimal tests and tools more generally in clinical research.

As a first step towards developing a consensus set of tests and tools for measuring outcomes in CF studies, we aimed to evaluate and summarise the characteristics and properties of tests and tools that have been used in previous CF studies. In the interim, we hope this review will be used by clinicians, people with CF, researchers and policy makers to guide optimal selection of these tests and tools, and to encourage validation or development of new tests or tools for measurement where required.

Methods

Search strategy and selection criteria

This was a PROSPERO registered systematic review (CRD42020151785). The search strategy is provided in the supplementary material. Medline, Embase and the Cochrane database were searched from inception until July 2020. Outcome measures proposed for evaluation in the Clinicaltrials.gov registry were also evaluated.

Inclusion criteria were reports of randomised controlled trials, observational studies, conference abstracts and reviews written in English, evaluating one or more measurement properties of a test or tool used to measure health outcomes in studies among people with CF. Original studies were sought to provide additional information about the characteristics or measurement properties of the tests and tools where necessary. Registered trials without published results proposing evaluation of one or more measurement properties of novel tests or tools were also included. Exclusion criteria were tests or tools developed for diagnostic purposes or used for evaluation of microbiological outcomes, or validation studies written in languages other than English. Tests and tools used in people with CF but validated in non-CF populations were beyond the scope of this review.

Titles and abstracts were screened independently by two reviewers (C. McLeod and J. Wood). Potentially eligible studies were downloaded to Endnote and duplicates removed. Full text articles were retrieved and eligibility confirmed by consensus of the reviewers. A third reviewer (T.L. Snelling) was used to confirm eligibility where consensus could not be achieved. Relevant data were extracted by C. McLeod and recorded in an Excel database and cross-checked by J. Wood.

The following characteristics of selected tests or tools were recorded: the outcome construct measured; the target population; mode of administration of the test or tool; recall period (if relevant); time taken to perform the test; the range of possible scores; and the ease of use (feasibility). The following properties of measurement were critically appraised: 1) validity, including content validity, construct validity (including convergent and discriminant performance of the test, the structural validity and cross-cultural validity) and criterion validity (including concurrent and predictive validity); 2) reliability, including the test–retest and inter-/intra-rate reliability, internal consistency and measurement error; 3) responsiveness; and 4) the minimal clinically important difference (MCID). Definitions for these measurement properties were based on those provided by the Consensus-based Standards for the selection of health Measurement INstruments initiative (COSMIN); these are presented in supplementary appendix 1.

Definitions, abbreviations and citations

Quality of life (QoL) tools were broadly defined as those which capture an individuals’ perception of their life satisfaction relative to their goals in the context of their culture and value systems, and not those that capture QoL based solely on the health status of the individual per se (health-related quality of life; HRQoL) [6]. Disease-specific QoL tools were defined as those developed for measuring QoL in people with CF, whereas generic QoL tools were defined as those originally developed for use in other populations that have also been applied in studies involving people with CF.

A full list of abbreviations and their meanings used throughout this manuscript and supplementary materials are alphabetically listed in supplementary appendix 2. References for information presented in the tables throughout this manuscript are provided in the supplementary materials.

Results

The review process is depicted in figure 1. 118 studies evaluating the measurement properties of 74 tests and tools used in studies among people with CF identified from Medline, Embase or Clinical Trials met the inclusion criteria [7–119]; a summary of these studies is provided in table S3. This review included three registered studies proposing validation of tests or tools used in people with CF with unpublished results [120]. Nine source articles describing the characteristics or measurement properties of tests or tools were also included [121–129].

FIGURE 1.

Search strategy flow chart.

Characteristics of tests and tools

Tests or tools were categorised as PROMs capturing QoL or other patient-reported outcomes, clinical scores, radiological scores or tests capturing functional exercise performance, CF transmembrane conductance regulator (CFTR) function or sputum characteristics.

QoL tools

17 generic QoL tools evaluated in CF populations and seven CF-specific QoL were identified. Characteristics of these generic and CF-specific QoL tools are detailed in table S5 and table 1, respectively.

TABLE 1.

Characteristics of quality of life (QoL) tools developed for use in people with cystic fibrosis (CF)

Test or tool	Description	Constructs(s)	Target population	Administration	Recall period	Range of scores	Feasibility/cultural validity
CFIQ	40 items; 5–6 min to complete	Activity limitation (physical, social, leisure), school/work limitations, vulnerability/lack of control, emotional impact, treatment burden and future outlook	Children and adults with CF and their carers; interview templates for children aged 6–11 years, adolescents, adults >12 years and carers of children aged 0–18 years	Paper	36 items 7 day recall, the remainder “current status”	5- or 7-point verbal rating scale	Largely developed in Caucasian population; further validation required
CFQoL	QoL and symptom scoring tool; 52 items over nine domains; 15–20 min to complete	Two symptom scales (chest and emotional) and seven QoL domains: physical functioning (10), social functioning (4), treatment issues (3), future concerns (6), interpersonal relationships (10), body image (3), career issues (4)	Adults and adolescents	Paper	14 days	Each response 6-point Likert scale; total score 0–100	Time consuming
CFQoL scale (single item)	VAS: how has CF affected your QoL in the last 2 weeks? Couple of mins to complete	Single QoL question	Adults	Paper	14 days	0–10	Quick Validated in population who had not had pulmonary exacerbation for past 6 weeks
CFQ-R	QoL and symptom scoring tool; four scales: 1) CFQ-R >14 years (44 items over 9 domains); 2) CFQ-R child (8 domains, 35 items), interviewer-administered 6–13 years and self-report 12–13 years; 3) CFQ-R parent: 44 items, 10 min; 4) pre­school 3–6 years, 28 items, 15 min to complete	Activity limitation (physical, social, leisure), school/work limitations, vulnerability, lack of control, emotional impact, treatment burden and future outlook	CFQ-R >14 years; CFQ-R child: interviewer- administered 6–13 years and self-report 12–13 years CFQ-R parent: proxy report for children 6–13 years Separate are not proxy report for children 4–60 months	Paper/electronic	14 days	4-point Likert scale; total score 0–100	Most widely used HRQoL questionnaire in CF; translated into 34 languages; EMA/FDA supports use in clinical trials
eCF-QUEST	Electronic, 3 domains, 4 items	Global measure (40 items), gastrointestinal (5 items) and general health (2 items)	Adults	Paper/electronic	NR	NR	NR
DISABKIDS-CF	6 items, 2 min to complete	Impact and treatment dimensions	Exclusively for use in children and adolescents 8–17 years; self-report and proxy version (carer)	Paper	Each dimension 0–100	5-point Likert scale; scores 0–100% (higher score=higher QoL)	English and Spanish versions
FLZ-CF	Healthy and general patient population; 18 items over 8 domains, 9-item weighted scale, 5 min to complete	9 items: cough/dyspnoea, abdominal, sleep, eating, therapy routine, adherence, understanding by others, being needed by others, disadvantage	Adolescents >15 years and adults	Paper	28 days	0–100 (high score=high satisfaction)	Screening test
CQOLCF	35 item carer QoL instrument, <10 minutes	Physical, emotional, family and social functioning	Carers of people with CF	Paper	NR	Each response 5-point Likert scale	NR

CFIQ: CF impact questionnaire; CFQoL: CF QoL; CFQ-R: revised CF questionnaire; eCF-QUEST: electronic version of the CFQoL evaluative self-administered test; FLZ-CF: Questions of Life Satisfaction; CQOLCF: caregiver QoL CF scale; VAS: visual analogue scale; HRQoL: health-related QoL; EMA: European Medicines Association; FDA: US Food and Drug Administration; NR: not reported.

The Cystic Fibrosis Questionnaire-Revised (CFQ-R; original version 2003) has been the most widely used QoL tool reported in CF studies and is available in 34 languages [130]. It is endorsed for use in clinical trials by the US Food and Drug Administration (FDA) and European Medicines Agency [131, 132]. There are five versions available; these are described in table 1.

One QoL questionnaire for use by carers of people with CF was identified, the Carer QoL in CF questionnaire (CQOLCF); this is a 35-item questionnaire designed to evaluate how providing care for someone with CF impacts on a carers’ physical, emotional and social functioning and family [69].

Tools capturing patient-reported outcomes (excluding QoL)

Six questionnaires designed to evaluate self-reported levels of physical activity were identified [86, 101]. Two questionnaires capturing body image for use in people aged >14 years [116] and one tool measuring dietary intake in children aged between 7 months and 12 years (table S5) were also found [40]. One PROM has been used to evaluate the impact of CF on stigma, disclosure, public attitudes and negative self-image among adults with CF and their carers [96]. A separate PROM originally developed for use in people with asthma has been used to capture work productivity and activity in people with CF [92].

12 clinical scores calculated from outcome data reported by people with CF were identified; three were developed for use in CF pulmonary exacerbations [12, 106], three captured respiratory symptoms [22, 74, 85, 101, 115], three characterised pain [90, 133], two quantified abdominal symptoms [28, 31, 41] and one has been used to evaluate physical and psychological symptom burden [105, 106] (table 2). Of these, the CFRSD-CRISS (chronic respiratory infection symptom score) has been the most widely used in CF studies and is available in 38 languages; it evaluates eight respiratory symptoms and is validated for use in people with CF aged >12 year [44].

TABLE 2.

Scoring tools incorporating patient-reported outcome

Test or tool	Description	Constructs(s)	Target population	Administration	Recall period	Range of scores	Feasibility/cultural validity
Pulmonary exacerbations
AWESCORE	5 domains, each with 2 items	Respiratory (cough and sputum), physical (energy and exercise), nutritional (appetite and weight), psychological (mood and anxiety) and general health “wellness” and sleep	Adults	Paper	NR	0–100	NR
CFRSD/CFRSD-CRISS	Symptom score: respiratory and emotional items (respiratory only in short version); developed for pulmonary exacerbation	8 respiratory symptoms, 4 emotional items and 4 other items (or short-version CFRSD-CRISS; 8 respiratory items: difficulty breathing, fever, tired, mucus, chills/sweats, cough, mucus, chest tightness, wheezing)	>12 years	Paper/electronic	Daily	3–4 point Likert scale, total score 0–100	Available in 38 languages
Symptom score system	4 items; pulmonary exacerbation assessment	Cough, sputum volume and viscosity, breathlessness, fatigue	Adults	Paper	Daily	4-point Likert scale	NR
Respiratory
ReS-CF	4-item questionnaire; <1 min to complete	Self-reported VAS for respiratory symptoms, cough, chest congestion and sputum	Adults	Paper	NR	Each VAS scored separately 0–10 (worst)	Screening tool: respiratory symptoms
SOBQ	Developed for PEX assessment; 17 items (13 respiratory and 4 CF-related impacts)	0–6 years and 7–11 years		NR	NR	Not applicable
SOBQ	24 items; patients with COPD, CF and lung transplant recipients	Measures SOB while performing activities of daily living	Adults	Paper	NR	5-point Likert scale for each response; scores 0–120 (worst)	NR
Pain
BPI	Severity and impact of pain on daily functions in people with chronic diseases; short: 5 min, long: 10 min to complete	7 domains: general activity, mood, walking ability, normal work (including housework), relationships with others, sleep, enjoyment in life	Adults	Paper	Daily	NR	Psychometrically and linguistically validated in 24 languages
DPAQ-CF	7 items	Frequency, duration, intensity, location and coping response to pain	Adolescents and adults	Paper/electronic	Daily	5-point Likert scale for each response; total score 0–10	NR
MPI	52 items, 3 domains, 12 subscales; 15–30 min to complete	Pain experiences, responses of others to the patient's communicated pain, the extent to which patients participate in activities of daily living	18–64 years and >65 years versions	Paper	NR	NR	Available in 6 languages
Abdominal
CF-Abd score	28-item PROM for assessment of gastrointestinal involvement	Abdominal pain, appetite, bowel movement and symptom-related QoL	>6 years	Paper	14 days	NR	NR
Gastrointestinal symptom tracker	PROM for assessment of gastrointestinal and nutritional issues	4 domains; abdominal pain, stools, eating challenges and adherence	Adolescents and adults	Electronic (iPad)	10–14 days	0–100 (worst)	Easy to administer and complete
Symptoms involving multiple systems
MSAS-CF	Physical and psychological symptom burden	QoL: respiratory (6 items), psychological burden (5 items) and gastrointestinal (4 items)	Adults	Paper	7 days	Each symptom 4- or 5-point Likert scale	Previously validated in people with cancer, heart disease, HIV and critical illness

AWESCORE: Alfred Wellness Score; CFRSD: cystic fibrosis (CF) respiratory symptom diary; CRISS: chronic respiratory infection symptom score; ReS-CF: respiratory symptoms in CF tool; SOBQ: Shortness of Breath Questionnaire; BPI: brief pain inventory; DPAQ-CF : Daily Pain Assessment Questionnaire in CF; MPI: multidimensional pain inventory; CF-Abd: CF abdominal; MSAS: Memorial Symptom Assessment Scale; NR: not reported; VAS: visual analogue scale; PEX: pulmonary exacerbation of CF; SOB: shortness of breath; PROM: patient-reported outcome measures; QoL: quality of life.

Clinical scores

The modified Schwachman scale (first described in 1964) was developed as a longitudinal clinical assessment tool. It includes activity levels, chest findings, cough, growth, nutrition, the character of stool and radiological changes; lung function is not included in this measure [26]. A test developed by Radine et al. [45] measuring nocturnal cough over two consecutive nights was found to be safe and feasible.

Three prognostic scoring tools were found. The most recent, in 2004, was a 5-year survival prediction tool [63], which was designed to guide eligibility for lung transplant; survival is predicted based on age, sex, forced expiratory volume in 1 s (FEV₁ % pred), weight for age z-score, pancreatic function, presence of diabetes, infection with Staphylococcus aureus or Burkholderi cepacia and the annual number of pulmonary exacerbations. The second, the modified Huang score (first described in 1997) [26], was developed for use as a prognostic and longitudinal assessment tool for those with terminal disease and captures clinical features (including lung function), radiological features and complications of disease. The oldest tool, first described in 1973, is the National Institutes of Health score (NIH) [26, 108], which was developed for people aged 5–30 years. It predicts the probability of death within 3 years based on lung function, chest radiograph (CXR) changes, and physiological and psychological features.

Lung function tests

Tests used to measure lung function for which measurement properties were described include spirometry, raised volume rapid thoracic compression (RVRTC), impulse oscillometry and lung clearance index (LCI). Characteristics of these tests are reported in table S5.

Spirometry has been the most frequently used lung function test in CF studies, and the measure of lung function most commonly reported has been FEV₁ [30]. This has been variously measured as the crude volume (in litres) or as the percentage predicted volume for age and height, or z-score for age, sex, height and ethnicity; within-individual changes in the FEV₁ have been reported as either the absolute change, or change relative to the baseline measure [54, 72].

Imaging scoring tools

Four CXR scoring systems used to quantify the degree of structural lung disease were identified. The oldest, the Chrispin–Norman score in 1974, is based on chest configuration and the presence or absence of different types of “shadows” [123]. First described in 1979, the Brasfield or Birmingham score (scored between 3 and 25) aims to capture radiographic evidence of air trapping, bronchial wall thickening, bronchiectasis, atelectasis and general severity [73, 122]. From 1993, the Wisconsin score (0–100) has been used to evaluate six attributes including hyperinflation, peribronchial thickening, bronchiectasis, opacities and atelectasis [54, 129]. The Brasfield scoring system has been reported to be easier to use and quicker to perform than the Wisconsin score [73]. The Northern score (introduced in 1994) is calculated based on the presence of linear, cystic or confluent opacities in each lung quadrant rated on a four-point Likert scale (normal to very severe) on a single film [124].

Three computed tomography (CT) scoring tools were identified: the Brody score (I and II), originally developed in 1999, and the CF-CT score (introduced in 2011), which is based on the Brody II score and aims to improve standardisation of the latter [111]. These tools have been used in people aged >5 years. In 2014, the Perth Rotterdam Annotated Grid Morphometric Analysis method (PRAGMA-CT) score was developed for application in children and infants [102] and takes an experienced person ∼30 min to calculate per CT scan [111].

Scoring tools based on quantitative magnetic resonance imaging (MRI) are still in development [54].

Functional exercise performance

The most frequently studied field exercise test performed in people with CF is the 6-min walk test (6MWT) [36]. Characteristics of tests used to measure functional exercise performance are summarised in table 3.

TABLE 3.

Functional measures of exercise capacity

Test or tool	Description	Construct(s)	Target population	Administration	Recall period	Range of scores	Feasibility/cultural validity
iSTEP	Externally paced test; speed increases every 2 min Expired gas analysis	Expired gas analysis	Younger, fitter patients	Performance test	10 min	Variable	Portable, standardised and easy to administer field exercise test
MSWT	15-level modification of ISWT Office based walk/run test	Peak oxygen uptake	Children	Performance test; standard protocol	20 min	NA	10 m required, used in younger and fitter patients Excludes those with i.v. lines or those requiring oxygen support
PowerSTS	1-min sit-to-stand power index	Quadriceps power	Moderate–severe CF	Performance test; standard protocol	1 min	N/A	Quick and easy to perform
Triple hop distance	Starting at one end of a tape, asked to hop three times consecutively on dominant leg, trying to cover as much distance as possible	Lower extremity power	Older children and adults	Performance test	NR	Distance recorded in cm	NR
Vertical jump test	90-cm2 mat connected to a timer next to a wall; time off mat converted to a vertical jump (cm) using a controlled (90 degree) and uncontrolled knee angle	Power and posture	Older children and adults	Performance test	NR	Vertical distance recorded in cm	NR
3MST	Submaximal stress test (distance covered in m)	Externally paced test (metronome paced at 12 beats·min−1) step up and down a 6-inch step for 3 min	Good choice in severe CLD	Performance test; standard protocol	3 min	NA	Requires the least amount of space; 6-inch step required
6MWT	Submaximal stress test (distance covered in m)	Standard protocol; distance walked within 6 min (enough O2 to maintain saturations >90%)	Validated in 7–23 years Good choice in severe CLD	Performance test; standard protocol	6 min	NA	Most frequently studied exercise test in CF; easy but requires 30 m
30 s or 1-min-STS	Cardiorespiratory response during a 30-s or 1-min STS test (chair height 40 cm without armrest; full knee extension); as many repetitions as possible in 1 min	Exercise capacity	Moderate–severe CF	Performance test; standard protocol	1 min	Total number of full repetitions in 30 s or 1 min	Quick and easy to perform

iSTEP: incremental field step test; MSWT: modified shuttle walk test; Power_STS: 1-min sit-to-stand power index; 3MST: 3-min sit-to-stand test; 6MWT: 6-min walk test; STS: sit-to-stand test; ISWT: incremental shuttle walk test; N/A: not applicable; CF: cystic fibrosis; CLD: chronic lung disease.

CFTR function

Characteristics of tests used to directly (e.g. sweat chloride tests) or indirectly (e.g. nasal potential difference tests) measure CFTR function are summarised in supplementary table S5.

Sputum tests

Rheometry tests which capture the characteristics of sputum, and tests used to capture markers of inflammation in sputum, are summarised in table S5.

Measurement properties of tests, tools or instruments

QoL tools

The measurement properties of generic QoL tools based on their evaluation in CF populations are detailed in table S6, and the properties of CF-specific QoL tools are summarised in table 4.

TABLE 4.

Measurement properties of quality of life (QoL) tools specific for people with cystic fibrosis (CF)

Test or tool	Content validity	Convergent validity	Discriminant validity	Concurrent	Predictive	Intra- or inter-rater and test-retest	Internal consistency	Measurement error	Responsiveness	Comments/MCID
CFIQ	Demonstrated; people >6 years with CF and carers used in construction	NR	NR	NR	NR	NR	NR	NR	NR	Requires further validation; content validity established
CFQoL	Easy to understand/ complete; people with CF involved in construction	Correlation of emotional scores with SF-36 r=0.64; p<0.001	Chest and emotional scores distinguished between severity of chronic lung disease (FEV1 % pred >70, 40–70 or <40)	Chest score correlation with FEV1 not tested	NR	Test–retest rs=0.74–0.94 (p<0.01) Robust after 7–10 days; 0.9 for emotional scores and 0.93 for respiratory	Cronbach's α=0.3	NR	Chest symptom scores increased during pulmonary exacerbation treatment Chest and emotional score responsive over a 2-week application period in hospital (47–70.3, p=0.006) versus home groups (49.7–68.8, p=0.03)	NR
CFQ-R	Patients involved in testing clarity of items Preschool version: children able to understand and answer questions	Correlation between CFQ-R and SF-36 on physical (r=0.81, p<0.01), health perceptions/general health (r=0.79, p<0.01), vitality (r=0.84, p<0.01), role/role-physical (r=0.73, p<0.01), emotional functioning/mental health (r=0.74, p<0.01) and social (r=0.57, p<0.01) domains Strong convergence between child and parent proxy reports, although children generally reported better HRQoL than parents	CFQ-R: no significant difference between age groups (6–11 years, 12–13 years versus >14 years) for all domains except treatment between 6- to 11-year-olds and >14-year-olds Significant association between CFTR genotype and CFQ-R scores (K=9.34, p<0.01) Strong parent–child agreement found for scales measuring respiratory symptoms, but children reported more fatigue and difficulty running/walking	Respiratory score established using FEV1; correlation with FEV1 % pred r=0.42, p-value NR; correlation with number of intravenous antibiotic courses r=−0.27, p-value NR	NR	Acceptable	Cronbach's α=0.6–0.76 with the exception of treatment burden (α=0.44) Parent proxy report for CFQ-R physical, eating and respiratory subscales α=0.73–0.86	NR	Based on clinician judgement, a moderate change=0.5 units and an important change=0.8 units pre- & post- exacerbation	NR
CF-QUEST	NR	NR	NR	rs was 0.951 for the total CF-QUEST score, 0.929 for gastrointestinal module and 0.941 for GHQ module for paper/electronic versions	NR	NR	NR	NR	NR	Excellent correlation and agreement of electronic version with its validated paper counterpart Patient preference tended towards electronic version
DISABKIDS-CFM	NR	Convergent validity with KINDL-R established; r=0.6	NR	NR	NR	NR	Cronbach's α=0.55 (p=0.011) for the impact dimension and 0.480 (p=0.02) for the treatment dimension	NR	NR	NR
FLZ-CF	NR	Pearson's correlation r=0.75 with the generic satisfaction with health scale of the FLZ-CF, r=0.3 with FEV1 % pred and r=−0.26 with daily time for home therapy Leisure time/hobbies, physical condition, ability to relax, energy for life and satisfaction with health rs >0.5 with positive mood and ability to relax and SF-36 physical functioning, general health, vitality, social function and mental health	The scale discriminated significantly between patients with and without need for assistance with daily life and between patients with and without a partner	Physical condition/fitness and FEV1 % pred rs=0.66	NR	NR	Cronbach's α=0.82–0.89	NR	NR	Reliable and valid Targets general healthy and general patient population Short enough to be used as a screening instrument
Single item CFQoL questionnaire	NR	Most of the CFQoL variables were moderately correlated (r=0.38–0.61, p<0.001) with the single item scale weakly correlated with body image (r=0.25), p<0.01 Higher scores correlated negatively with frequency of hospital admissions in the previous year (r=−0.39, p<0.001)	Ability to distinguish adult CF patients with lower compared to higher CFQoL scores	Single-item scale correlation with FEV1 r=0.21	NR	ICC 0.78 (95% CI 0.59–0.88)	NR	NR	NR	Acceptable, valid and repeatable measurement tool that can be easily used
CQOLCF	NR	Correlation with mental health r=0.634, emotional distress r=−0.687 and physical health r=0.049	NR	NR	NR	NR	Cronbach's α=0.909	NR	NR	Appears to be valid, reliable and internally consistent scale

MCID: minimal clinically important difference; CFIQ: CF Impact Questionnaire; CFQoL: CF QoL Questionnaire; CFQ-R: revised CF Questionnaire; CF-QUEST: electronic version of the CFQoL evaluative self-administered test; FLZ-CF: Questions of Life Satisfaction; CQOLCF: Caregiver QoL CF scale; NR: not reported; SF-36: Short-Form-36 Item Questionnaire; HRQoL: health-related QoL; CFTR: CF transmembrane regulator; FEV₁: forced expiratory volume in 1 s; GHQ: General Health Questionnaire; KINDL-R: Child QoL Questionnaire-Revised; ICC: internal consistency coefficient; rs: Spearman's correlation coefficient.

The development of the CFQ-R involved people with CF, and it has been shown to be reliable, with sound content (including face) validity. The tool has been shown to have good internal consistency for all constructs examined, including parent proxy report of physical, eating and respiratory subscales (α=0.73–0.86), but not for treatment burden (r=0.44). The CFQ-R score correlates with FEV₁ and body mass index, and discriminates different degrees of disease severity [130], but a correlation with mortality has not been reported. Based on clinician judgement, a change of 0.8 units in the CFQ-R score was considered the MCID in the context of treatment for pulmonary exacerbations [134].

Patient-reported symptoms and function

The measurement properties of patient-reported symptoms and function are summarised in table 5.

TABLE 5.

Measurement properties of scoring tools based on outcomes reported by people with cystic fibrosis (CF)

Test or tool	Content validity	Convergent validity	Discriminant validity	Concurrent	Predictive	Intra or inter-rater and test–retest	Internal consistency	Measurement error	Responsiveness	Comments/MCID
Pulmonary exacerbations
AWESCORE	NR	NR	NR	Correlation of total AWESCORE and CFQ-R scores: r=0.632 (p=0.003)	NR	Pearson's correlation coefficient 0.854, p<0.0005	NR	NR	For exacerbation, score 47.5 (sd 11.2) at start of treatment and 21.6 (sd 15.6) at end of treatment (100=highest symptom severity)	11 points Mean change of −16.5 (95% CI −13.2 to −19.7 for exacerbation reported) No MCID for emotional score
CFRSD/CFRSD-CRISS	Involved people with CF in testing clarity of items	Step-rate significantly higher in those who did NOT experience difficulty breathing, cough, tightness or feeling tired (respiratory items) or feeling worried, cranky or frustrated (emotional items)	Respiratory scores distinguished between moderate/severe and mild/severe disease; emotional scores distinguished between mild/severe disease	Respiratory and emotional score established using daily step count (not FEV1)	NR	ICC 0.79 for respiratory scale using a 1-day interval	Cronbach's α for CFRSD-CRISS was 0.77	Test–retest reliability after 7–10 days; 0.9 for the emotional and 0.93 for the chest score	Total score been demonstrated to improve over 2 weeks' i.v. treatment	No MCID suggested on the basis of statistical analysis, but MCID >1 after 2 weeks of i.v. ABX suggested based on experience with COPD patients
Symptom score	Patients not involved in construction	All 4 items correlated with each other r>0.38; p<0.001	NR	Total score correlation with FEV1: r=−0.41, (p<0.0001) and respiratory score on CFQ-R: r=−0.62 (p<0.001) and CFQ-R: r=−0.47 (p<0.001)	NR	NR	NR	NR	NR
Respiratory
Borg Dyspnoea Scale	NR	NR	NR	NR	NR	ICC=0.933	NR	NR	Mean change in score −3.1 with mean effect size 1.3 from baseline to 4 weeks	Appears to be valid, reliable and responsive in CF For those reporting improvement, scores changed −2.9 overall, −3.5 for cough, −3.5 for congestion and −3.1 for sputum domains
ReS-CF	NR	NR	NR	Correlation between ReS-CF and CFQ-R; rs=−0.5 (p<0.001)	NR	ICCs for 4 scores >0.7	NR	NR	NR
SOBQ	NR	SOBQ scores correlated negatively with physiological measures of disease severity (FVC % pred: r=−0.36, p<0.05 and FEV1 % pred: r=−0.5, p<0.01) Scores correlated positively with Borg scale ratings of perceived breathlessness after 6MWT and QWB (r=−0.41, p<0.01)	NR	NR	NR	NR	α=0.96	NR	NR	MCID: 5 unit change
Pain
BPI	NR	Correlation of BPI pain interference and airway clearance therapy (p=0.002), coughing and breathing (p<0.012), pain prevalence and CFQoL physical function (p=0.01), CFQoL treatment (p=0.03), CFQoL work/school (p=0.02), CFQoL social (p=0.013) and CFQoL emotional scale (p=0.017) Pain intensity also correlated with CFQoL physical function, CFQoL treatment and CFQoL school/work (p<0.01)	NR	Correlation of BPI pain prevalence and sleep quality (p=0.045), sleep disturbance (p<0.001), daytime dysfunction (p=0.001) and sleep interference and global BPI score rho-0.56, p<0.0001 OR 1.27 (p=0.012) of impaired sleep quality in those with pain	BPI pain severity correlated with risk of exacerbations (OR 1.65, p=0.04) for exacerbations with higher pain intensity and OR of 2.28 (p=0.008) of death with higher pain intensity	NR	NR	NR	NR
DPAQ-CF	NR	CFQ-R social function (r=0.269, p<0.01), CFQ-R treatment burden (r=0.269, p<0.01), CFQ-R respiratory symptoms (r=0.241, p<0.05), HADS- depression scale (r=0.29, p<0.01) and HADS-anxiety (r=0.29, p<0.01) DPAQ-CF pain intensity correlated with CFQ- treatment burden and respiratory symptoms (p<0.01) DPAQ-CF pain duration correlated with CFQ-R treatment burden and respiratory symptoms (p<0.01)	NR	Correlation of pain and ppFEV1 R=0.239, P<0.05	NR	NR	NR	NR	NR
MPI	NR	Correlation of BPI pain severity and Shwachman scale history scale; r=0.24 (p=0.04) and BPI pain interference and total Shwachman score r=0.2 (p=0.09)	NR	NR	NR	NR	NR	NR	NR
Abdominal
CF-Abd score	NR	NR	Differentiated patients with CF and healthy controls with large effect size (17.3+1.1 versus 8.0 +0.7 points; p<0.001; Cohen's d=0.85)	NR	NR	ICC 0.932 (95% CI 0.874–0.963)	Cronbach's α=0.7–0.92	NR	NR
Gastrointestinal symptom tracker	NR	Nutritional status is related to more stable lung function and fewer exacerbations	NR	NR	NR	Reliability established based on test–retest and internal consistency (unspecified)	NR	NR	NR
Symptom and impact score
MSAS-CF	Developed in accordance with COSMIN recommendations; patients not consulted	MSAS-Resp: correlation with CFQ-R (r=−0.60, p<0.05) and CFQoL (r=−0.7, p<0.05) MSAS-Psych: emotional scale good correlation with CFQ-R (r=−0.69, p<0.05) Poor correlation with MSAS-GI: strongest with weight (CFQ-R r=−0.49, p<0.05) Subscales moderately correlated with symptoms on CFQ-R and CFQoL	Respiratory, gastrointestinal and psychiatric scores were higher in patients with low FEV1 <40% pred (p<0.05)	Correlation with CFQ-R respiratory score (r=−0.6) and CFQoL chest score (r=−0.7, p<0.05) and CFQ-R emotional functioning score (r=−0.69, p<0.05). Weak correlation with CFQ-R digestive score (r=−0.19, p<0.05)	NR	NR	α 0.74–0.86 High in all domains; MSAS-Physical α 0.92, MSAS-Psych α 0.95, MSAS-Global α 0.82	NR	NR	General tool; not specific for exacerbations; originally developed for an oncology population

MCID: minimal clinically important difference; AWESCORE: Alfred Wellness Score; CFRSD: CF respiratory symptom diary; CRISS: chronic respiratory infection symptom score; ReS-CF: respiratory symptoms in CF tool; SOBQ: Shortness of Breath Questionnaire; BPI: brief pain inventory; DPAQ-CF: Daily Pain Assessment Questionnaire in CF; MPI: multiple pain inventory; CF-Abd: CF abdominal; MSAS: Memorial Symptom Assessment Scale; NR: not reported; FEV₁: forced expiratory volume in 1 s; FVC: forced vital capacity; ICC: internal consistency coefficient; ABX: antibiotics ;6MWT: 6-min walk test; QWB: Quality of Well-Being Questionnaire; HADS: Hospital Anxiety and Depression Scale; Resp: respiratory; GI: gastrointestinal; CFQ-R: CF Questionnaire-revised; COSMIN: Consensus-based Standards for the selection of health Measurement INstruments initiative.

Clinical scores

A validation study that evaluated nocturnal cough as an outcome found people with CF coughed more than healthy subjects (p<0.001); the reliability for repeated measurements was higher when cough epochs were scored (multiple coughs with <2 s between individual coughs) compared to discrete coughs (internal consistency coefficient (ICC) 0.75 versus 0.49, respectively) [45].

The interobserver reliability of the modified Schwachman score captured as Pearson's r coefficient was 0.71, 0.64 and 0.85 for the history, examination and growth domains, respectively [26]; the correlation was 0.92 with the NIH score and 0.67 with FEV₁.

The internal consistency of the modified Huang score was reported to be α=0.6 (except the domain relating to complications). The correlation of this score with FEV₁ % pred in moderate (score 35–60, r²=0.3) and severe disease (<35 points, r²=0.43) was greater than in asymptomatic or mild disease [26]. The NIH score was found to be significantly lower in the 5 years before death compared to CF controls who did not die (p<0.001); those with a score between 61 and 70 had a 25% chance of dying within 3 years. The internal consistency of this score was reported to be α=0.81 and the inter-rater (Pearson's r) score was 0.90; this was predominantly attributed to the robustness of the pulmonary domain on subscale analysis [26].

Lung function tests

Low FEV₁ was shown to correlate with death, with a relative risk of death within 2 years of 2.0 (95% CI 1.9–2.2) for each 10% reduction in FEV₁ below the predicted value after adjustment for age and sex [126]. Among people with the same FEV₁, the risk of death was more than double for females compared to males (RR 2.2 (95% CI 1.6–3.1)). FEV₁ was also shown to correlate with QoL; a 5% change in FEV₁ was associated with a change in CFQ-R score from 0.5 to 2.3 points [125].

The RVRTC test demonstrated good test–retest reliability with a coefficient of variation reported to be 2–6%; it differentiated people with CF from healthy controls, including among children aged <6 years [91]. Parameters were shown to improve in children aged between 4 months and 1 year, raising the possibility that lung damage may be reversible during this time [91]. However, RVRTC testing has not been appropriately standardised and consequently has not yet been recommended by authoritative bodies such as the European CF Society Clinical Trial Network as a primary outcome measure for use in CF studies [32].

Measurement of LCI has been found to be reliable, valid and responsive during treatment of pulmonary exacerbations and for monitoring disease progression [23, 37, 89, 110]. Measurements obtained by N₂ washout and by SF₆ were comparable (limits of agreement −2.5 to 1.2) [23]. These tests were found to discriminate between people with CF and healthy controls, as well as those at different disease stages based on age, infection and structural abnormalities identified on high-resolution CT imaging or MRI [110]. A correlation with clinical outcomes has not been established.

Imaging scoring tools

While the Brasfield and Wisconsin CXR scores performed similarly and both have been found to be reproducible (intra-observer agreement r=0.86–0.99 and 0.78–0.96, respectively) and reliable (inter-rater agreement 0.76–0.90 and 0.74–0.97, respectively), they appear to be insensitive to early disease [122]. The correlation between these scores was reported as r=0.86, p<0.0001. Both scores correlated with lung function (FEV₁ and forced vital capacity, all p<0.001) [73]. The correlation of scores with FEV₁ was highest for the Northern score (r=−0.82) compared to the Brasfield (r=0.81) or Crispin–Norman scoring methods (r=−0.83) [124].

CT scoring tools have been found to have higher sensitivity for detecting lung disease progression than FEV₁ % pred. The test–retest reliability based on the intraclass correlation coefficient of the PRAGMA-CF score was shown to be >0.9 for percentage disease, 0.85 for percentage bronchiectasis and 0.96 for percentage air trapping; the intra-observer reliability was >0.90 for bronchiectasis, air trapping and percentage disease [50].

The test–retest reliability of a semi-quantitative MRI score was r²=0.76 (p=0.0047) [54] and correlation with FEV₁ was r=0.81 (p=0.0023) [54].

Functional exercise performance

A summary of the measurement properties of tests used to capture functional exercise capacity is provided in table S6.

Many tests capturing functional exercise performance were compared to cardiopulmonary exercise testing (CPET), which has historically been viewed as the gold standard for assessing exercise capacity according to Von Berg et al. [57]. Rand et al. [47] found that the incremental field step test had acceptable concurrent validity compared to CPET in children for measuring peak oxygen uptake, minute ventilation, heart rate, change in oxygen saturation and CO₂ ventilation and perceived exertion [47].

Submaximal exercise tests included the 6MWT, 3-min step test (3MST), modified shuttle walk test (MSWT) and 30-s or 1-min sit-to-stand test [57]. Good concurrent validity of the MSWT with maximum oxygen capacity on CPET has been reported; however, results for concurrent validity were inconsistent for the 6MWT and 3MST. The ability of the 6MWT to predict pre-transplant survival was variable [36]. A reduction of 50 m or more in the modified shuttle test was associated with a hazard ratio of death or lung transplant within 1 year in adults with CF of 1.91 (95% CI 1.09–3.35, p<0.024) [20]. Convergent validity of 3MST and MSWT with FEV₁ (r=0.61, p=0.002) was found [36], but this was variable for the 6MWT.

CFTR function tests

Intestinal current measurement and nasal potential difference (NPD) tests, which directly measure CFTR function, were strongly correlated and have been found to distinguish people with CF from healthy controls (k=0.83 versus k=0.33, respectively, p<0.001) [68]. Changes in NPD have been reported over 14 days in trials of the CFTR function-modifying drug ivacaftor. Some evidence for the reliability of intestinal organoid volume has been found, but evidence to support its validity has not [68]. Some evidence for the validity and reliability of indirect measures of gastrointestinal CFTR function such as intestinal pH, faecal calprotectin and faecal elastase-1 has been found; however, these data are not described in detail in the review included in our study (table S6).

Sputum tests

Tests characterising sputum rheology, including viscoelasticity and solid content properties, demonstrated poor to fair test–retest reliability with ICCs ranging from 0.22 to 0.42 (with wide confidence intervals) [46, 103]. Reproducibility of biomarkers in the sputum such as total cell count, neutrophils, tumour necrosis factor-α, interleukin-8 and neutrophil elastase was demonstrated in one study [128] as follows: ICC=0.76, 0.82, 0.93, 0.82 and 0.74, respectively; however, there was marked between-patient variability [103, 128].

Measurement error

The systematic and random error of a patient's score not attributable to true changes in the construct that was measured was poorly reported across all studies (table 4, table 5 and table S6).

Discussion

While the measurement properties of PROMs evaluating HRQoL in CF studies have been previously evaluated [135], this is the first effort to systematically review evidence of the measurement properties of all tests and tools used in CF studies. A diverse range of tests and tools were identified which vary with respect to their reliability, responsiveness and validity. There was inconsistency in the use of tests and tools to measure the same or similar outcomes across studies. This highlights the need to establish consensus over which outcomes should be measured in CF studies and how they should be measured; this has been recommended by the COSMIN initiative group [136]. Compared to older tools, many recently developed tools incorporate self-reported outcomes by patients (e.g. CFRSD-CRISS, CFQ-R, CF Impact Questionnaire and CF Quality of Life (CFQoL)) and have involved people with CF in their development, consistent with the recommendation made by the US FDA in 2017 [137].

Evidence to support the reliability of spirometry testing was found; this has also been substantiated in other populations, such as in people with other chronic lung disease [138]. Poor FEV₁ is strongly correlated with death, progression to lung transplant (most transplant recipients have a FEV₁ <30% pred) [139] and reduced QoL [110] in people with CF and is also associated with a greater risk of hospitalisation, pulmonary exacerbations and colonisation with Pseudomonas aeruginosa [140]. Compared to crude or percentage predicted FEV₁ values, z-scores have been proposed as a less biased and more accurate measure for defining meaningful changes in lung function since they take into account sex and ethnicity in addition to age and height; this approach has been endorsed by the Global Lung Initiative since 2013 [141]. This, however, has not yet been universally adopted as the preferred measure for capturing lung function in CF studies. Consensus regarding the MCID for FEV₁ was not identified in this review, but MCIDs have been proposed. In the TRAFFIC and TRANSPORT phase 3 trials, which evaluated lumacaftor–ivacaftor versus placebo for people homozygous for the Phe508del CFTR mutation [142, 143], a mean relative difference of 3.3% (2.3–4.4, p<0.0001) and 2.8% (1.7–3.8, p<0.0001) was found in those with baseline FEV₁ ≥40% pred and baseline FEV₁ <40%, respectively. It was proposed that this represents a clinically significant improvement since the annual rate of decline of FEV₁ % pred has been estimated to be 1.92% per annum for people with CF aged 1824 years (n=2793) and 1.45% for those aged >25 years [144].

While FEV₁ has been shown to be reproducible and repeatable in children aged >6 years and adults, its variability is affected by the person's age and the severity of their underlying lung disease [110]. In the early stages of CF disease, FEV₁ often remains within the normal range, while in severe lung disease FEV₁ is significantly compromised and unlikely to demonstrate variability [89]. LCI testing represents an alternative test for children aged <6 years who are incapable of performing spirometry. Since measurement is dependent on body size, the relative rather than the absolute change is considered more appropriate, at least before 6 years of age [145]. LCI has been shown to correlate strongly with structural abnormalities detected on high-resolution CT and abnormal preschool LCI is associated with spirometry deficits performed within 3 years from baseline in school-age children [146]. However, further standardisation and evaluation of the relationship of LCI with morbidity and mortality is warranted.

Evidence of the reliability, responsiveness and validity of two commonly used QoL tools, the CFQ-R and the CFQoL, as well as the CFRSD-CRISS symptom scoring tool has been reported previously and has been substantiated by this review. The content validity (including face validity) of these tools is strengthened by involving people with CF in their development [26]. The CFQ-R has been shown to correlate moderately with FEV₁ % pred [97].

There have been significant advances in treatment and long-term health outcomes for people with CF in recent decades, which raises a concern about the current content validity of some of the outcome scoring tools developed in the second half of the 20th century, many of which did not involve people with CF in their development [43]. Many of these have not undergone sufficient validation and consequently have not been recommended for use in clinical practice or in research.

The use of imaging modalities and scoring tools in CF has evolved with time; however, considerable variability exists between treatment centres, for example whether to use CXR or CT for longitudinal disease monitoring. An important limitation of CXR imaging is its poor sensitivity for detecting structural lung changes in early disease and progression in those with established disease [73]. This modality, however, is still used for monitoring disease progression in some treatment centres, and it has an established role in identifying pathology in the context of an acute clinical deterioration, such as consolidation or pneumothorax. Extensive collaboration has occurred within the CF community to standardise CT and MRI radiological scores, especially in young children, to enable quantification of the degree of structural lung damage. While CT is currently the most sensitive method for detecting structural airways disease [147], MRI shows promise because it delivers non-ionising radiation and allows assessment of functional aspects of the lung such as perfusion, pulmonary haemodynamics and ventilation [111]. It may be possible to automate imaging scoring algorithms in the future, which may improve the efficiency and reliability of results. However, further assessment of the validity and reproducibility of MRI scoring tools is required, and the extent to which imaging scores predict clinical outcomes of significance requires further elucidation, including in children [111].

The strengths of this review include the use of a systematic approach to identify studies by two independent reviewers. There were four major limitations. First, tests and tools used in practice in people with CF that have been validated in non-CF populations (e.g. generic scores capturing abdominal symptoms) were considered beyond the scope of this review. Secondly, details about the systematic error (bias) and random error (noise) for each of the tests and tools (i.e. variation beyond that attributable to the outcome of interest) have been poorly described in the literature. Measurement error is an important source of bias; this information is necessary to appraise the quality of tests and tools and should be an important factor influencing selection. Thirdly, medical devices used to capture outcomes were beyond the scope of this review (such as weighing scales or stadiometers used to capture anthropometric outcomes). Finally, given the large scope of this review, an exhaustive critique of the measurement properties of individual tests and tools was not feasible.

Conclusions

This systematic review highlights the diversity of tests and tools which have been used for outcome measurement in CF studies and their variable characteristics and properties. While there have been concerted efforts within the CF research community to improve and standardise these tests and tools, further work is needed, particularly to optimise tools for outcome measurement in young children and those with mild or severe disease. A consensus set of tests and tools for measurement in CF studies is needed; this should be developed together with people with CF and other relevant stakeholders. This would likely improve the consistency of reporting and measurement of similar outcomes, allowing comparison and synthesis of evidence across studies and improving the value of the research that is conducted.

Provenance: Submitted article, peer reviewed

Supplementary material

Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.

Supplementary_tables_and_appendices ERR-0354-2020.supplementary_tables_and_appendices ^{(160.8KB, xlsx)}