Raters Interpret Positively and Negatively Worded Items Similarly in a Quality of Life Instrument for Children

Abstract

Measurement invariance is an important assumption to meaningfully compare children’s quality of life (QoL) between different raters (eg, children and parents) and across genders. Moreover, QoL instruments may combine using negatively and positively worded items—a common method to reduce response bias. However, the wording effects may have different levels of impact on different raters and genders. Our aim was to investigate the measurement invariance of Kid-KINDL, a commonly used QoL instrument, across genders and raters and to consider the wording effects simultaneously. Third to sixth graders (208 boys and 235 girls) completed the self-rated Kid-KINDL, and 1 parent each of 241 children completed the parent-rated Kid-KINDL. The wording effects were accounted for by correlated traits-uncorrelated methods model. The measurement invariance was examined using multigroup confirmatory factor analysis. Item loadings and item intercepts were invariant across gender and rater when we simultaneously accounted for the wording effects of Kid-KINDL. Our results suggest that Kid-KINDL could be used to compare QoL across gender and that parent-rated Kid-KINDL could be used to measure children’s QoL. Specifically, the invariant factor loadings across child-rated and parent-rated Kid-KINDL suggest that the score weights in each item were the same for both children and parents (ie, the important items identified by the children are the same items identified by the parents). The invariant item intercepts suggest that both children and parents share the same threshold for each item. Based on the results, we tentatively recommend that each score of a parent-rated Kid-KINDL can stand for each child’s QoL.

Introduction

Quality of life (QoL)—an important and unique patient self-reported assessment of one’s health status—has been researched substantially and has been involved in clinical decision making.^1,2 Health care providers caring for pediatric populations also encourage the use of QoL in clinical evaluations; thus, a wide array of QoL procedures have been developed over the past few years.^3,4 Among the variety of QoL instruments for pediatric populations, KINDL is a multidimensional measurement of QoL, with physical, psychological, and social health subdomains. It is a feasible, reliable, and valid tool to measure an individual’s QoL in line with the definition given by the World Health Organization.^5-9

Several advantages of KINDL have been concluded by Lin et al.¹⁰ The inclusion of both positively worded (eg, having fun and laughing a lot) and negatively worded (eg, feeling ill) items distinguishes KINDL from other pediatric QoL instruments. For example, Pediatric Quality of Life Inventory (PedsQL)—widely used in pediatric research—consists of negatively worded items only. In this case, participants may give biased reports because of response acquiescence and social desirability. Using both positively and negatively worded items in a single questionnaire can reduce ceiling effects, floor effects, and response bias.^11,12 Despite these potential benefits, the coexistence of positively and negatively worded items may threaten the internal consistency and validity of the instruments if wording effects are not considered.¹³ However, the threat of wording effects on construct validity can be minimized and controlled by using a multitrait-multimethod (MTMM) approach combined with confirmatory factor analysis (CFA) models.¹⁴

The MTMM plus CFA approach compares several CFA models, where each model has its unique underlying construct.¹⁴ In general, three types of models are compared, including positively worded and negatively worded items. We take the KINDL as an example as follows: (1) a conventional 6-QoL-factor model depending on the 5 dimensions from the KINDL, (2) a wording-method effect model, and (3) a model that takes into account both trait (dimensions in KINDL) and wording-method (positive and negative wording) factors.¹⁰ Because the third type of model assists in separating method effects (an effect that is distinct from but may interfere with the trait effects) on the constructs of the instruments,^15-17 it can be used to verify construct validity of the instruments that have poor psychometric properties due to the wording effect. A previous study has shown that CFA models belonging to the third type¹⁰—especially the correlated traits-uncorrelated methods (CTUM) model (Figure 1)—have satisfactory fit indexes and outperform the other 2 types of models in the KINDL. Consequently, the KINDL has solid construct validity.

Figure 1.

Correlated traits-uncorrelated methods framework represents 6 quality of life factors (physical, emotional, self-esteem, family, friends, and school) and 2 orthogonal wording factors (positive and negative wording).

Note. Negatively worded items are in bold font, and linked with Negative Wording concept by dashed lines.

In addition to wording effects, factorial invariance (aka measurement invariance) is another important issue when developing and validating QoL instruments.^18,19 Before using an instrument made up with latent structures, a prerequisite is to examine the equivalence of the underlying constituent factors across different groups.^20,21 The cross-gender and rater (viz, parents and children) differences are particularly crucial for the use of pediatric QoL instruments.^22,23 Researchers need to ensure that boys and girls, and parents and children, have concordant interpretations of the pediatric QoL instrument.

Family or culture usually has different expectations for boys and girls;^24,25 thus, they may have different perspectives toward the same concepts. To prevent the discrepant interpretation of QoL constructs because of gender, we need to ensure that the cross-gender measurement invariance before comparison could be made between the QoL scores rated by different gender groups. Furthermore, health care providers sometimes need parents to rate their children’s QoL when these children are too sick to report by themselves.¹⁸ Health care providers may use the parent-rated scores for decision making. However, a wrong decision may be made if factorial invariance of a pediatric QoL instrument across parents and children does not exist.

Studies examining the factorial invariance of the KINDL questionnaire across different groups are sparse. To the best of our knowledge, only Jafari et al²⁶ have recently examined the cross-rater factorial invariance between parents and children. No studies have discussed the role of gender in the measurement invariance of the KINDL questionnaire. Furthermore, no studies have simultaneously examined the wording effects and measurement invariance in the use of KINDL. Therefore, the purposes of the present study were as follows: (1) to examine the measurement invariance of Kid-KINDL across gender and (2) to examine the measurement invariance of Kid-KINDL across raters, simultaneously considering the wording effects.

Methods

Participants and Procedures

The Institutional Review Board of the National Cheng Kung University Hospital approved this study, and the data were collected from March to May 2010. Using a convenience sampling, the first author invited third- through sixth-grade students in 10 different schools (6 urban and 4 suburban) to participate in the study. All students completed a child-reported Kid-KINDL (version of KINDL questionnaires for children aged 8-12 years) in their classrooms under the supervision of their homeroom teachers and the first author. All parents filled out a parent-reported Kid-KINDL at home; the questionnaire was taken home by their children. The author collected the parent-reported Kid-KINDL from the students 1 to 3 days after each student filled out the child-reported Kid-KINDL.

Instruments

The Kid-KINDL questionnaire contains 6 dimensions (viz, physical well-being, emotional well-being, self-esteem, friends, family, and school), each of them including 4 items (Supplementary Table S1). The questionnaire was translated into Chinese for a pediatric population aged 8 to 12 years in Taiwan.¹⁰ The Kid-KINDL can be obtained from the developers (www.kindl.org). Each item in the test is 5-point Likert-scaled (positively worded items: never scores 1 and always scores 5; negatively worded items: never scores 5 and always scores 1).⁷ From the item scores, the total Kid-KINDL score and each dimension score can be calculated and transformed into a 0 to 100 result where a higher score represents better QoL.⁷ Parallel child-reported and parent-reported Kid-KINDLs have been developed. Both questionnaires show acceptable psychometric properties,^5,27 including the Chinese version of Kid-KINDL for Taiwanese children.¹⁰ Although studies found that some dimensions, including simultaneously both positively and negatively worded items, demonstrate unsatisfactory internal consistency,^7,8,28-30 some researchers argue that the unsatisfactory internal consistency does not necessarily threaten the construct validity of the Kid-KINDL.¹⁰ However, it has been suggested that one item (Fr4: felt different from other children) be deleted from the Taiwanese version of Kid-KINDL due to its different meaning in Taiwanese and German, in which the original Kid-KINDL was developed.¹⁰

Data Analysis

Multigroup CFA was conducted to test the measurement invariance. CTUM models were used to examine invariance across genders (boys and girls) and raters (parents and children). Because fathers and mothers may also have different interpretations toward the same item description, we further performed sensitivity analyses to test the measurement invariance across mothers and children. We used only mothers for the examination because parent-proxies are often completed by mothers; the number of fathers was not sufficient for conducting CFA (n = 55). Four baseline CTUM models were conducted separately for boys, girls, parents, and children to examine the goodness of fit of the models to the data for each group. The CTUM models separate the underlying concepts of each item into two parts: trait and method. The traits in the CTUM models are the QoL concepts in the Kid-KINDL (physical well-being, emotional well-being, self-esteem, friends, family, and school). The methods in the models are the wording effects (the positively and negatively worded items). The 6 QoL concepts were correlated in the model; the methods were not correlated (Figure 1). Moreover, the QoL concepts and the method effects were not correlated in the model because we assumed that QoL concepts and method effects are different and uncorrelated underlying constructs.

The following fit indexes were used to evaluate the fit of our CTUM models: the acceptable fit indexes according to published guidelines, a goodness of fit index (GFI), a nonnormed fit index (NNFI) and a comparative fit index (CFI) > 0.9. A root mean square error of approximation (RMSEA) and a standardized root mean square residual (SRMR) < 0.08 indicate acceptable model fit.^31,32

We considered 4 kinds of measurement invariance in our analyses: configural, metric invariance, scalar, and uniqueness.²¹ Configural invariance indicates conceptual equivalence of the underlying theoretical factors across groups, metric invariance suggests equivalent associations between items and latent factors across groups, scalar invariance means the same origins across groups (the groups have their scores rated unbiased), and uniqueness invariance suggests that items are influenced to the same degree and perhaps by the same unique factors across groups.²¹

Based on the suggestions mentioned above, 3 configural CTUM models (1 for gender and 2 for rater groups) were performed to test the equivalence of the overall structure (6 correlated QoL traits and 2 uncorrelated wording effects). In addition, 3 nested models were compared with each other; each model includes the restrictions from the previous models with the additional constraints of equal factor loadings, equal item intercepts, and equal item uniqueness across gender and rater groups. When comparing these nested models, Δ values >−0.01 for CFI and <0.015 for the RMSEA indicate that the null hypothesis of invariance should not be rejected.³³ Although some researchers^19,34-36 state an alternative Δ value >−0.02 for CFI, we intended to use ΔCFI values >−0.01 for measurement invariance.

LISREL 8.8 (Scientific Software International Inc, Lincolnwood, Illinois) was used to perform CFAs; other analyses were done using SPSS 16.0 (SPSS Inc, Chicago, Illinois).

Results

The mean age of the children (n = 443) was 10.57 (SD = 1.08) years; 47.0% of them were boys. The mean ages of fathers and mothers were 43.62 (SD = 5.70) and 40.81 (SD = 5.18) years, respectively. Nearly 30% of the fathers had senior high education (29.5%) and college (28.7%), while more than 30% of the mothers had senior high education (38.2%) and college (32.3%) (Table 1). Also, as we anticipated, most parent-proxies were completed by mothers (n = 176, 73.0%).

Table 1.

Demographics of Children and Their Parents.

	Children (n = 443)	Father (n = 185)a	Mother (n = 183)a
Age, mean ± SD, y	10.57 ± 1.08	43.62 ± 5.70	40.81 ± 5.18
Gender, n (%), male	208 (47.0)	173 (100.0)	0 (0.0)
Grade, n (%)
Third	97 (21.9)	—	—
Fourth	116 (26.2)	—	—
Fifth	150 (33.9)	—	—
Sixth	80 (18.1)	—	—
Education, n (%)
Elementary	—	7 (2.9)	1 (0.4)
Junior high	—	12 (5.0)	6 (2.5)
Senior high	—	71 (29.5)	92 (38.2)
College	—	69 (28.7)	78 (32.3)
Graduate school	—	26 (10.8)	6 (2.5)

^aWith missing values on demographic data.

Except for 1 nonsignificant factor loading (item S3 in parent groups; P = 0.19), all other item factor loadings were significant in the baseline CTUM models among all groups: boy (n = 208), girl (n = 235), parent (n = 241), mother (n = 176), and children (n = 443) (Supplementary Table S2). Furthermore, the baseline CTUM models show satisfactory or approximately acceptable data-model fit for all groups, including global configural models in gender (GFI = 0.894 to 0.900, NNFI = 0.948 to 0.965, CFI = 0.960 to 0.974, RMSEA = 0.0476 to 0.0491, SRMR = 0.0547 to 0.0609) and rater groups (GFI = 0.882 to 0.934, NNFI = 0.965 to 0.970, CFI = 0.974 to 0.977, RMSEA = 0.0438 to 0.0478, SRMR = 0.0452 to 0.0535), except for the significant χ² values (Supplementary Table S3).

Based on the good fit indexes in each CTUM model, the following measurement invariance tests were affirmative. Measurement invariance was supported by item loadings, item intercepts, and item uniqueness across gender (ΔCFI = −0.007 to −0.001 and ΔRMSEA = −0.0004 to 0.0029). Item loadings and item intercepts were invariant across raters (ΔCFI = −0.014 to −0.009 and ΔRMSEA = 0.0057 to 0.0079 using all parents’ parent-proxies; ΔCFI = −0.014 to −0.008 and ΔRMSEA = 0.0058 to 0.0062 using mothers’ parent-proxies only), but item uniqueness was not (ΔCFI = −0.062 and ΔRMSEA = 0.0199 using all parents’ parent-proxies; ΔCFI = −0.045 and ΔRMSEA = 0.0263 using mothers’ parent-proxies only) (Table 2).

Table 2.

Measurement Invariance Across Gender and Raters.

	χ2	Δχ2	df	Δdf	CFI	ΔCFI	ΔRMSEA	RMSEA
Gender
Model 1G	622.433*	—	384	—	0.969	—	0.0484	—
Model 2G − 1G	—	111.15*	—	61	—	−0.007	—	0.0022
Model 3G − 2G	—	21.27	—	15	—	−0.001	—	−0.0004
Model 4G − 3G	—	58.88*	—	23	—	−0.004	—	0.0029
Rater
Model 1R	663.802*	—	384	—	0.976	—	0.0444	—
Model 2R − 1R	—	224.80*	—	61	—	−0.014	—	0.0079
Model 3R − 2R	—	122.89*	—	15	—	−0.009	—	0.0057
Model 4R − 3R	—	746.11*	—	23	—	−0.062	—	0.0199
Ratera
Model 1M	653.840*	—	384	—	0.975	—	0.0456	—
Model 2M − 1M	—	209.68*	—	61	—	−0.014	—	0.0062
Model 3M − 2M	—	109.14*	—	15	—	−0.008	—	0.0058
Model 4M − 3M	—	129.76*	—	23	—	−0.045	—	0.0263

Note. Models 1G, 1R, and 1M: Configural model. Models 2G, 2R, and 2M: Constrained all item loadings equal across gender (Model 2G) and raters (Models 2R and 2M). Models 3G, 3R, and 3M: Constrained all item loadings and item intercepts equal across gender (Model 3G) and raters (Models 3R and 3M). Models 4G, 4R, and 4M: Constrained all item loadings, item intercepts, and item uniqueness equal across gender (Model 4G) and raters (Models 4R and 4M). Models 2G, 3G, and 4G were based on Model 1G with additional constraints; Models 2R, 3R, and 4R were based on Model 1R with additional constraints; Model 2M, 3M, and 4M were based on Model 1M with additional constraints. CFI = comparative fit index; RMSEA = root mean square error of approximation.

^aParent-proxies only used mothers’ reports.

^*p < .05.

Discussion

This study examines the measurement invariance of the Chinese version of the Kid-KINDL across genders and raters among Taiwanese children. The most important finding was that the underlying concepts of Kid-KINDL (QoL-trait and wording-method effects)^7,10 are equivalent across boys and girls, and across parents and children. Although we found that item uniqueness was not invariant across raters, this finding does not contradict our conclusion that underlying concepts of Kid-KINDL are equivalent across parents and children. Item uniqueness indicates the measurement errors of each item, which is not essential for measurement invariance.

Our results show that boys and girls in Taiwan have similar interpretations of the items on the Kid-KINDL^7,10 under the same impact of wording effects. This indicates that they can capture the concepts of Kid-KINDL in a very similar way, thus making their scores comparable across genders, instead of viewing them separately. Studies have often used Kid-KINDL to measure and analyze children’s QoL without distinguishing between boys and girls.^5,28-30 Our current results support that the merged use is reasonable. Although our results seem to be the first to detect the measurement invariance across genders for Kid-KINDL, our results could somewhat correspond to other studies which have found measurement invariance in different QoL instruments, such as PedsQL.¹⁸ Our results also show satisfactory data-model fit in the CTUM model, which agrees with the statement that the wording effects exist in Kid-KINDL.^10,37

Several researchers state that standardized QoL instruments should explore whether the test items are interpreted similarly across age, gender, language, socioeconomic, and race/ethnicity subpopulations.^20,36,38 Confirmation of strict levels of factorial invariance, as provided by the present study, is especially suggested. That is why factorial invariance studies should be repeated periodically due to the change of subpopulations over time.²⁰ Due to the characteristics of our sample—sample size was not large enough for participants with different ages, our data did not include socioeconomic information, all participants used the same language and were the same race/ethnicity—we could only test the measurement invariance across gender. Future studies on other factors are needed.

However, our study provided information on the factorial invariance of another important factor: the rater. Some evidence shows that children aged 3 to 8 years can understand response terms and underlying concepts of QoL.³⁹ They are able to self-rate their QoL, but valuable supplementary information for health care providers can be gathered by parent reports on QoL. For example, a parent-rated QoL proxy can be used as a substitute for a child who is too young or too ill to complete a self-report.³⁴ In addition, health care providers can evaluate the agreement of children’s QoL between parents’ and children’s perceptions and use this information on agreement to improve the children’s QoL.^24,40-42 Our results support the assumption of measurement invariance across children and parents for the Kid-KINDL; therefore, the use of a parent-rated Kid-KINDL is justified. As a result, health care professionals may want to use Kid-KINDL for their clinical decision making.

Nonetheless, health care professionals should still use the parent-rated Kid-KINDL with caution because other studies^5,26 on KINDL demonstrated results that were inconsistent with ours. For example, Erhart et al⁵ concluded that parent-rated KINDL achieved slightly higher reliability; Jafari et al²⁶ found that 12 KINDL items may be noninvariant across children and parents. Two reasons may explain the different results: First, Erhart et al⁵ and Jafari et al²⁶ did not consider the wording effects for KINDL and may have their results somewhat different from ours; second, Jafari et al²⁶ used different methods from ours (item response theory [IRT] and ordinal logistic regression [OLR] methods) to examine the factorial invariance; Erhart et al⁵ did not conduct a factorial invariance analysis. Because different methods on factorial invariance may have different findings,^26,43 future studies examining factorial invariance with different factorial invariance methods on a large sample size are warranted. Specifically, IRT methods usually need a sample size over 500 to achieve the correct estimations.⁴⁴

In addition, the 3 methods on factorial invariance (viz, IRT, OLR, and multigroup CFA) have their own advantages and disadvantages. For IRT and multigroup CFA, both share the advantage of nonuniform factorial invariance; that is, they estimate both item intercepts and factor loadings for invariance, while OLR cannot.^26,45 Both OLR and multigroup CFA have the advantage of adjusting confounders in the model; that is, they can allow additional variable in the model, while IRT cannot.²⁶ Multigroup CFA, however, treats the items as continuous variables, which is apparently not the case in the Kid-KINDL that adopts Likert-type scales.²⁶

This study has some limitations. First, our sample may not be representative of the pediatric population in Taiwan because we did not recruit children with any chronic conditions. Therefore, our results may not be generalized beyond healthy children. Second, the measurement invariance was only examined for CTUM models, while there are different kinds of models that can serve to examine the construct validity that account for wording effects in addition to the CTUM model, such as the correlated trait-correlated methods model and correlated trait-correlated methods minus one model. We only demonstrated that boys and girls, and parents and children, understand the items of the Kid-KINDL and understand the construct behind the Kid-KINDL in the same way under the CTUM model. We cannot guarantee that our results can be generalized to other models accounting for wording effects. However, the CTUM model was found to be the best model among all models accounting for wording effects.¹⁰ We justified that only the test in the CTUM model is appropriate. Third, the examination of factorial invariance across parents and children did not consider the gender of the responding parent.

Although our results support the measurement invariance across boys and girls, the inference of gender invariance of the parents should be verified before applying to fathers’ and mothers’ report. Therefore, the merged use of fathers’ and mothers’ reports as a single proxy rating to represent the parents’ rating may weaken the verification of measurement invariance across parents and children. Therefore, we examined the measurement invariance across mothers and children and showed that the measurement invariance was supported. However, sample size was not sufficient in our study to test the measurement invariance across fathers and children. Therefore, additional studies to examine the measurement invariance across fathers and children in Kid-KINDL are highly recommended.

Researchers may also want to test the measurement invariance across fathers and mothers. Following the third limitation, the small sample size in our study should be noted. The recommended sample size for CFA is 200 per group, which indicates that our mother-rated Kid-KINDLs were slightly fewer than the sufficient number. Finally, the Kid-KINDL is designed for children 8 to 12 years of age; findings for other age versions of KINDL questionnaires could differ from our results. Future research on other age versions of KINDL questionnaires (eg, Kiddo-KINDL for children 13 to 16 years of age) is thus required.

Conclusion

Despite the limitations, this study demonstrated that Kid-KINDL exhibits good psychometric properties, including wording effects and measurement invariance across gender groups and between parents and children. This study provided support for clinicians to use Kid-KINDL among boys and girls. More importantly, this study provides a foundation to use parent-reported QoL in necessary situations, when children are unable to report by themselves. Moreover, the tested Kid-KINDL is a 23-item Chinese version; researchers may want to know whether the original 24-item version has similar findings to ours.