Original quantitative research Validation of the Children’s Intrinsic Needs Satisfaction Scale among Canadian youth: psychometric properties, criterion-related validity and multitrait multimethod confirmatory factor analysis

Abstract

Introduction:

Based on self-determination theory, the Children’s Intrinsic Needs Satisfaction Scale (CINSS) measures autonomy, competence and relatedness at school, home and with peers. The factor structure and criterion-related validity of the CINSS in the Canadian youth population are tested using data from the Canadian Student Tobacco, Alcohol and Drugs Survey (CSTADS).

Methods:

Data from the 2014/2015 CSTADS were analyzed for evidence of convergent and discriminant validity and for method variance. A multitrait multimethod (MTMM) confirmatory factor analysis (CFA) was conducted to account for the conceptual structure of the measure. Criterion-related validity was demonstrated through correlations between related constructs, prosocial behaviours and behavioural problems, and the CINSS subscale scores. Mean differences on CINSS subscale scores between those who reported and did not report being bullied or bullying others were also examined.

Results:

Correlation analyses demonstrated that, in general, correlations were higher between concept/context item pairs and lowest between items measuring different concepts and contexts. Cronbach’s alpha for concept and context subscales were high: α = 0.77 for autonomy, α = 0.85 for competence and α = 0.79 for relatedness. A MTMM CFA demonstrated that the model fit the data well, with no modifications. Criterion-related validity was demonstrated through correlations between CINSS subscales and related concepts or mean differences on CINSS subscales between groups.

Conclusion:

The CINSS demonstrates good internal consistency, factorial validity and criterion-related validity in this sample of Canadian students. The measurement of positive mental health among Canadian youth is central to surveillance efforts which will help inform mental health promotion activities across Canada.

Highlights

• The factor structure of the Children’s Intrinsic Needs Satisfaction Scale (CINSS) was confirmed using multitrait multimethod confirmatory factor analysis.

• The CINSS tool showed good internal reliability as well as criterion-related validity through correlations between subscales and related constructs.

• The CINSS tool is a promising measure of positive mental health among Canadian youth.

Introduction

The mental health and well-being of children and youth is a public health priority in Canada¹ and in many countries around the world.^2,3 The Public Health Agency of Canada (PHAC) defines positive mental health as “the capacity of each and all of us to feel, think, and act in ways that enhance our ability to enjoy life and deal with the challenges we face.”⁴ Similarly, well-being is defined as “good mental states, including all of the various evaluations, positive and negative, that people make of their lives, and the affective reactions of people to their experiences.”⁵ However, the data on the mental health and well-being of children and youth in Canada, especially at the population level, are limited. Furthermore, mental health promotion approaches acknowledge a need to focus on strengths and resources to promote positive mental health for all children and youth, rather than on deficits and stress.⁶ Strengths-based approaches focus on “what is going right,” as a foundation for promoting the well-being and health of the population.

Surveillance efforts in the area of mental health have traditionally focused on what is going wrong, using measures of disorder, distress and problematic behaviour. For example, the Strengths and Difficulties Questionnaire (SDQ), developed as a short behavioural screening questionnaire for use in clinical settings, program evaluation and surveillance, includes only one subscale that deals with the positive attributes of the child or youth.⁷ The Child Behaviour Checklist, one of the most widely used measures in child mental health, also focusses on problematic behaviours, including anxious, depressed, withdrawn and aggressive behaviours.⁸ For public health to focus on strengthsbased approaches and positive mental health, foundational measurement and surveillance efforts of these constructs are required.

To measure positive mental health among Canadians, PHAC developed the Positive Mental Health Surveillance Indicator Framework (PMHSIF), which includes three positive mental health outcomes: emotional, psychological and social wellbeing. ⁹ Positive mental health outcomes are measured by self-rated mental health, happiness, life satisfaction and psychological well-being in adults,¹⁰ and a sense of community belonging for social wellbeing. However, measures that are appropriate for adults are often not appropriate for children or youth due to developmental and other differences.

Few measures of positive mental health for youth and children have been implemented on large-scale Canadian surveys. These include the Strengths and Difficulties Questionnaire on the Canadian Health Measures Survey (CHMS), the Health Utilities Index’s single item on emotion (CHMS) and the Children’s Intrinsic Needs Satisfaction Scale (CINSS). The CINSS was implemented for the SHAPES Mental Fitness survey component,¹¹ as well as the 2014/2015 Canadian Student Tobacco, Alcohol and Drugs Survey (CSTADS).¹² After comparing these measures, the CINSS was chosen as a strong candidate measure of positive mental health among children and youth because it covered two of the three positive mental health outcomes included in the PMHSIF.

The CINSS is based in self-determination theory (SDT), a theory of motivation and personality that proposes the basic psychological human needs of autonomy, competence and relatedness.^13-16 Autonomy is the experience of choice for one’s activities; competence is a feeling of mastery and self-efficacy; and relatedness is a feeling of closeness with significant others. These concepts map onto the constructs of psychological well-being (autonomy and competence) and social well-being (relatedness) in the PMHSIF. These three concepts are also hypothesized to contribute to emotional well-being. Situations or environments that support individual autonomy, competence and relatedness are predicted to lead to greater subjective well-being and happiness.¹³ On the other hand, situations that do not support these basic needs may lead to decreased well-being.

Although autonomy, competence and relatedness in the context of well-being have been widely studied in adult populations, children and youth remain an inadequately studied population. In order to measure the core concepts of self-determination theory among children and youth, Véronneau et al. (2005) developed the CINSS.¹⁷ The scale’s 18 questions cover the three concepts (competence, relatedness, autonomy) each across three contexts (with home, school and peers). While this measure has been implemented on the School Health Planning and Evaluation System and the CSTADS,^11,12there is limited information about its reliability and validity, and we were unable to identify any published reports of its factor structure.

The purpose of this paper is to test the factor structure and construct validity of the CINSS in the Canadian youth population using data from the CSTADS. Convergent validity is demonstrated by high correlations among measures of the same trait by the same method, and discriminant validity is demonstrated by relatively lower correlations between different traits and different methods. We anticipate that CINSS subscales will be positively associated with prosocial behaviour and negatively associated with behavioural problems, bullying and being bullied.

Methods

Data from the 2014/2015 CSTADS—obtained through a sharing agreement with Health Canada and PHAC—were analyzed. The CSTADS is a school-based survey conducted every second year, with funding from Health Canada. The target population for the 2014/2015 cycle was students in Grades 6 to 12 (Secondary 5 in Quebec) who were attending public, private and Catholic schools in the 10 provinces in Canada. The three territories were excluded. During this cycle, schools on military bases and on First Nations reserves and schools for youth with visual or hearing impairments or special needs were also excluded. While New Brunswick was included in the sample, estimates could not be calculated for this province due to a very low participation rate. A stratified single-stage cluster design, based on the smoking rate for the health region as well as type of school, was used, with schools randomly selected within strata. The final sample included 336 schools from 128 school boards, with a schoollevel response rate of 47%; 42 094 students responded, representing a 66% student-level response rate. The paper and pencil survey, which took less than 30 minutes to complete, was administered in classrooms, in English and French, between October 2014 and May 2015.

Ethics approval was granted by the Health Canada and the PHAC’s Research Ethics Board, the Office of Research Ethics at the University of Waterloo, the provincial institutional ethics review boards and the ethics boards of participating school boards. Consent procedures varied by school board and included active permission or active information/passive permission by home and guardians. Participation in the survey was at the students’ discretion.

Measures

The CINSS consists of 18 questions,¹⁷ which respondents answer using a fourpoint Likert scale ranging from 1 (“Really false for me”) to 4 (“Really true for me”). There were six questions per concept (autonomy, competence and relatedness) and six questions per context (with peers, at home and at school), for a total of two questions per context/concept pair. Subscale scores were created by summing responses to each of the six questions for the autonomy, competence and relatedness subscales. Sample questions and their corresponding concepts and contexts include: “My teachers like me and care about me” (relatedness/school); “I feel free to express myself at home” (autonomy/ home); and “I feel my friends think I am good at things” (competence/with peers). If a respondent was missing data on an item, the value for that subscale was not computed.

For measures of criterion-related validity, correlations between the CINSS total score and subscale scores were examined for the following modules: bullying, prosocial behaviours and behavioural problems.

Bullying

Bullying and being bullied were based on questions about frequency over the past 30 days. Students were asked “1) In the last 30 days, how often have you been bullied by other students?” and “2) In the last 30 days, how often have you bullied other students?”

Students were grouped according to whether they reported 1) being bullied (responses b through e) or not bullied (response a), and whether they 2) reported bullying others (responses b through e) or not bullying others (response a):

a. I have not been bullied by/have not bullied other students in the last 30 days;

b. Less than once a week;

c. About once a week;

d. 2 or 3 times a week;

e. Daily or almost daily.

Prosocial behaviours

The prosocial behaviour scale of the Health Behaviours in School Aged Children Brief Symptom Checklist ¹⁸ was measured using the following five positive statements:

a. I often do favours for people without being asked;

b. I often lend things to people without being asked;

c. I often help people without being asked;

d. I often compliment people without being asked;

e. I often share things with people without being asked.

Behavioural problems

The behavioural problems scale of the Health Behaviours in School Aged Children Brief Symptom Checklist ¹⁸ was measured using the seven following negative statements:

a. I cut classes or skip school;

b. I make other people do what I want;

c. I disobey my parents;

d. I talk back to my teachers;

e. I get into fights;

f. I often say mean things to people to get what I want;

g. I take things that are not mine from home, school, or elsewhere.

Responses to the prosocial behaviours and behavioural problems scale statements were chosen from a six-point Likert scale that ranged from 1 (“definitely not like me”) to 6 (“definitely like me”). The responses were summed for all questions answered, then divided by the number of questions answered for all respondents who answered three or more questions.

Demographic variables

Demographic variables included sex and grade. Ethnicity was based on the question “How would you describe yourself?” Response categories included: White, Black, West Asian/Arab, South Asian, East/Southeast Asian, Latin American/ Hispanic, Aboriginal and “Other.” School socioeconomic status was determined by using median household income from the 2011 Census of the households in the school’s forward sortation area (first three digits of the postal code). Urban/rural status was also determined using the school’s postal code and the Statistical Area Classification system variable from the Postal Code Conversion File + version 6a1.

Analysis

Descriptive statistics and correlation analyses were conducted in SAS EG (SAS Institute Inc., Cary, NC, USA). Descriptive analyses were stratified by sex.

Multitrait multimethod (MTMM) confirmatory factor analysis (CFA) was conducted to test the structural validity of the CINSS. The MTMM approach to CFA makes it possible to measure several traits using a number of methods.¹⁹ In this case, we used MTMM CFA to test the factor structure of three “traits” (autonomy, competence and relatedness) as measured in three “method” contexts (home, school and with friends). This method is required to take into account the conceptual structure of the measure, which includes both traits and contexts, whereas a standard CFA would only take into account either the traits or the contexts, but not both simultaneously. Our analytic plan involved testing two models, as described by Marsh and Grayson:²⁰ correlated traits correlated methods (CTCM) and correlated traits correlated uniqueness (CTCU). The CTCM estimates factors for traits and methods, as well as correlations among traits and among methods, separately (i.e. does not estimate correlations between traits and methods). An advantage of this model is that it closely resembles what MTMM strives to achieve, theoretically; however, it rarely converges in practice because the model is usually underidentified.²¹ The CTCU, which provides an estimate of trait factors and intercorrelations among the method item residuals, is normally tested as a backup.

To assess model fit, we used Hu and Bentler’s ²² criteria for adequate fit: a value of 0.95 or greater for the comparative fit index (CFI) and Tucker-Lewis index (TLI); a value of less than 0.08 for the standardized root mean square residual (SRMR); and a value of less than 0.06 for root mean square error of approximation (RMSEA). All analyses were weighted to account for the sampling design of the CSTADS and bootstrapping was used to obtain standard errors for descriptive statistics.¹²

Results

Study participants included youth in Grades 6 through 12, with about 15% of the sample in each of these grades (Table 1). The sample comprised 51.4% males and 48.6% females. All provinces were represented proportionally except for New Brunswick, which comprised less than 1% of the sample. The median household income of the neighbourhood in which a school was located was $66 509, and 79.8% of schools were located in urban settings. While most students identified as White (66.5%), considerable diversity is apparent, with 10.8% of students identifying as South Asian/ Indian, 8.7% as East Asian/Chinese, 5.3% as Black, 4.2% as Aboriginal, 3.2% as Asian/Arab, 2.6% as Latin American/ Hispanic and 7.3% as “Other.”

Table 1. Description of sample composition (N = 42 094).

Table 2 shows descriptive statistics for CINSS items reported separately/split by sex and for males and females combined. All items had relatively high mean responses on the four-point scale. Item and subscale means were similar for males and females. There were low levels of missing data, ranging from 3% to 5% per item. Cronbach’s alpha for the three domain subscales were as follows: autonomy α = 0.77; competence, α = 0.85; and relatedness α = 0.79. The appropriate level of alpha depends on the intended use of the measure, with an alpha of at least 0.70 being adequate for most research and 0.80 for most applied purposes.²³

Table 2. Means, standard deviations and percent missing data by item, Children’s Intrinsic Needs Satisfaction Scale, 2014/15 (N = 42 094).

Table 2 shows mean scores by concept and context for both sexes combined and for females and males separately. There were no statistically significant differences in subscale scores for females and males, with the exception of relatedness, where mean scores were 3.41 (95% CI: 3.40–3.41) and 3.34 (95% CI: 3.34–3.35), respectively.

Table 3 shows the correlation matrix for CINSS items. Convergent validity was demonstrated by higher correlations for same trait, same context items. In general, correlations were highest between trait/ context pairs (e.g. relatedness with friends, r = 0.63) and between trait items (e.g. competence) and context items (e.g. at school). However, item pairs for autonomy at home (r = 0.34) and at school (r = 0.35) were lower than expected. Discriminant validity was demonstrated through lower correlations between different trait, different context items (e.g. having a choice over when to do chores (autonomy at home) and likes to be with friend (relatedness with peers, r = 0.21).

Table 3. Correlation matrix of items of the Children’s Intrinsic Needs Satisfaction Scale, 2014/15 (N = 39 734–40 650).

As tests of criterion-related validity, we also examined the relative strength of the relationships between the CINSS subscales with prosocial behaviour and problematic behaviour measures (Table 4). Relatedness was most highly correlated with problematic behaviour (r = −0.33), as was competence (r = −0.31) (Table 4). Scores on all subscales were significantly lower among those reporting being bullied in the past 30 days compared to those who did not report this, and among those reporting bullying others in the past 30 days compared to those who did not report this. Those who reported being bullied, or bullying others, had scores approximately one-quarter point lower than those who were not bullied or did not bully others.

Table 4. Subscale means by being bullied or bullying in the past 30 days, and correlations between subscale scores and prosocial behaviours and problematic behaviours.

The CTCM model was tested in MPlus²⁴ using robust maximum likelihood estimation. As anticipated, the model did not converge. The CTCU model, however, fit the data well: the Chi-square test value for the model was 75 545.40 (df = 153; p < 0.001). The CFI exceeded Bentler and Hu’s criterion at 0.97; TLI of 0.94 was slightly lower than the 0.95 cutoff. RMSEA was 0.026 (95% CI: 0.025–0.027; p < 0.001) and SRMR was 0.028. All factor loadings were significant, with standardized values ranging from .49 to .74. Statistically significant intercorrelations between item residuals ranged from r = .04 to r = .45. Correlations were highest among item residuals measured for home and peer contexts and lowest for the school context, where five item residual correlations were not significant. No modifications were needed to obtain an adequate fit according to the criteria set by Hu and Bentler.²² High correlations among the trait factors (raut-rel = .91; raut-comp = .90; rrel-comp = .95) suggested that a second order factor might better represent the factor structure of the data. When relatedness, autonomy and competence factors were loaded onto a second-order factor of basic psychological need satisfaction, the factor variance was significant (Z = 6.49, p < .001). Factor loadings for items and first order factors are presented in Table 5.

Table 5. Standardized factor loadings for the correlated traits–correlated uniqueness model.

Discussion

These analyses of the 2014/2015 CSTADS support the factorial and criterion-related validity of the CINSS scale. In this sample of youth in Grades 6 to 12, mean levels of autonomy, competence and relatedness were relatively high, with item means ranging from 2.84 to 3.74 on a four-point scale and subscale mean scores of 3.23 for autonomy, 3.35 for competence, to 3.37 for relatedness. There were no substantive differences between females and males, although the difference in the relatedness scales between females and males was small but statistically significant. The level of missing data was relatively consistent between items, ranging from 2% to 5% depending on the item and sex. We observed similar levels of internal consistency as Véronneau et al. in their initial study in Montréal,¹⁷ with Cronbach’s alphas in the high 0.70s to mid 0.80s for the subscales.

The pattern of correlations largely supported the MTMM measurement structure of the CINSS, although the autonomy subscale had lower item-pair correlations than anticipated. Autonomy is measured in each context by two questions: the first is about expressing oneself; the second is about choosing when to do schoolwork, homework or activities with friends. It may be that expression and choice of activities are both conceptually related to autonomy, but reflect different aspects. It may be worthwhile to explore these items further in order to determine if they should be reflected in lower order factors under autonomy. The CFA confirmed that the CTCU model is a good fit to the data, with no modifications needed to the proposed factor structure to obtain acceptable fit.

The CINSS demonstrated good criterionrelated validity, with a moderate positive correlation between the relatedness subscale and prosocial behaviours and a moderate negative correlation with behavioural problems. Correlations between autonomy and prosocial and problematic behaviours were lower. Children reporting being bullied or bullying others in the past 30 days had lower scores on all three subscales, with the lowest of these being on the competence subscale for both variables. This relationship between perceptions of competence and bullying or being bullied should be explored further, and enhancing competence may be an avenue for addressing bullying in schools. Competence is the experience of “oneself as effective in one’s interactions with the social and physical environments,”25,p. 27 and future research could explore whether bullying behaviours stem from a need to assert control when this is low in other domains. A Hong Kong study found no association between bullying and being bullied and teacher support for competence or autonomy, but did find an association with teacher support for relatedness. ²⁶ However, this study measured teacher support for autonomy, competence and relatedness, while the CINSS measures these concepts in three contexts: at home, at school and with peers.

Strengths and limitations

This study is the first of its kind to validate the CINSS in a large, representative school-based sample of youth in Canada. Other studies have examined the CINSS in

more specific contexts and with smaller samples: among school-aged children in New Brunswick and PEI;²⁷ an examination of intrinsic needs satisfaction and depression;²⁸ and an examination of the impact of a school-based intervention.²⁹ While we did not have additional measures that closely aligned with the concepts of autonomy, competence and relatedness on the CSTAD survey, which focusses primarily on tobacco, alcohol and drugs, there were related concepts that we were able to include in tests of criterion-related validity such as prosocial behaviours and problematic behaviours. It would be useful to examine how the CINSS subscales are associated with other closely related concepts, such as self-esteem, mastery and perceived control in the future. This survey was administered at a single point in time, and thus we were not able to examine its stability over time. Similarly, the data we analyzed only included children and youth in Grades 6 to 12. Further evaluation of this measure is required for younger children. While the concepts measured by the CINSS align well with our three-factor concept of positive mental health, this instrument was not developed as a measure of positive mental health. Given the high level of attention to promoting positive mental health among children in research and practice, it may be useful to develop an instrument measuring the concept of positive mental health.

Conclusion

Based on self-determination theory, the CINSS measures competence, autonomy and relatedness in three contexts: at home, at school and with peers. The CINSS scale is a promising measure of positive mental health in children and youth for national surveillance purposes. The CINSS subscales align well with the concepts of psychological well-being (competence, autonomy) and social well-being (relatedness) reported in the PMHSIF. The availability of validated scales of positive mental health for children and youth is an essential foundation for research that can inform policies and programs that aim to improve the well-being of this population group. Future research should examine levels of competence, autonomy and relatedness in different groups of students, and whether the CINSS is sensitive to change for use in intervention research.

Acknowledgements

We thank the CSTADS team at the University of Waterloo for leading the collection of these data, Health Canada for funding the CSTADS program and Joanna Odrowaz for her editing of this manuscript.

Conflicts of interest

There are no conflicts of interest to report.

Authors’ contributions and statement

HO and CP conceived the project and developed the research study. LK provided advice on methods. CP and HO conducted analysis. HO, CP, RD and LK interpreted the results. All authors drafted the manuscript, reviewed and revised the manuscript.

The content and views expressed in this article are those of the authors and do not necessarily reflect those of the Government of Canada.