Abstract
Background
Parents play an essential role in helping an adolescent who has a mental health concern; however, there are no measures of parental mental health literacy for parents of adolescents. Few measures of mental health literacy assess the underlying components of the construct, and measures that assess facets of mental health literacy in parents are limited in psychometric quality.
Objective
To develop and psychometrically evaluate a theoretically informed measure of parental mental health literacy, the Parental Mental Health Literacy (ParM-Lit) scale.
Method
The ParM-Lit was developed through the generation of items across key domains, expert review of items, and parental feedback. Parents of adolescents (N = 698) completed an online survey including the ParM-Lit and measures of parental attitudes toward help-seeking and knowledge of mental health disorders.
Results
Exploratory and confirmatory factor analyses supported a 4- and 5-factor model; however, a 31-item, 4-factor model showed slightly superior fit. Internal consistency of the overall ParM-Lit scale was very good (α = .89), and test-retest reliability was moderate (ICC = .68; 95% CI = .61–.75). The ParM-Lit was strongly associated with parental attitudes towards help-seeking and moderately associated with knowledge of mental health disorders.
Conclusions
The ParM-Lit is the first measure of parental mental health literacy, and our findings support its psychometric properties. This final 31-item measure holds promise for advancing measurement of parental mental health literacy in clinical and research settings.
Keywords: teenager, psychopathology, parenting, health promotion, help-seeking
Introduction
Mental health concerns commonly begin in adolescence (1). It is estimated that 22% of adolescents have mental health concerns with severe impairment (2); however, many adolescents do not seek help for their distress (3,4). Untreated mental health issues are detrimental to social and academic activities and result in poorer outcomes once treatment is received (5–7). Parents play a vital role in the early detection of mental health symptoms (8), and youth report that across their social networks, parents have the strongest influence on their decisions to access help (3,9,10). It is parents who often instigate help-seeking and connect their teen with appropriate services (8). Consequently, parents must be equipped with skills and attitudes to adequately aid an adolescent navigating mental health concerns (11).
Mental health literacy (MHL) was first defined by Jorm (12, p.182) as the “knowledge and beliefs about mental disorders which aid in their recognition, management, or prevention”. Since this early conceptualization, the definition has been broadened, first to emphasize stigma reduction and help-seeking (13, 14), and more recently, to focus on the prevention and promotion of mental wellness (15). Currently, MHL is recognized as a multidimensional construct, with several key components (13), including the ability to identify mental health concerns; attitudes and beliefs that facilitate help-seeking; and knowledge of professional and self-help treatment options, risk and protective factors, how to seek mental health information, and skills to promote wellbeing (13,15–17). MHL is important to positive mental health outcomes; higher levels of MHL are linked with earlier identification of mental health challenges (18) increased help-seeking behavior,(19) and lower sense of self-stigma (20).
In the context of child and youth mental health, teachers (21–24) and health professionals have commonly been a focus of MHL research, with consideration of how attitudes and knowledge can impact the ability of these individuals to assist youth. MHL in adolescents, has also been examined (25). However, parental MHL among parents of adolescents has received much less attention compared to parents of younger children (26). In a recent review of programs for parents of adolescents, Kusaka and colleagues (27) identified ten studies that aimed to improve MHL. They concluded that the impact of these programs was challenging to assess, given the limitations of available tools to assess literacy among parents (27). Thus, our understanding of MHL among parents of adolescents is limited due to a lack of research and measurement tools.
Parents are positioned to act on early indicators of mental health challenges, making a parent-specific measure valuable for supporting adolescent mental health. Moreover, measurement tools are vital for informing intervention and prevention programs, as well as the association of MHL with other relevant constructs. Across MHL measures, vignette-based approaches are common (12,16), but have been criticized as respondents have demonstrated an inability to accurately identify individuals with a mental health concern (28). Scale-based approaches using a dichotomous (true/false) or Likert-scale responses are a common alternative to vignette-based approaches (26). Such scales often lack psychometric investigation and require more rigorous development and validation across populations (29,30).
The dimensional nature of MHL has also impacted the measurement of MHL. Researchers often measure a single facet of literacy, such as stigma (e.g., 31), knowledge (e.g., 32), and help-seeking behaviours (16). This piecemeal approach limits comparison across investigations and overlooks critical aspects of MHL (15). In response to this limitation (29), several measures have been developed to incorporate a broader assessment of MHL (e.g., 34–36). Although an improvement, to our knowledge, no multifaceted measures of parental MHL exist. Therefore developing a measure from a framework that considers the unique needs of parents and adolescents is necessary to ensure important aspects of the construct are not overlooked. Measures developed without consideration of the parents of adolescents raise questions about the appropriateness and developmental sensitivity of the tools.
The purpose of this investigation was to develop and psychometrically assess a measure of parental Mental Health Literacy (MHL), the ParM-Lit Scale, which assesses multiple domains of MHL. We aimed to bring attention to mental health disorders commonly emerging in adolescence and literacy considerations specific to this developmental period.
Methods
Phase 1 - Development
Recommended best practices for developing and validating scales were followed (37). Given the evolving definition of MHL, we elected to rely on Jorm’s conceptualization (12, 13, 17) because it offers a comprehensive and actionable framework that aligns well with the roles of parents in supporting adolescent mental health. This conceptualization focuses on essential knowledge and attitudes about mental health, which we have adapted to emphasize the parent role and adolescent development. Items were designed to assess the following domains: (1) recognition of mental health concern in adolescence (recognition of symptoms); (2) awareness of developmentally relevant risk and protective factors of mental health problems (knowledge of contributors); (3) knowledge of when to seek help for a mental health problem in an adolescent (knowledge of when to seek help); (4) awareness of treatment option (knowledge of how to seek help, including self-help and professional health); and (5) attitudes or beliefs about mental health problems in youth that might affect help-seeking (attitudes towards help-seeking). A list of areas to be assessed within each domain was generated (e.g., symptoms commonly associated with mental health symptoms in youth; risk and protective factors in adolescence). Items were then developed based on these areas. Throughout this process, items from the first domain (recognition of symptoms) and the third domain (knowledge of when to seek help) were determined to have substantial overlap and combined into a single domain assessing recognition of symptoms. A preliminary 45-item version of the measure was sent to reviewers with expertise in MHL and/or youth mental health for feedback. Wording changes were made, and seven items were added, resulting in a 52-item pilot version of the ParM-Lit with four proposed subscales. Each item was tested against an online reading-level assessment to ensure that it was below a grade 8 reading level, using the Flesh-Kincaid Grade Level calculation. The final measure was a grade 7.3 reading level.
Preliminary testing of the items was conducted with 10 parents of adolescents aged 13 to 18, via an electronic survey. Ethics approval was waived by the institutional review board, as it was determined to meet exemption status (38) given the goal of improving the quality of the measure prior to commencing the study. Parent reviewers commented on their perception of the appropriateness, clarity of the content, and the wording of items. The preliminary testing resulted in wording modifications, but all items were retained.
Phase 2 – Psychometric Assessment
Participants
Parents of adolescents aged 13 to 18 were recruited via multiple platforms, including social media (Facebook, Twitter, and Instagram), as well as Kijiji and Craigslist, and through paid advertisements that targeted parents who had subscribed to the mailing list of a parenting magazine. Parents had to be living in Canada or the United States. No additional inclusion or exclusion criteria were outlined, including the nature of the parenting relationship. As a result, participation by legal guardians and step-parents was not restricted. Participants could enter a draw for 1 of 5 $100 (CAD) gift cards to a department store or provider of their choice.
Sample size recommendations in factor analyses vary widely. Researchers have recommended 10 participants per item (39,40), or 5 participants per model parameter (41); have graded sample sizes of 200 as “fair”, 300 as “good”, and 500 as “very good” (42); or identified that small samples (i.e., <100) can yield reliable results (43,44). Given our goal to test both exploratory and confirmatory models, which require separate samples (45), we aimed to recruit 300 to 500 participants per model, totalling 600 to 1000 participants. Participants who completed less than 80% of the questionnaire were excluded, as this was specified in the consent form as an indication of withdrawal from the survey. A total of 3,854 individuals opened the survey, but 2,895 did not enter any data. A remaining 958 started the questionnaire, with 260 removed due to completing <80% of the questionnaire, resulting in a final sample of 698 participants.
Measures
Demographic and Mental Health Information Questionnaire
A demographic questionnaire queried parental age, gender, nationality (United States or Canada), ethnicity, relationship status, and income. We also inquired about current and past mental health diagnoses and treatment for the parent and for any of their children.
Parental mental Health Literacy (ParM-Lit) Scale
The 52-item version of the ParM-Lit, described above, was used. The items were scored on a Likert-style scale (0 = Strongly Disagree, 4 = Strongly Agree), with 14 items reverse-scored. Scores were then summed, with higher scores indicating better MHL.
Parental Attitudes Toward Psychological Services Inventory (PATPSI)
The PATPSI (46) is a 14-item measure of parents’ willingness to access mental health services. The PATPSI is based on the Attitudes Toward Seeking Professional Psychological Help Scale (ATSPPHS; 47) questionnaire, and modified for parents. Questions are responded to using a 6-point Likert-style scale (0 = Strongly Disagree; 5 = Strongly Agree). Higher scores indicate poorer attitudes towards mental health services. The PATPSI has excellent psychometric properties, with Cronbach’s alpha across diverse parental populations ranging from 0.86 to 0.90 (46). Cronbach’s alphas for the total score in the current investigation were .86.
Knowledge of Mental Disorders Questionnaire (KDMQ)
The KDMQ (49) is an 11-item measure of the ability to differentiate mental health disorders from other health disorders (e.g., Multiple Sclerosis, Cirrhosis). From a list of 11 disorders, respondents indicate whether the condition is a mental health disorder (“Yes, it is,” “No, it is not,” or “I do not know”). Correct responses are scored as “1” and incorrect or unsure responses are scored as “0”. Internal consistency in previous studies has been acceptable (α = 0.59). In the current investigation, Cronbach’s alpha was 0.65.
Procedure
This study was approved by the University of Saskatchewan Behavioural Research Ethics Board (BEH 16-373).
Participants accessed the survey via a link in recruitment materials. The survey was hosted on the Voxco survey platform. Following a review of the consent information, free and informed consent was considered implied if participants proceeded to the subsequent survey page. At the end of the survey, participants were asked if they would be willing to be contacted to complete a second questionnaire. Those who agreed were sent a questionnaire containing only the ParM-Lit one month after completing the first survey. If needed, a reminder email was sent 2 weeks following the initial follow-up invitation.
Analyses
Demographic characteristics of the sample were examined using frequency calculations. The sample was then randomly split into two subsamples, with one used for exploratory factor analysis (n = 349) and the other for confirmatory factor analysis (n = 349).
Exploratory Factor Analysis (EFA; Subsample 1)
A correlation matrix and anti-image covariance matrix were examined to identify and remove potentially problematic items. Subsequently, a principal axis factor analysis with a Promax rotation was performed. Promax rotation was selected because some correlations were anticipated among the factors, and as a result, oblique rotation (e.g., Promax) was recommended (50). The number of factors retained was determined by considering the scree plot, eigenvalues greater than 1.0, and the use of parallel analysis procedures. Parallel analysis was conducted using an online application (51), with parameters set to generate 100 random correlation matrices and a percentile of eigenvalues of 95%. The EFA was conducted using SPSS version 28 (52).
Confirmatory Factor Analysis (CFA; Subsample 2)
The CFA of the model identified through the EFA procedures was completed using R (53) with Robust Maximum Likelihood (MLR) estimators and scaled scores reported. The Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), Root Mean Square Error of Approximation (RMSEA), and Standardised Root Mean Square Residual (SRMR) were used to assess goodness of fit. Based on previously established criteria, we considered a CFI above .90 as acceptable, with a CFI of .95 or higher as ideal; a TLI value of .90 or above as indicative of good fit; an RMSEA close to 0.06 as ideal with values between 0.08 and 0.1 representing poor fit; and a SRMR values of 0.08 or lower are indicative of good fit (54). χ2 values were also reported; however, other fit indices were considered more heavily, given the influence of sample size.
Reliability and Validity Analyses
The subsamples were combined for the remaining analyses. Internal consistency was measured with Cronbach’s alpha. Test-retest reliability was calculated by examining Pearson’s R and intraclass coefficients (ICC) between the first and second administrations of the ParM-Lit. The 95% confidence intervals of the ICC estimate were considered “poor” if <.5, “moderate” if .5–.75, “good” if .75–.9, and “excellent” if greater than .90 (56). In total, 604 participants agreed to participate in the follow-up survey, and 242 completed it.
Total and subscale scores of the ParM-Lit were correlated with the total scores of the KMDQ and the PATPSI to assess convergent validity. Significance of the correlation was examined, and Cohen’s (57) criteria of .10 = small, .30 = medium, and .50 = large were used to assess the strength of the correlation’s effect size (58). Reliability and validity analyses were conducted using SPSS version 28 (52).
Patient and Public Involvement
Minimal patient and public involvement occurred due to the focus of this study on the psychometric evaluation of the instrument. However, an initial consultation with experts and parents of adolescents was conducted to confirm the relevance and clarity of the items. This ensured the measure would be useful and understood by this population.
Results
Participant Characteristics
Participant characteristics can be found in Table 1. Most participants were female and identified themselves as the mothers of adolescents. Most participants reported having one (n = 401; 57.4%) or two (n = 213; 30.5%) children aged 13 to 18 at the time of completing the survey. Many participants indicated that their adolescent had a current (56%) or past (29%) mental health concern.
Table 1.
Demographic Characteristics of Participants (N= 698).
| Characteristics | |
|---|---|
| Age (years) | |
| Mean (SD) | 41.58 (6.14) |
| Range | 25–63 |
| Gender, n (%) | |
| Male | 10 (1.4) |
| Female | 686 (98.3) |
| Other | 2 (0.3) |
| Country | |
| Canada | 321 (46) |
| United States | 377 (54) |
| Ethnicity, n (%) | |
| Aboriginal, First Nations, Inuit, Metis | 29 (4.2) |
| Asian/Pacifica Islander | 3 (0.4) |
| Black | 22 (3.2) |
| Latino | 21 (3) |
| Middle Eastern | 1 (0.1) |
| White | 581 (83) |
| Prefer not to disclose | 12 (1.7) |
| Other | 29 (4.2) |
| Relationship status, n (%) | |
| Single, never married | 86 (12.3) |
| Married/Common law | 466 (66.6) |
| Widowed | 9 (1.3) |
| Separated/Divorced | 130 (18.6) |
| Prefer not to disclose | 7 (1.0) |
| Education, n (%) | |
| Some high school | 33 (4.7) |
| High School or Equivalent | 361 (51.7) |
| Associate degree or Diploma | 118 (16.9) |
| Bachelor’s degree | 120 (17.2) |
| Post-graduate training | 61 (2.1) |
| Income, n (%) a | |
| <$20,000 | 96 (13.8) |
| $20,000–$39,000 | 156 (22.3) |
| $40,000–$59,000 | 99 (14.2) |
| $60,000–$79,000 | 99 (14.2) |
| $80,000–$99,000 | 67 (9.6) |
| $100,000–$119,000 | 55 (7.9) |
| >$120,000 | 64 (9.2) |
| Prefer not to disclose | 62 (8.9) |
Values were not separated by USD or CAD.
EFA
Upon inspection of the correlation matrix, several items (n = 12) were identified as having a high number of correlations with very low loadings (i.e., < .10) and were therefore deleted. No items were found to be below the recommended cutoff of .5 within the anti-image matrix. Analyses were rerun excluding the 12 items with low loadings.
The KMO statistic was acceptable at .95, and the Bartlett test of sphericity was significant ( df = 780, p < .001).
Principal axis factoring with Promax rotation was conducted on the 40-item scale. Through the evaluation of eigenvalues, seven factors had an eigenvalue greater than 1; however, parallel analysis suggested that a four-factor solution would be more appropriate. Examination of the scree plot showed potential inflexion at both four and five factors. Consequently, both a 4-factor and 5-factor solution were considered and reported.
When forcing a 4-factor solution, the model explained a total of 49.06% of the variance. Item loadings were examined, and items that loaded above a value of. 40 and did not have cross-loadings of greater than .30 were retained. Nine items had cross-loadings above .30 or factor loadings below .40 and were removed. The 4-factor solution closely reflected the intended subscale areas. Factor 1 (13 items) reflected Recognition of a potential mental health concern. Factor 2 (6 items) reflected knowledge of Contributors to mental health concerns. Factor 3 (6 items) reflected knowledge of Supports available to parents. Factor 4 (6 items) represented Attitudes and beliefs about mental health concerns in adolescence. Factor loadings are summarised in our online supplementary materials.
When forcing a 5-factor solution, the model explained 52.53% of the variance. Item loadings were examined using the same criteria as reported above. Seven items were removed because they did not meet the loading criteria, resulting in a 5-factor solution with 33 items. Factor 1 (11 items) reflected Recognition of potential mental health concerns. Factor 2 (6 items) reflected knowledge of Supports available to parents. Factor 3 (5 items) reflected Attitudes and Beliefs about mental health concerns in adolescents. Factor 4 (5 items) was comprised of a subset of recognition items that related specifically to energy and irritability of the child, which were labelled as Energy Changes. Factor 5 (5-items) represented Contributors to mental health concerns. The factor loadings of the 5-factor solution are summarised in our online supplementary materials.
CFA
The 4- and 5-factor solutions were examined using CFAs. The resulting index scores are summarized in Table 2. Overall, indices varied in terms of meeting acceptable cutoffs. RMSEA and SRMR were well within the recommended ranges for both models. CFI reached the minimum of 0.90 for the 4-factor model, but not for the 5-factor model. Additionally, for both models, the TLI was outside the recommended limits.
Table 2.
Fit Indices of the 4- and 5- Factor Models of the ParM-Lit Scale.
| Model | χ2 | df | CFI | TLI | RMSEA [90%CI] | SMRM |
|---|---|---|---|---|---|---|
| 4-Factor Solution | 815.75 | 428 | 0.90 | 0.89 | 0.06 [.05–.06] | 0.06 |
| 5-Factor Solution | 944.17 | 485 | 0.89 | 0.88 | 0.06 [0.05–.06] | 0.06 |
Note. Robust statistics and scaled scores reported. ParM-Lit = Parental Mental Health Literacy Scale; CFI = Comparative Fit Index; TLI = Tucker-Lewis Index; RMSEA = Root Mean Square Error Approximation; SRMR = Standardised Root Mean Square Residual.
Although both models were largely comparable, a slightly better fit was observed in the 4-factor model. As a result, we used the 4-factor model for all remaining analyses. The 4-factor model, with standardised factor loadings, is illustrated in Figure 1.
Figure 1.
Confirmatory factor analysis path diagram for the 4-factor model. All reported values are standardised estimates. Significant paths at p <. 05 are identified with solid lines, and non-significant paths are denoted with dashed lines.
Reliability and Validity
Internal consistency was calculated for the full sample (i.e., N = 698) and was found to be very good (Cronbach’s alpha = .89). Subscale reliability was also excellent for the Recognition domain (Cronbach’s alpha .91), very good for the Supports domain (Cronbach’s alpha = .88) and Attitudes domain (Cronbach’s alpha = .83) and was acceptable for the Contributors domain (Cronbach’s alpha = .71). Corrected factor-total correlations were examined as an assessment of construct validity. Results suggested that each factor significantly contributed to the overall construct being measured, as follows: Recognition = 0.48; Supports = 0.73; Attitudes = 0.55; Contributors = 0.73.
A total of 242 participants completed the re-test survey. There was no significant difference between those who completed the re-test survey and those who did not on age, gender, or number of children. Test-retest reliability, as measured by Pearson’s r4, was moderate for all four domains and the overall subscale score; ICCs were moderate for all domains, but the Recognition domain had a lower confidence interval outside acceptable ranges (Table 3). The Supports and Attitudes domains, along with the total scale, showed the strongest evidence of stability over the 4-week period. There were no significant differences between time intervals for the overall score (t(1, 241) = 1.24, p = .22), nor for the domains of Supports (t(1, 241) = −.73, p = .46), Attitudes (t(1, 241) = −.91, p = −.91) or Contributors (t 1, 241) = 1.64, p = .10. The Recognition domain was also not significant, t (1, 241) = 2.00, p = .05; however, the p value approached significance suggesting some stability concerns with this domain.
Table 3.
Test-Retest Reliability of the 31-Item ParM-Lit
| Scale and Subscales | Pearson’s R | ICC | 95%CI |
|---|---|---|---|
| Total ParM-Lit | .69 | .68 | .61–.75 |
| Recognition | .54 | .54 | .45–.62 |
| Contributors | .63 | .62 | .54–.69 |
| Supports | .67 | .67 | .59–.73 |
| Attitudes | .73 | .72 | .66–.78 |
ParM-Lit = Parental Mental Health Literacy Scale; ICC = Intraclass Correlation.
The results of the convergent validity analyses are presented in Table 4. The PATPSI was significantly and negatively associated with the ParM-Lit total score, with an effect size ranging from medium to large. A negative association was expected given the valence of each scale. The Contributors domain was not significantly correlated with the PATPSI, but all remaining domains of the ParM-Lit questionnaire were significantly associated with the PATPSI. The Attitudes domain had a strongly significant association and a large effect with the PATPSI. The KMDQ was significantly and positively associated with the total and domain scales of the ParM-Lit questionnaire, and all effect sizes fell within the small to moderate range.
Table 4.
Pearson’s Correlation Between the ParM-Lit scale and subscale scores and the KDMQ and the PATPSI.
| Scale and Subscales | KMDQ | PATPSI |
|---|---|---|
| Total ParM-Lit | .22 ** | −.44** |
| Recognition | .14 ** | −.23** |
| Contributors | .16* | .03 |
| Supports | .09* | −.34** |
| Attitudes | .24** | −.68** |
ParM-Lit = Parental Mental Health Literacy Scale; KMDQ = Knowledge of Mental Disorders Questionnaire; PATPSI = Parental Attitudes Towards Professional Services Inventory.
p = .01,
p = .05
Discussion
We psychometrically tested the newly developed ParM-Lit scale, a measure of MHL for parents of adolescents aged 13 to 18. The tool was theoretically based, informed by expert feedback, and pre-tested with parents. The ParM-Lit is the only multifaceted measure of MHL for parents, the only measure specific to parents of adolescents, and one of the few MHL measures to have undergone rigorous psychometric evaluation (16). MHL among parents of adolescents remains an understudied area of investigation (27); our 31-item measure provides a useful tool for researchers to characterise literacy levels in parents.
The internal structure of the ParM-Lit scale was tested using exploratory and confirmatory approaches. Our EFA findings supported a 4- and 5-factor model; however, an examination of factor loadings suggested a more parsimonious structure in the 4-factor model. The 5-factor model performed slightly more poorly through confirmatory procedures. In contrast, the 4-factor model met minimum criteria for all fit indices except one. Our findings support a construct of MHL comprised of four dimensions (i.e., Recognition, Supports, Attitudes, Contributors). Although researchers have questioned the utility of a single measure of MHL and suggested that it might not identify a unified underlying construct (17), our results suggest an overarching construct with interrelated underlying domains. All four of our domains were significantly interrelated, except for Contributors with Attitudes and Supports. This may reflect different underlying aspects of literacy; the Contributors’ items most heavily relied on factual knowledge, compared to items more associated with personal experiences and beliefs. Interestingly, apart from one item, all items in the 4-factor model loaded onto the subscales intended in the developmental phase of the ParM-Lit, suggesting face validity for the subscales and overall measure.
Other researchers have similarly identified a four-factor structure in their measures of MHL (i.e., 36), which may further elucidate the underlying nature of the MHL construct. That said, researchers who have developed multifaceted measures of MHL have found a unidimensional construct (34). Notably, however, in the Mental Health Literacy Scale (MHLS) (34), potential underlying facets were measured by very few items in some cases (i.e., 2 or fewer items), which may have limited the dimensionality of the emerging factors, as four or more items are recommended for factor identification (59,60). While Aller and colleagues (35) also proposed a multifaceted structure, their investigation did not examine the factor structure of all items; thus, it is unclear what underlying structure their items hold (35). It has been argued that measuring MHL is challenging as there is no consensus or systematic research on the components that comprise MHL (29). Despite this, extensive theoretical work has found some consistency in these facets, and our findings support them as possible underlying features of the overarching construct.
Individual factors demonstrate good to excellent internal consistency, indicating that the items within subscales are strongly related. We found an elevated degree of association among factors and strong scale-level statistics; however, the Recognition and Support subscales were noted to contribute less to the overall factor, despite a moderate association with the overarching construct. We noted concerns with stability across four weeks, as indicated by test-retest reliability, particularly in the Recognition domain, but also across all domains and the overall scale. One possible explanation is that items related to knowledge may have caused participants to learn more about these areas, resulting in practice effects (61) or response shifting (i.e., a change in a person’s evaluation of the items) (62). Our sample may have comprised participants with a natural interest in mental health who sought additional information more after the first administration, based on the questions. Future investigations may consider the interval at which the construct is stable, along with the contributors to score changes over time, to better understand the potential stability of the measure.
Convergent validity scores within some subscales were lower than expected. Both the PATPSI and the KDMQ only measured a single facet of MHL (i.e., attitudes and knowledge, respectively). Thus, although significant correlations were observed among the KDMQ and the full and subscale scores of the ParM-Lit, the ParM-Lit did not have a specific knowledge subscale that would have been most similar to the KDMQ, and the low observed correlations are not entirely unexpected.
Conversely, the Attitudes domain of the ParM-Lit scale was most strongly correlated with the PATPSI, as expected, given the construct assessed in the PATPSI. Of note, the reliability of the KMDQ was only acceptable. With a broader inclusion of measures to assess construct validity, further validation is needed.
This investigation is not without limitations. Our sample was homogenous, being predominantly female and white, and identifying as mothers, which may limit the generalizability of the measure. Moreover, the ParM-Lit was developed from a Western lens and may not be appropriate for non-Western cultures. Further validation of the ParM-Lit in diverse samples is needed. Future iterations of the ParM-Lit may be adapted to have more inclusive language (i.e., “they” rather than “he/she”). Lastly, our convenience sample may have drawn participants who are familiar with mental health concerns. Despite these limitations, the ParM-Lit is a promising tool for use in clinical and research settings.
Supplementary Information
Footnotes
Conflict of Interest: The authors have no financial relationships or other ties to disclose.
References
- 1.Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime Prevalence and Age-of-Onset Distributions of DSM-IV Disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry [Internet] 2005 Jun 1;62(6):593. doi: 10.1001/archpsyc.62.6.593. Available from: http://archpsyc.jamanetwork.com/article.aspx?doi=10.1001/archpsyc.62.6.593. [DOI] [PubMed] [Google Scholar]
- 2.Merikangas KR, He J, Burstein M, Swanson SA, Avenevoli S, Cui L, et al. Lifetime Prevalence of Mental Disorders in U.S. Adolescents: Results from the National Comorbidity Survey Replication–Adolescent Supplement (NCS-A) J Am Acad Child Adolesc Psychiatry. 2010 Oct;49(10):980–9. doi: 10.1016/j.jaac.2010.05.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Rickwood DJ, Mazzer KR, Telford NR. Social influences on seeking help from mental health services, in-person and online, during adolescence and young adulthood. BMC Psychiatry [Internet] 2015 Dec 7;15(1):40. doi: 10.1186/s12888-015-0429-6. Available from: http://bmcpsychiatry.biomedcentral.com/articles/10.1186/s12888-015-0429-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Mauerhofer A, Berchtold A, Michaud P-A, Suris J-C. GPs’ role in the detection of psychological problems of young people: a population-based study. Br J Gen Pract [Internet] 2009 Sep 1;59(566):e308–14. doi: 10.3399/bjgp09X454115. Available from: https://bjgp.org/lookup/doi/10.3399/bjgp09X454115. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Mohamed Ibrahim OH, Ibrahim RM, Al-Tameemi NK, Riley K. Challenges associated with mental health management: Barriers and consequences. Saudi Pharm J [Internet] 2020 Aug;28(8):971–6. doi: 10.1016/j.jsps.2020.06.018. Available from: https://linkinghub.elsevier.com/retrieve/pii/S1319016420301444. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Altamura AC, Dell’Osso B, Vismara S, Mundo E. May duration of untreated illness influence the long-term course of major depressive disorder? Eur Psychiatry [Internet] 2008 Mar 16;23(2):92–6. doi: 10.1016/j.eurpsy.2007.11.004. Available from: https://www.cambridge.org/core/product/identifier/S0924933800259051/type/journal_article. [DOI] [PubMed] [Google Scholar]
- 7.de Diego-Adeliño J, Portella MJ, Puigdemont D, Pérez-Egea R, Álvarez E, Pérez V. A short duration of untreated illness (DUI) improves response outcomes in first-depressive episodes. J Affect Disord [Internet] 2010 Jan;120(1–3):221–5. doi: 10.1016/j.jad.2009.03.012. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0165032709001256. [DOI] [PubMed] [Google Scholar]
- 8.Logan DE, King CA. Parental facilitation of adolescent mental health service utilization: A conceptual and empirical review. Clin Psychol Sci Pract [Internet] 2006 May 11;8(3):319–33. Available from: http://doi.wiley.com/10.1093/clipsy.8.3.319. [Google Scholar]
- 9.Aguirre Velasco A, Cruz ISS, Billings J, Jimenez M, Rowe S. What are the barriers, facilitators and interventions targeting help-seeking behaviours for common mental health problems in adolescents? A systematic review. BMC Psychiatry [Internet] 2020 Dec 11;20(1):293. doi: 10.1186/s12888-020-02659-0. Available from: https://bmcpsychiatry.biomedcentral.com/articles/10.1186/s12888-020-02659-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Wahlin T, Deane F. Discrepancies between parent- and adolescent-perceived problem severity and influences on help seeking from mental health services. Aust New Zeal J Psychiatry [Internet] 2012 Jun 22;46(6):553–60. doi: 10.1177/0004867412441929. Available from: http://journals.sagepub.com/doi/10.1177/0004867412441929. [DOI] [PubMed] [Google Scholar]
- 11.Hurley D, Swann C, Allen MS, Ferguson HL, Vella SA. A systematic review of parent and caregiver mental health Literacy. Community Ment Health J [Internet] 2020 Jan 20;56(1):2–21. doi: 10.1007/s10597-019-00454-0. Available from : http://link.springer.com/10.1007/s10597-019-00454-0. [DOI] [PubMed] [Google Scholar]
- 12.Jorm AF, Korten AE, Jacomb PA, Christensen H, Rodgers B, Pollitt P. “Mental health literacy”: a survey of the public’s ability to recognize mental disorders and their beliefs about the effectiveness of treatment. Med J Aust [Internet] 1997 Feb 17;166(4):182–6. doi: 10.5694/j.1326-5377.1997.tb140071.x. A vailable from: http://www.ncbi.nlm.nih.gov/pubmed/9066546. [DOI] [PubMed] [Google Scholar]
- 13.Jorm AF. Mental health literacy: Public knowledge and beliefs about mental disorders. Br J Psychiatry. 2000 Nov;177(05):396–401. doi: 10.1192/bjp.177.5.396. [DOI] [PubMed] [Google Scholar]
- 14.Gulliver A, Griffiths KM, Christensen H. Perceived barriers and facilitators to mental health help-seeking in young people: a systematic review. BMC Psychiatry [Internet] 2010 Dec 30;10(1):113. doi: 10.1186/1471-244X-10-113. Available from: http://bmcpsychiatry.biomedcentral.com/articles/10.1186/1471-244X-10-113. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Kutcher S, Wei Y, Coniglio C. Mental health literacy. Can J Psychiatry [Internet] 2016 Mar 3;61(3):154–8. doi: 10.1177/0706743715616609. Available from: http://journals.sagepub.com/doi/10.1177/0706743715616609. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Wei Y, McGrath PJ, Hayden J, Kutcher S. Mental health literacy measures evaluating knowledge, attitudes and help-seeking: a scoping review. BMC Psychiatry [Internet] 2015 Dec 17;15(1):291. doi: 10.1186/s12888-015-0681-9. Available from: http://www.biomedcentral.com/1471-244X/15/291. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Jorm AF. International Handbook of Health Literacy [Internet] Policy Press; 2019. The concept of mental health literacy; pp. 53–66. Available from: https://bristoluniversitypressdigital.com/view/book/9781447344520/ch004.xml. [Google Scholar]
- 18.Tambling RR, D’Aniello C, Russell BS. Mental health literacy: a critical target for narrowing racial disparities in behavioral health. Int J Ment Health Addict [Internet] 2023 Jun 11;21(3):1867–81. doi: 10.1007/s11469-021-00694-w. Available from: https://link.springer.com/10.1007/s11469-021-00694-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Goodfellow C, Macintyre A, Knifton L, Sosu E. Associations between dimensions of mental health literacy and adolescent help-seeking intentions. Child Adolesc Ment Health [Internet] 2023 Sep;28(3):385–92. doi: 10.1111/camh.12608. Available from: http://www.ncbi.nlm.nih.gov/pubmed/36375813. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Koutra K, Pantelaiou V, Mavroeides G. Breaking Barriers: unraveling the connection between mental health literacy, attitudes towards mental illness, and self-stigma of psychological help-seeking in University students. Psychol Int [internet] 2024 May 2;6(2):590–602. Available from: https://www.mdpi.com/2813-9844/6/2/35. [Google Scholar]
- 21.Mikami AY, Smit S, Johnston C. Teacher attributions for children’s attention-deficit/hyperactivity disorder behaviors predict experiences with children and with classroom behavioral management in a summer program practicum. Psychol Sch [Internet] 2019 Jun 8;56(6):928–44. Available from: https://onlinelibrary.wiley.com/doi/10.1002/pits.22250. [Google Scholar]
- 22.Ní Chorcora E, Swords L. Mental health literacy and help-giving responses of Irish primary school teachers. Irish Educ Stud [Internet] 2022 Oct 2;41(4):735–51. Available from: https://www.tandfonline.com/doi/full/10.1080/03323315.2021.1899029. [Google Scholar]
- 23.Whitley J, Gooderham S. Exploring mental health literacy among pre-service teachers. Except Educ Int [Internet] 2016 Dec 22;26(2) Available from: https://ojs.lib.uwo.ca/index.php/eei/article/view/7741. [Google Scholar]
- 24.Johnson CL, Gross MA, Jorm AF, Hart LM. Mental health literacy for supporting children: a systematic review of teacher and parent/carer knowledge and recognition of mental health problems in childhood. Clin Child Fam Psychol Rev [Internet] 2023 Jun 10;26(2):569–91. doi: 10.1007/s10567-023-00426-7. Available from: https://link.springer.com/10.1007/s10567-23-00426-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Mansfield R, Patalay P, Humphrey N. A systematic literature review of existing conceptualization and measurement of mental health literacy in adolescent research: current challenges and inconsistencies. BMC Public Health [Internet] 2020 Dec 1;20(1):607. doi: 10.1186/s12889-020-08734-1. Available from: https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-020-08734-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Kucera M, Tomaskova H, Stodola M, Kagstrom A. A systematic review of mental health literacy measures for children and adolescents. Adolesc Res Rev [Internet] 2023 Sep 2;8(3):339–58. Available from: https://link.springer.com/10.1007/s40894-022-00202-8. [Google Scholar]
- 27.Kusaka S, Yamaguchi S, Foo JC, Togo F, Sasaki T. Mental health literacy programs for parents of adolescents: A systematic review. Front Psychiatry [Internet] 2022 May 2;:13. doi: 10.3389/fpsyt.2022.816508. Available from: https://www.frontiersin.org/articles/10.3389/fpsyt.2022.816508/full. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Pescosolido BA, Jensen PS, Martin JK, Perry BL, Olafsdottir S, Fettes D. Public knowledge and assessment of child mental health problems: Findings From the National Stigma Study-Children. J Am Acad Child Adolesc Psychiatry [Internet] 2008 Mar;47(3):339–49. doi: 10.1097/CHI.0b013e318160e3a0. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0890856709623170. [DOI] [PubMed] [Google Scholar]
- 29.O’Connor M, Casey L, Clough B. Measuring mental health literacy – a review of scale-based measures. J Ment Heal [Internet] 2014 Aug 30;23(4):197–204. doi: 10.3109/09638237.2014.910646. Available from: http://www.tandfonline.com/doi/full/10.3109/09638237.2014.910646. [DOI] [PubMed] [Google Scholar]
- 30.Wei Y, McGrath PJ, Hayden J, Kutcher S. Measurement properties of tools measuring mental health knowledge: a systematic review. BMC Psychiatry [Internet] 2016 Dec 23;16(1):297. doi: 10.1186/s12888-016-1012-5. Available from: http://bmcpsychiatry.biomedcentral.com/articles/10.1186/s12888-016-1012-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Boyd Ritsher J, Otilingam PG, Grajales M. Internalized stigma of mental illness: psychometric properties of a new measure. Psychiatry Res [Internet] 2003 Nov;121(1):31–49. doi: 10.1016/j.psychres.2003.08.008. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0165178103002075. [DOI] [PubMed] [Google Scholar]
- 32.Evans-Lacko S, Little K, Meltzer H, Rose D, Rhydderch D, Henderson C, et al. Development and psychometric properties of the Mental Health Knowledge Schedule. Can J Psychiatry [Internet] 2010 Jul 1;55(7):440–8. doi: 10.1177/070674371005500707. Available from: http://journals.sagepub.com/doi/10.1177/070674371005500707. [DOI] [PubMed] [Google Scholar]
- 33.Fischer EH, Farina A. Attitudes toward Seeking Professional Psychological Help A shortened form and considerations for research. J Coll Stud Dev. 1995;36(4):368–73. [Google Scholar]
- 34.O’Connor M, Casey L. The Mental Health Literacy Scale (MHLS): A new scale-based measure of mental health literacy. Psychiatry Res [Internet] 2015 Sep;229(1–2):511–6. doi: 10.1016/j.psychres.2015.05.064. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0165178115003698. [DOI] [PubMed] [Google Scholar]
- 35.Aller TB, Fauth EB, Novak JR, Schwartz S. Measuring mental health literacy: Development of the Mental Health Awareness and Advocacy Assessment Tool. J Multidiscip Eval [Internet] 2021 Mar 17;17(39):15–31. Available from: https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/671. [Google Scholar]
- 36.Soleimani-Sefat E, Parandeh A, Rahmati F, Kamalikhah T, editors. Development and psychometric properties of the Mental Health Literacy Questionnaire (MHLQ) among young Iranian soldiers. Int J Prev Med [Internet] 2022;13(1):101. doi: 10.4103/ijpvm.IJPVM_468_20. Available from: https://journals.lww.com/10.4103/ijpvm.IJPVM_468_20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer. Front Public Heal [Internet] 2018 Jun 11;:6. doi: 10.3389/fpubh.2018.00149. Available from: https://www.frontiersin.org/article/10.3389/fpubh.2018.00149/full. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, Social Sciences and Humanities Research Council of Canada. Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS2 2018) 2018. Available from: https://ethics.gc.ca/eng/documents/tcps2-2018-en-interactive-final.pdf.
- 39.Clark LA, Watson D. Constructing validity: Basic issues in objective scale development. Psychol Assess [Internet] 1995 Sep;7(3):309–19. doi: 10.1037/1040-3590.7.3.309. Available from: http://doi.apa.org/getdoi.cfm? [DOI] [Google Scholar]
- 40.Devellis RF. Sage Publications, editor. Scale development: theory and applications. 2nd ed. Newbury Park: 2003. [Google Scholar]
- 41.Bentler PM, Chou C-P. Practical issues in structural modeling. Sociol Methods Res [Internet] 1987 Aug 30;16(1):78–117. Available from: http://journals.sagepub.com/doi/10.1177/0049124187016001004. [Google Scholar]
- 42.Comrey AL, Lee HB. Lawrence Eribaum Associates, editor. A First Course in Factor Analysis. Hillsdale, NJ: 1992. [Google Scholar]
- 43.de Winter JCF, Dodou D, Wieringa PA. Exploratory factor analysis with small sample sizes. Multivariate Behav Res [Internet] 2009 Apr 24;44(2):147–81. doi: 10.1080/00273170902794206. Available from: https://www.tandfonline.com/doi/full/10.1080/00273170902794206. [DOI] [PubMed] [Google Scholar]
- 44.Jung S. Exploratory factor analysis with small sample sizes: A comparison of three approaches. Behav Processes [Internet] 2013 Jul;97:90–5. doi: 10.1016/j.beproc.2012.11.016. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0376635712002513. [DOI] [PubMed] [Google Scholar]
- 45.Fokkema M, Greiff S. How performing PCA and CFA on the same data equals trouble. Eur J Psychol Assess [Internet] 2017 Nov;33(6):399–402. Available from: : https://econtent.hogrefe.com/doi/10.1027/1015-5759/a000460. [Google Scholar]
- 46.Turner EA. The parental attitudes toward psychological services Inventory: Adaptation and Development of an Attitude Scale. Community Ment Health J [Internet] 2012 Aug 21;48(4):436–49. doi: 10.1007/s10597-011-9432-7. Available from: http://link.springer.com/10.1007/s10597-011-9432-7. [DOI] [PubMed] [Google Scholar]
- 47.Fischer EH, Turner JI. Orientations to seeking professional help: Development and research utility of an attitude scale. J Consult Clin Psychol [Internet] 1970 Aug;35(1 Pt 1):79–90. doi: 10.1037/h0029636. Available from: https://doi.apa.org/doi/10.1037/h0029636. [DOI] [PubMed] [Google Scholar]
- 48.Choy SKJ, Vincent AM, Maesaka TJ, Nakamura BJ. Psychometric analysis of the parental attitudes toward psychological services inventory with a diverse community sample. J Child Fam Stud [Internet] 2024 Jan 17;33(1):114–28. Available from: https://link.springer.com/10.1007/s10826-023-02699-6. [Google Scholar]
- 49.Serra M, Lai A, Buizza C, Pioli R, Preti A, Masala C, et al. Beliefs and attitudes among Italian high school students toward people with severe mental disorders. J Nerv Ment Dis [Internet] 2013 Apr;201(4):311–8. doi: 10.1097/NMD.0b013e318288e27f. Available from: https://journals.lww.com/00005053-201304000-00008. [DOI] [PubMed] [Google Scholar]
- 50.Costello AB, Osborne J. Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Pract Assess Res Eval. 2005;10(1–9) [Google Scholar]
- 51.Patil VH, Singh SN, Mishra S, Donavan DT. Parallel Analysis Engine to Aid in Determining Number of Factors to Retain using R [Computer software] [Internet] 2017. Available from: https://analytics.gonzaga.edu/parallelengine/
- 52.IBM Corp. IBM SPSS Statistics for Macintosh, Version 28.0. Armonk, NY: IBM Corp; 2021. [Google Scholar]
- 53.R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing [Internet]; 2023. Available from: https://www.r-project.org/ [Google Scholar]
- 54.Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct Equ Model A Multidiscip J [Internet] 1999 Jan;6(1):1–55. Available from: http://www.tandfonline.com/doi/abs/10.1080/10705519909540118. [Google Scholar]
- 55.Saris WE, Satorra A, van der Veld WM. Testing structural equation models or detection of misspecifications? Struct Equ Model A Multidiscip J [Internet] 2009 Oct 6;16(4):561–82. Available from: http://www.tandfonline.com/doi/abs/10.1080/10705510903203433. [Google Scholar]
- 56.Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med [Internet] 2016 Jun;15(2):155–63. doi: 10.1016/j.jcm.2016.02.012. Available from: https://linkinghub.elsevier.com/retrieve/pii/S1556370716000158. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Cohen J. Routledge Academic, editor. Statistical Power Analysis for the Behavioral Science. New York, NY: 1988. [Google Scholar]
- 58.Bosco FA, Aguinis H, Singh K, Field JG, Pierce CA. Correlational effect size benchmarks. J Appl Psychol [Internet] 2015 Mar;100(2):431–49. doi: 10.1037/a0038047. Available from: https://doi.apa.org/doi/10.1037/a0038047. [DOI] [PubMed] [Google Scholar]
- 59.Russell DW. In Search of Underlying Dimensions: The Use (and Abuse) of Factor Analysis in Personality and Social Psychology Bulletin. Personal Soc Psychol Bull [Internet] 2002 Dec 25;28(12):1629–46. Available from: http://journals.sagepub.com/doi/10.1177/014616702237645. [Google Scholar]
- 60.Fabrigar LR, Wegener DT, MacCallum RC, Strahan EJ. Evaluating the use of exploratory factor analysis in psychological research. Psychol Methods [Internet] 1999 Sep;4(3):272–99. Available from: http://doi.apa.org/getdoi.cfm?doi=10.1037/1082-989X.4.3.272. [Google Scholar]
- 61.Calamia M, Markon K, Tranel D. Scoring Higher the Second Time Around: Meta-Analyses of Practice Effects in Neuropsychological Assessment. Clin Neuropsychol [Internet] 2012 May;26(4):543–70. doi: 10.1080/13854046.2012.680913. Available from: http://www.tandfonline.com/doi/abs/10.1080/13854046.2012.680913. [DOI] [PubMed] [Google Scholar]
- 62.Polit DF. Getting serious about test–re-test reliability: a critique of re-test research and some recommendations. Qual Life Res [Internet] 2014 Aug 7;23(6):1713–20. doi: 10.1007/s11136-014-0632-9. Available from: http://link.springer.com/10.1007/s11136-014-0632-9. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.