Self-assessment scale of basic mental health action competencies: development and evaluation of the psychometric properties

Abstract

Background

The assessment of adolescents’ mental health competencies requires validated instruments. Thus, we sought to develop and evaluate the psychometric properties of a scale in which adolescents self-assess their competencies to act in mental health.

Methods

A psychometric study was conducted. The data was collected in Portugal between May 2022 and September 2023. Firstly, a focus group was held with 9 experts to generate the scale items. The data was examined using content analysis. Secondly, a modified e-Delphi technique was used to obtain the opinion of 97 healthcare workers on the scale statements and the classification method. An analysis of response frequencies was carried out. Thirdly, cognitive interviews about the scale were conducted with 8 adolescents. Content analysis was performed. Finally, the scale was applied to 514 adolescents. Item sensitivity assessment and factor analysis were performed. The scale’s convergent and divergent validity, internal consistency and test-retest reliability were also examined.

Results

A 20-item scale emerged, featuring four factors: perception of knowledge about mental health, mental health self-care, interpersonal relationships and mental health first aid. Cronbach’s alpha and McDonald’s omega varied from 0.79 to 0.92, the composite reliability from 0.70 to 0.89, and the intraclass correlation coefficient from 0.77 to 0.80.

Conclusions

The scale was created and demonstrated good validity and reliability. It facilitates the assessment of adolescents’ mental health needs and the results of educational interventions.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12888-025-07676-5.

Background

Competence is a term generally used to define a set of knowledge, skills and attitudes that enable the successful performance of tasks [1]. According to Salman et al., the concept of competence involves individual visible characteristics (knowledge, skills and behaviour) and hidden features (traits, motives, attitudes, values and self-image) [2].

Competencies can be improved with training [2]. In turn, populations usually have access to training in schools. Worldwide, compulsory education begins at age 6 and lasts 10 years (on average) [3]. In Portugal, 12 years of schooling must be completed by law (upper secondary education) [4]. Thus, adolescents aged 15 to 18 are expected to acquire knowledge and skills to carry out tasks, solve problems, and take responsibility for their actions [4]. At 18, individuals commonly reach the age of majority, assuming full responsibility for their actions [5]. Therefore, it is crucial to understand the level of competence of people in this period (15 to 18 years old), including in mental health matters.

Developing health competencies can promote involvement in individual and collective health issues [6]. Adolescents can be empowered with basic/generic competencies in mental health, actively participating in self-care and supporting those with mental health problems [7]. Zábó et al. recognised mental health competence as a predictive factor of mental health [8].

Laypeople can maintain or improve their mental well-being, respond to mental health problems to prevent mental disorders, and manage and recover from mental illnesses [9]. Hence, they need the competencies to recognise their mental health status, choose and use adaptive self-care strategies, seek assistance and comply with therapeutic regimens [10, 11]. When individuals possess mental health competencies, they are better equipped to cope with their own mental health challenges, which can lead to lower healthcare costs and reduced pressure on mental health services [12]. Greater competence is associated with less mental health stigma and, consequently, with a more inclusive and understanding society [13]. The acceptance of mental health problems makes it more likely that individuals will seek help early and receive appropriate support [14].

Furthermore, people without mental health expertise often encounter individuals experiencing problems, playing a significant role in their stabilisation [15]. Consequently, they must be able to recognise signs and symptoms of mental distress in others, offer initial help and refer them to appropriate support services [10, 15]. Competent individuals can effectively preserve life when someone poses a danger to themselves and/or others, prevent the worsening of mental health problems, promote recovery and provide comfort to those experiencing such problems [15]. Their suggestion may increase the likelihood of another person seeking professional help [16]. Empowering the general public increases the availability of mental health support resources, fostering a more supportive society [10].

Mental health competence is critical during adolescence, as it is a period marked by significant emotional and social changes [17], and mental health problems often emerge during this stage [18]. Important habits for long-term mental well-being are also established, including coping and problem-solving skills, social abilities and emotion regulation [17].

The actual competencies in mental health of laypeople (including adolescents) have been studied [19]. They refer to the cognitive and practical abilities that individuals possess and can effectively apply, independently of their own perception [20]. Several constructs - and their corresponding assessment instruments - have been proposed to represent these objective competencies [21]. For example, mental health literacy refers to the actual knowledge and skills related to the prevention and recognition of mental health problems, help-seeking, self-care, and supporting others in crisis [20, 21]. Within mental health literacy, knowledge forms the basis upon which individuals recognise mental health states and consider possible responses [20]. However, knowing alone does not necessarily translate into acting. Individuals are more likely to engage in effective behaviours when they also perceive themselves as capable of applying that knowledge [22, 23]. In this sense, perceived competence serves as the motivational link that allows individuals to translate knowledge into concrete mental health actions.

Perceived competence involves the need to master a personally challenging task [24]. It includes the perception of the ability to perform a task and the assessment of the personal importance of the activity [25]. Perceived competence can be associated with generic areas of action in mental health. We conceptualise perceived basic competence to act in mental health as the importance attributed by laypeople to caring for their mental well-being and that of others, alongside the perception of ability in these activities.

Perceived competencies do not necessarily align with actual competencies [26], that is, individuals’ beliefs about what they know or can do may differ from their true abilities. These self-assessments are subjective inferences based on past experiences, influenced by normative comparisons (such as the performance of others), absolute standards (for example, task requirements), or intrapersonal benchmarks (such as one’s previous performance) [27]. Even so, perceived health competence (not specific to mental health) has been shown to positively influence health behaviours and quality of life [28, 29]. Building on this premise, adolescents who perceive themselves as competent to act in mental health may be more engaged in protective behaviours, such as self-care, early help-seeking, and supporting others in distress, contributing to more positive health trajectories and to a community environment that is more responsive to mental health needs.

During adolescence, people develop abstract and hypothetical-deductive thinking [30], which allows them to understand more complex issues (such as mental health) and to reflect on their own competencies. They are also forming a clearer sense of self [31], which may involve evaluating their abilities. Therefore, it is important to explore adolescents’ self-evaluations of their mental health competencies.

Peplau highlighted that health professionals care for their patients based on what the latter consider they need [32]. Such perceptions can be assessed using psychometric instruments. However, there appears to be a lack of tools that assess perceived basic competencies to act on mental health issues. An exploratory search focused on instruments for assessing these perceived competencies was conducted in March and April 2022. This search was conducted in several databases: Web of Science Core Collection, MEDLINE complete, CINAHL complete, PsycINFO, SciELO, Scopus, COSMIN Database of Systematic Reviews, Cochrane Library, JBI Evidence Synthesis and PROSPERO. Only one instrument was found to assess the perceived competence of students and teachers to support the mental health of university students (College Mental Health Perceived Competency Scale) [33]. The scale specifically measures three factors: engagement (faculty involvement in supporting students with mental health disorders), fear (anxiety or concern about mental health issues on college campuses), and knowledge (perceived familiarity with mental health issues in academic settings) [33]. We recognise limitations in this scale, such as the lack of clarity in the item development process, the “Fear” factor containing only two items and the absence of a comprehensive assessment of psychometric properties (e.g., reliability was only supported by Cronbach’s Alpha). Moreover, given its specificity in supporting others, we believe it is still pertinent to construct a self-assessment scale of different basic competencies to act on mental health issues and to evaluate its psychometric properties.

Costa et al. highlighted the importance of developing instruments to assess perceived competencies in mental health to determine the educational needs of adolescents and the outcomes of mental health training programmes [10]. Thus, this study aimed to develop a scale that measures the basic competencies perceived by adolescents aged 15 to 18 to act on mental health issues and evaluate its psychometric properties. The scale assesses individuals’ subjective perception of their competencies and does not objectively verify them. We called the instrument under study the Self-assessment Scale of Basic Mental Health Action Competencies (Basic-MAC scale).

Methods

Design

A psychometric study with a multimethod approach was carried out. It was divided into four main stages [34], represented in Fig. 1.

Fig. 1.

Study stages

A guideline for reporting studies on measurement properties of patient-reported outcome measures was followed [35] (Additional File 1).

Stage 1: Item generation

A qualitative methodology was used to generate Basic-MAC scale items.

Participants

Experts were selected using purposive sampling. They needed to meet al.l the following criteria for inclusion: being Portuguese health professionals, having experience working in mental health education and with adolescents, regularly using validated assessment instruments or having developed one. Health professionals from different disciplines were considered to better define the construct, distinguishing it from perceived clinical competencies. We sought out an expert panel consisting of four to twelve experts [36]. Potential participants were identified by consulting our contact lists and literature on the topic. They were invited to join the study via email.

Data collection

An interview guide was developed and then pre-tested for three eligible healthcare workers. The script did not require changes. Focus group sessions were conducted via videoconference [36]. They were recorded (audio and video) and transcribed. Transcripts were shared with participants. After each session, participants were able to send, via email, suggestions on the topic and comments regarding the session transcripts. The discussions occurred until the participants produced no new information (data saturation) [37]. This point was identified when no additional codes emerged. It was also confirmed by the participants, who indicated that all their ideas were represented.

Data analysis

The collected data underwent content analysis [38]. Transcripts of discussion sessions and expert feedback messages were considered units of analysis. The content was organised into two meaning areas: psychometric instrument construct and rating scale. The units of meaning about the construct guided the writing of the items. The codes and categories expressed the core of the items. The themes indicated the possible domains of the scale.

Stage 2: Content validity assessment

A quantitative methodology was used to evaluate the content validity of the Basic-MAC scale and its items.

Participants

Experts were recruited in a similar way to the previous stage. However, a larger and more heterogeneous sample was sought to ensure greater validity of the results [39]. Portuguese health professionals with experience working in mental health education and/or with adolescents were recruited.

Data collection

A modified e-Delphi technique was performed [39]. The insights gained from the focus group sessions were included in a structured online questionnaire. The essentiality, relevance and clarity of ideas about the construct and agreement with ideas about the rating scale were assessed. Likert scales of essentiality with 3 points, relevance with 4 points, clarity with 3 points and agreement with 4 points were used [40]. Suggestions to improve the scale and its items were possible.

Pre-tested with ten eligible healthcare professionals, the questionnaire received no suggested changes. Subsequently, the questionnaire was emailed to the panel of experts for actual completion.

Data analysis

Frequencies of responses were analysed using IBM SPSS Statistics 27 software. Considering the responses concerning the ideas about the construct to be measured, the Content Validity Ratio (CVR), the Content Validity Index (CVI) (with respective Kappa Statistical Coefficient (κ)) and the average clarity score for each item were calculated [40]. Firstly, the critical CVR value was calculated using the formula proposed by Ayre & Scally [41]. Whenever the CVR values obtained for each item did not reach the critical CVR value, we discussed the possibility of eliminating the item. Secondly, the CVI was represented by the Item-CVI (I-CVI) and the Scale-level-CVI (S-CVI). The S-CVI was based on the average of the I-CVI scores of all scale items (S-CVI/Ave). I-CVIs ≥ 0.78 and S-CVI/Ave ≥ 0.9 were considered appropriate values for the scale [42]. Whenever the I-CVI value was lower than 0.78, the item was considered to be eliminated or revised. In turn, κ values between 0 and 0.20 indicated no agreement, 0.21–0.39 minimal agreement, 0.40–0.59 weak agreement, 0.60–0.79 moderate agreement, 0.80–0.90 strong agreement and above 0.90 almost perfect agreement beyond chance [43]. Finally, a mean clarity score ≤ 2 signalled unclear items. Revising the item based on participants’ comments was considered in these situations. Regarding ideas about the rating scale, a consensus of at least 75% was determined a priori [44]. The consensus was deemed positive when at least 75% of the responses were rated 3 or 4 on the Likert scale and negative when at least 75% were rated 1 or 2.

If a new idea emerged from the participants’ open-ended responses, we discussed its relevance and considered submitting it to the expert panel’s opinion.

Stage 3: Scale pre-test

A qualitative methodology was used to evaluate the adequacy of the scale items to the target population and the strength of the responses.

Participants

Participants were selected using purposive sampling. Portuguese adolescents aged 15 to 18 (upper secondary school students) with different sociodemographic characteristics were recruited until data saturation was reached [37]. Therefore, recruitment continued until the interviews no longer yielded new ideas or codes.

Data collection

Cognitive interviews were conducted in person. The questions focused on the content and form of the scale, as well as the mental process underlying its response. The interviews were audio recorded and transcribed shortly after they ended. Transcripts were returned to participants for comment.

Data analysis

The transcripts obtained in the pre-test were subjected to content analysis [38]. The units of meaning guided the improvement of the items. The content was organised into three areas of meaning: mental process underlying the response, content and form of the scale.

Stage 4: Scale application

A quantitative methodology was used to test the Basic-MAC scale and assess its psychometric properties.

Participants

Upper secondary school students aged 15 to 18 living in Portugal were recruited using snowball sampling. This approach was chosen for its practicality, efficiency, ability to reach adolescents from diverse contexts and potential to enhance participant engagement [45]. We disseminated the study to groups (school, cultural, religious and sporting) potentially comprised of adolescents and groups of guardians. We invited adolescents to participate and share the study with their social networks. A sample of at least 500 adolescents was sought (10 participants for each item on the scale) [34].

Data collection

The scale was administered online to adolescents. It was requested to be completed in 2 moments, spaced 7–14 days apart [46]. We provided a QRCode/link to potential participants that allowed them to access the scale for the first time. A notification was sent via email/SMS 7 days after the first completion of the scale to invite the second completion.

Data analysis

The answers given by the participants were analysed with descriptive statistics. The sensitivity of the items was examined based on the analysis of skewness and kurtosis. Serious deviations from the normal distribution of items were considered absolute values of skewness and kurtosis greater than three and 10, respectively [47].

Exploratory Factor Analysis (EFA) was performed to reduce the complexity of the data, identify underlying patterns and understand the relationships between variables [48]. Confirmatory Factor Analysis (CFA) was conducted to confirm the factor structure obtained in the EFA and evaluate its adjustment to the observed data [49]. For this purpose, the sample was pseudo-randomly divided (using IBM SPSS Statistics 27) into two equal-sized sub-samples. Difference tests were applied between the two subsamples for different sociodemographic characteristics (Students’ t-test for age and Mann-Whitney test for the remaining features).

The suitability of the data for EFA was assessed using the Kaiser-Meyer-Olkin (KMO) sampling adequacy measure and Bartlett’s test of sphericity [50, 51]. KMO ≥ 0.5 and p < 0.05 in the Chi-Square test were necessary assumptions [50, 51]. Considering this is exploratory research, we opted for the principal axis factoring method to extract factors in the EFA [52]. A Direct Oblimin rotation was used as the factors were expected to correlate [48]. A scree plot was created with eigenvalues plotted against the number of components, extracting factors until a sharp drop in eigenvalues occurs (Cattell’s Scree Test) [53]. A parallel analysis was also conducted, meaning the ideal number of factors was identified by comparing the real eigenvalues with the expected mean eigenvalues generated from random data [53]. Factors with real eigenvalues greater than the random eigenvalues were retained [53]. Items with factor loading greater than 0.50 and extraction communality greater than 0.40 were retained [48, 54]. After finding an acceptable factor solution, the researchers discussed the meaning of the pattern of factor loadings and assigned labels to each factor.

Cronbach’s Alpha Coefficient (α), McDonald’s Omega Coefficient (ω) and Composite Reliability (CR) were measures of reliability/internal consistency calculated for each factor [55]. Values above 0.70 were considered acceptable [56].

The CFA was performed with the maximum likelihood estimator. The factorial goodness of fit was assessed with global model fit indices. Chi-square (χ²), Root Mean Square Error of Approximation (RMSEA) and Standardised Root Mean Squared Residual (SRMR) were used to understand how well the model fits the data compared to no model. Comparative Fit Index (CFI) was used to compare the model fit with the assumption of variable independence. A chi-square value of χ²≤3df (df: degrees of freedom) with a p ≥ 0.01 was considered acceptable [47]. Rather than applying traditional fixed cutoffs for the RMSEA, SRMR, and CFI fit indices, dynamic and model-dependent thresholds were adopted, as proposed by McNeish and Wolf [57]. The adequacy of the sample size to detect differences in model fit was evaluated based on the RMSEA, using the application proposed by Jak et al. [58]. The null hypothesis (H₀: RMSEA ≥ 0.10) assumed a poor model fit, while the alternative hypothesis (H₁: RMSEA = 0.08) assumed a more acceptable fit [49]. The statistical power was estimated using the noncentral chi-square distribution associated with RMSEA [58]. Statistical power reflects the probability of correctly rejecting a false null hypothesis (i.e., detecting a better model fit when it exists) and is calculated as π = 1–β [58].

Measurement invariance was verified between male and female participants. Configural, metric, scalar and strict invariance were estimated [49]. Acceptable values were ΔCFI ≤ 0.01, ΔRMSEA ≤ 0.015 and ΔSRMR ≤ 0.03 for metric invariance, and ΔSRMR ≤ 0.015 for scalar or strict invariance [59]. A post-hoc power analysis for the multi-group confirmatory factor analyses (MG-CFA) was conducted using the SSpower() function implemented in the semTools package [60] in R (version 4.4.1) [61]. The analysis was based on the population model derived from the configural solution estimated with lavaan [62], which included the observed residual correlations between items. Following the recommendations of Jak et al. [58], violations of metric (Δλ = 0.10) and scalar (Δλ = 0.10; Δτ = 0.20) invariance were simulated.

Convergent validity was assessed by the Average Variance Extracted (AVE), with AVE ≥ 0.5 indicating adequacy [63]. If AVE < 0.5, acceptability remains with a CR >0.6 [63]. Divergent validity was examined by comparing each factor’s AVE with the square of the Pearson correlation between factors. AVE greater than the square of the correlation between factors demonstrated appropriate divergent validity [63]. Discriminant validity was assessed using the CICFA(sys) method by inspecting the 95% confidence intervals of the latent factor correlations [64]. The upper limits of these intervals were classified as follows: values equal to or greater than 1.0 were considered a severe problem, between 0.90 and 0.99 a moderate problem, between 0.80 and 0.89 a marginal problem, and below 0.80 indicative of no problem [64].

The scale’s test-retest reliability was assessed using Intraclass Correlation Coefficients (ICC) with a two-way mixed effects model, absolute agreement type and multiple raters/measurements relationship - ICC (2,k) [65]. ICC < 0.5 reveals poor reliability, 0.5–0.75 moderate, 0.75–0.9 good and >0.90 excellent [65].

IBM SPSS Statistics 27 and JASP 0.18.0 software were used for data analyses.

Results

Stage 1: Item generation

Nine participants were invited and took part in two focus group sessions (May 2 and June 14, 2022). Sessions lasted an average of 80 min. The sociodemographic characteristics of the experts can be consulted in Additional File 2.

Four themes about the construct were identified (Theme 1: Perceived interpersonal relationship competence; Theme 2: Perceived competence to recognise one’s own and others’ mental health status; Theme 3: Perceived competence for self-care and helping others to care for themselves; Theme 4: Perceived competence to seek mental health help and support others in seeking help). A theme on the rating scale was obtained (“5-point Likert Scale of Agreement”). The content analysis process is presented in Additional File 3. A first version of the Basic-MAC scale was developed (Additional File 4) and returned to the participants. They expressed that the version covered all the ideas discussed.

Stage 2: Content validity assessment

Ninety-seven participants completed the questionnaire between July 4th and 31st, 2022. A total of 162 healthcare professionals were invited, resulting in an approximate response rate of 60%. Since the online form required full completion before submission, all responses were complete, with no missing data recorded. The sociodemographic characteristics of the experts can be consulted in Additional File 5.

The experts’ opinion on the scale items’ essentiality, relevance and clarity is summarised in Additional File 6. The critical CVR value was 0.177 for n = 97. The experts expressed that most scale items were essential, relevant and clear. Only the item “I am willing to relate to people with good mental health” had a CVR and I-CVI value lower than the reference values. We nevertheless retained the item provisionally, since it appeared to capture a theoretically important contrast between attitudes toward individuals with and without mental health problems. This distinction has also been observed in previous studies [66]. Social attitudes are context-dependent and may vary according to the characteristics or state of others [67]. Therefore, the item can discriminate meaningful variation in attitudes. Its inclusion did not compromise the overall content validity of the scale, as the S-CVI remained excellent. Considering the exploratory nature of instrument development and the limited theoretical and empirical evidence on the construct, we decided to maintain the item for subsequent psychometric analyses to determine its adequacy.

A positive consensus (97.9%, n = 95) was also obtained on the idea about the rating scale (“Each item on the scale must have 5 response options (1 – Strongly Disagree, 2 – Disagree, 3 – Neither Agree nor Disagree, 4 – Agree, 5 – Strongly Agree).“). No new and relevant ideas about scale were suggested. Therefore, no changes were made to the scale, preserving all its items.

A summary of the results was sent to participants.

Stage 3: Scale pre-test

Eight participants were contacted and interviewed between September 26 and November 14, 2022. Interviews lasted an average of 52 min. The sociodemographic characteristics of the adolescents can be consulted in Additional File 7.

The content analysis process is presented in Additional File 8. Three themes emerged from the adolescents’ cognitive interviews (Theme 1: Need to review the introduction, items 6 and 40 of the scale; Theme 2: No need to change the content; Theme 3: No need to change the form).

The scale was considered globally understandable, clear and without bias. The adolescents recognised that all the items were distinct from each other despite identifying similarities (for example, S7 expressed, “There are words that are similar, but I can differentiate (the items)”). A preference for an informal form of address was reported when introducing the scale, particularly for the Portuguese expressions corresponding to “rate your”, “You have” and “Respond”. The adolescents also mentioned the need to remove the terms “verbal and non-verbal” in item 6 and to replace the term “conditions” with “influences” in item 40 to improve understanding of the items. Regarding the content of the scale, the adolescents indicated that each item addressed only one topic and that the scale “as a whole” focused on the intended construct. Furthermore, the form of the scale and items was considered adequate.

Therefore, changes were made to the Basic-MAC scale: formal expressions were revised to informal forms, incomprehensible terms were removed from item 6, and the term in item 40 was replaced. The revised version of the scale was returned to the participants, who expressed that it was adequate and met their opinions.

Stage 4: Scale application

Responses to the scale took place between March 12 and September 5, 2023. All submitted questionnaires were complete, with no missing data observed. No missing data analysis or case deletion was performed. The initial sample (n = 514) was randomly split into two different groups, the EFA (n = 217) and CFA (n = 217) subsamples. The sociodemographic characteristics of the sample and subsamples can be consulted in Additional File 9.

Additional File 10 (section ‘Item Sensitivity’) presents the descriptive analysis of the participants’ responses to the scale. The items showed good sensitivity since adequate values of skewness (between − 2.45 and − 0.33) and kurtosis (between − 0.94 and 7.67) were verified.

A four-factor structure was identified in the scale: (1) Perception of mental health first aid, (2) Perception of knowledge about mental health, (3) Perception of mental health self-care, and (4) Perception of interpersonal relationships (see Additional File 10, section “EFA”). These factors together represented 68% of the cumulative variability in the original variables and 60% of the cumulative variability in the extracted solution. The communalities remained adequate (0.41–0.77). The correlation matrix was suitable for factor analysis (KMO test of 0.89; Bartlett test with p < 0.01). The factors had adequate values of ω (0.79–0.92), α (0.79–0.92) and CR (0.70–0.89) (see Additional File 10, section “Internal Consistency”).

It was hypothesised that there would still be residual covariances between items of the same factors, as relationships between attitude-motivation, attitude-self-efficacy and motivation-self-efficacy have been described [22, 24]. Residual covariances could also be due to similarities between items reported in the pre-test (Additional File 8). Modification indices were applied with a cutoff of 11 (critical chi-square value for 1 degree of freedom with a significance level of 0.001). We recognised covariances between items 19–35, 32–33, 46–48, 43–45, 3–8. Fit indices are shown in Table 1.

Table 1.

Fit indices obtained from confirmatory factor analysis

Fit Indices	Results
χ2	χ2 = 443.731; df = 159 χ2/df = 2.791
	p < 0.001
RMSEA	0.08
SRMR	0.06
CFI	0.92

Although the χ² was less than 3 times the degrees of freedom, the associated p-value was significant. The model was further evaluated based on its characteristics: a moderate sample size (n = 257), relatively high factor reliability (Coefficient H ranging from 0.80 to 0.92), and middle loadings between 0.69 and 0.82. According to the Dynamic Fit Index (DFI) cutoff tables provided by McNeish and Wolf [57], models with similar middle loading and H values are expected to meet stricter thresholds under ideal conditions: SRMR ≤ 0.115 (0.051 in the DFI for n = 500), RMSEA ≤ 0.053 (0.067 in the DFI for n = 500), and CFI ≥ 0.972 (0.959 in the DFI for n = 500). While the obtained fit indices are close to traditionally acceptable values, they do not satisfy the more stringent criteria of dynamic cutoff points.

The factorial model of the scale is represented in Fig. 2.

Fig. 2.

Factor model of the Basic-MAC scale [Legend: The standardised coefficients are presented.]

Limited invariance was found between male and female adolescents. Adequate fit values were obtained, except in the χ² test and ΔCFI in all forms of invariance and also in the CFI and ΔSRMR of strict invariance (Table 2).

Table 2.

Invariance for sex (female vs. male)

Invariance level	χ2			Δχ2(Δdf)	RMSEA	ΔRMSEA	SRMR	ΔSRMR	CFI	ΔCFI
	χ2 (df)	p	χ2/df
Configural	660.326(318)	p < 0.001	2.076	-	0.09	-	0.07	-	0.90	-
Metric	675.784 (334)	p < 0.001	2.023	232.053 (175)	0.09	0.01	0.07	0.01	0.90	0.02
Scalar	705.675 (350)	p < 0.001	2.016	261.944 (191)	0.09	0.01	0.07	0.01	0.90	0.02
Strict	797.938 (375)	p < 0.001	2.128	354.207 (216)	0.09	0.01	0.08	0.02	0.88	0.04

Power analyses showed that the sample was sufficient for detecting acceptable model fit in the single-group CFA, but the multi-group CFA had limited sensitivity to detect small-to-moderate invariance violations. Full details are in Additional File 10 (section “CFA”).

Limited AVE values (0.43–0.56) were obtained (see Additional File 10, section “Convergent & Divergent Validity”). However, convergent validity was supported by good CR values. An adequate divergent validity was also verified since the AVE values were consistently higher than the square of the correlation between factors (see Additional File 10, section “Convergent & Divergent Validity”). The upper limits of the confidence intervals of the latent factor correlations were below 1.0 in all cases (see Additional File 10, section “Convergent & Divergent Validity”), supporting adequate divergent validity. Only the correlation between Factor 1 and Factor 3 presented a marginal problem (upper limit = 0.828), and all the others demonstrated no problems (upper limits < 0.80).

The scale was later responded to by 198 out of the 514 participants (retention rate: 39%). They responded at both times with an average interval of 9.35 days (SD: 2.68). The ICC varied between 0.77 and 0.80 (see Additional File 10, section “Temporal Stability”), expressing good test-retest reliability of the scale.

A final version of the Basic-MAC scale is presented in Additional File 11.

Discussion

This study sought to develop a scale that assesses the competencies perceived by adolescents to act in mental health (Basic-MAC scale) and examine some of its psychometric properties.

Experts recommended that the scale’s statements focus on perceived competencies in interpersonal relationships, recognition of mental health status, self-care, helping others and seeking mental health help. These areas align with the established domains of Mental Health Literacy [11].

Experts expressed that the scale statements were globally essential, relevant and clear, enabling adequate representation of the construct to be measured [34]. They suggested and agreed that the classification of statements should occur with a 5-point Likert scale of agreement. Five-point scales seem appropriate for this target population, as they are commonly used in school assessments [68].

Adolescents pre-tested the scale and suggested minor sentence revisions. These changes may have facilitated the processing and response to items, minimising misunderstandings and measurement errors [34]. Subsequently, a broader group of adolescents responded to the scale. Thus, it was possible to determine that the instrument detected specific differences/variations in the intended construct (good sensitivity of all items). Furthermore, the underlying structure of the scale was also explored and the existence of 4 latent factors was confirmed. It has a broader scope than the College Mental Health Perceived Competency Scale [33], enabling a more comprehensive identification of educational needs. The scale developed by Kalkbrenner & Sink focuses on perceived competence to support the mental health of others, while disregarding the dimension of self-care [33]. However, mental health first aid models highlight that supporting others is inseparable from self-care [7]. In this regard, the Basic-MAC stands out by valuing an integrated approach to mental health action. The scale’s content does not appear to overlap with perceived clinical mental health competencies scales (e.g., the Perceived Competence Scale for Disaster Mental Health Workforce [69]). This highlights that the scale specifically focuses on perceived basic competencies, differentiating them from the advanced ones expected of specialised mental health professionals.

The factorial structure obtained showed an acceptable overall fit. Although the chi-square test was statistically significant - a common outcome due to its sensitivity to sample size - the χ²/df remained within thresholds considered appropriate [70]. This suggests that, despite statistical significance, the observed differences may not be practically meaningful [70]. Recent methodological discussions emphasise assessing model fit using a combination of indices tailored to the model’s complexity, rather than rigidly adhering to traditional cutoffs [57]. Thus, while the results provide preliminary support for the model’s structural validity, they should be interpreted cautiously, considering model complexity and evolving methodological recommendations. The model may benefit from future refinements.

The factorial structure showed limited equivalence between male and female adolescents, suggesting that some items may be interpreted differently across genders. Observed score differences may not accurately reflect true variation in the underlying construct. Therefore, comparisons and generalisations between groups should be made cautiously. Future research should explore these potential differences in larger and more diverse samples, potentially revising or adapting items to ensure measurement equivalence and confirm the scale’s robustness across demographic subgroups.

The scale demonstrated adequate convergent and divergent validity. In this way, the different items on the scale consistently measure the same construct, discriminating it from others [34]. This suggests that the scores appropriately reflect perceived mental health competencies rather than unrelated characteristics. Good scale internal consistency indicators express that the Basic-MAC scale items are correlated with each other, measuring the same construct reliably [53]. The number of participants allowed the detection of ICCs ≥ 0.60 between the two observations, considering a confidence level of 95% and an assurance of 90% in a bilateral comparison [71]. Adequate ICC levels expressed measurement stability over time. In this way, the scale can measure real and stable variations in the construct and not merely random temporal fluctuations or measurement errors [65].

From the initial to the final version, the Basic-MAC scale items were related to the perception of knowledge, capacity, behaviour, attitude and motivation to act in mental health. The content of this scale partly converges with assessment instruments based on the Self-Determination Theory, such as the Perceived Competence Scale [72]. Despite several modifications for different non-mental health areas, the Perceived Competence Scale presents a generic model with 4 items relating to confidence perception, ability, goal achievement, and overcoming challenges [72]. The Basic-MAC scale goes deeper into the concept of “competence”, covering various elements that Salman et al. identified [2]. It captures the perception of task performance ability through items on knowledge, capacity and behaviour while gauging the personal importance of the task through items on attitude and motivation [25]. On one hand, the visible components of competence (knowledge, capacity, and behaviour) are indicators of an individual’s performance in tasks, reflecting their operational proficiency [2, 73]. On the other hand, the hidden components of competence (attitude and motivation) represent how the individual feels and engages with the task [24]. Motivation and attitude toward the task are crucial determinants for the sustainability of performance [24]. Therefore, the Basic-MAC scale can offer valuable insights into the effectiveness and persistence of adolescents’ performance in mental health issues.

Although the experts in our study generated and reached consensus on the different items of the scale, it is important to note that several assessment instruments do not include perceived knowledge as part of perceived competence (items such as “I know…”) [74]. Furthermore, our instrument is oriented towards action-related competencies, whereas the perceived knowledge items appear to focus more on theoretical foundations. Therefore, in the future, it will be pertinent to consider whether these items - corresponding to Factor 2 - should be maintained as an integral part of action-oriented competence, or whether, given their more theoretical nature, they should be considered separately from the instrument. Despite these considerations, the psychometric properties obtained in our study (notably content, convergent and divergent validity) were positive and support the inclusion of these items/Factor 2 in the scale. To facilitate future research, we also provide the psychometric properties of the instrument without the perceived knowledge items (see Additional file 12).

The Basic-MAC scale serves as an alternative tool for assessing perceived mental health competencies. In this regard, the scale was developed transparently and a comprehensive evaluation of its psychometric properties was conducted. However, this study has limitations. Firstly, only an inductive method was used to generate items (focus group with experts). A deductive method, such as a literature review on the concept, could have been carried out. Secondly, non-probability sampling was used at the different stages, limiting the generalisability of the results. The adolescent samples were not representative of the Portuguese youth population, as the participants’ characteristics were not aligned with national statistics [75]. Snowball sampling and the possibility of remote (online) responses also made it difficult to control the environment in which the adolescents filled out the scale. Environments with distractions may have influenced the appropriate completion of the scale. This non-probability sampling approach may have introduced selection bias, as adolescents more interested in mental health topics might have been more likely to complete the survey. Additionally, the snowball sampling method made it impossible to account for the number of people invited to complete the form, and the online format did not allow tracking of participants who started but did not finish the questionnaire, preventing calculation of a reliable completion rate. Thirdly, the time interval between test and retest was relatively short, which may have inflated reliability estimates. Although this interval is frequently reported in the literature, participants could have remembered their previous responses and this may not fully reflect the scale’s stability over time. Fourthly, the use of self-report among adolescents could introduce social-desirability bias, potentially influencing how participants responded to certain items. Finally, some important psychometric properties were not evaluated in this study, including predictive validity, invariance across subgroups with different characteristics and the scale’s sensitivity to change (responsiveness) over time. Therefore, the findings should be interpreted with caution in light of all study limitations.

Conclusions

The Basic-MAC scale was developed and its psychometric properties were explored. The experts’ opinions allowed the generation and consensus of the scale items. The instrument classifies adolescents’ agreement with statements about relating to others, recognising mental health status, taking care of themselves and supporting others, and seeking help resources.

The scale was pre-tested by adolescents and its wording was improved. The response of a larger number of adolescents allowed for the examination of the scale’s validity and reliability. A 4-factor structure was found in the instrument. The scale showed good item sensitivity, internal consistency and test-retest reliability, as well as adequate divergent and convergent validity. The favourable attributes of this instrument suggest it can effectively assess the construct in adolescents.

This study introduces a scale that represents visible and hidden components of competence, emphasising the perspective of the adolescent evaluated. The Basic-MAC scale quantifies adolescents’ perceived mental health action competencies and can provide evidence regarding the construct. For educators, the scale can help identify students’ mental health competence needs and inform the planning of supportive school programmes or initiatives. For clinicians, the instrument can support the assessment of adolescents, monitor individual progress, and tailor interventions to better address mental health needs. For researchers, the scale provides a tool to investigate the development of perceived mental health competencies, examine relationships with other outcomes, and evaluate the effectiveness of interventions. For managers, the data collected can guide resource allocation and the design of programmes that enhance adolescent mental health literacy and support. For policymakers, aggregated findings can inform policy development, set priorities, and implement strategies to strengthen mental health action competencies in adolescents. Better identification of educational needs and the effectiveness of interventions can ultimately enhance adolescents’ engagement and autonomy in managing their mental health.

The complementary use of actual competencies assessment instruments (e.g., knowledge tests) may be relevant. However, adolescents’ perception of competence can be influenced by comparison with measures of actual competence. We also acknowledge that perceived and actual competence may diverge (e.g., the level of knowledge an adolescent actually possesses may differ from what they perceive to know) due to factors such as overconfidence, familiarity, or other cognitive biases. Additional research could combine this scale with an objective measure of mental health competence to further explore convergent and discriminant validity.

Future psychometric studies with this scale are essential to better understand the instrument’s properties. Adaptations and validations are needed for its application in diverse populations and contexts. Therefore, this study represents an important first step in developing and evaluating the Basic-MAC scale and further research is required before its widespread application.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

α

Cronbach’s Alpha Coefficient

κ

Kappa Statistical Coefficient

χ²

Chi-square

ω

McDonald’s Omega Coefficient

AVE

Average Variance Extracted

Basic-MAC scale

Self-assessment Scale of Basic Mental Health Action Competencies

CFA

Confirmatory Factor Analysis

CFI

Comparative Fit Index

CR

Composite Reliability

CVI

Content Validity Index

CVR

Content Validity Ratio

df

Degrees of Freedom

DFI

Dynamic Fit Index

EFA

Exploratory Factor Analysis

ICC

Intraclass Correlation Coefficients

I-CVI

Item-Content Validity Index

KMO

Kaiser-Meyer-Olkin

RMSEA

Root Mean Square Error of Approximation

S-CVI

Scale-level-Content Validity Index

S-CVI/Ave

Scale-level Content Validity Index based on the Average method

SRMR

Standardised Root Mean Squared Residual

Author contributions

All authors made substantial contributions to the conception and design of the study, and the first author (TC) performed the acquisition, analysis and interpretation of data. The first author (TC) wrote the manuscript, and all authors critically reviewed its content. All authors approved the final version of the manuscript and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Data availability

The datasets generated and/or analysed during the current study are not publicly available due to privacy or ethical restrictions but are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Bioethics Committee of the Universitat de Barcelona (Institutional Review Board: IRB00003099; date: 21 February 2021) and the Ethics Committee of the Portuguese Society of Mental Health Nursing (Institutional Review Board: CEASPESM_1/2023; date: 10 March 2023).

Informed consent

was obtained from all individual participants included in the study. In the case of the participation of minors, authorisations from legal guardians were obtained. Data from all participants were coded: each Expert or Student was assigned a letter (E or S, respectively) followed by a number.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.