Validation of Metacognitive Awareness Inventory from a Private Medical University in India

Abstract

BACKGROUND:

Metacognition is the awareness and the capability to regulate one's own thinking process. Metacognition is critical in medical education for clinical reasoning and management. Hence, the objective of this study is to evaluate the construct validity and reliability of the Metacognitive Awareness Inventory (MAI) among first-year medical and dental students, from a private medical university in India, using confirmatory analysis and internal consistency method.

MATERIALS AND METHODS:

This was a cross-sectional study using convenient sampling. Fifty-two-item MAI was administered to 933 first-year medical and dental students from a private medical university in India. Exploratory factor analysis (EFA), principal component analysis, Cronbach's α, and confirmatory factor analysis with global fit indices were performed. Structural equation modeling (SEM) was performed to evaluate the relationship between the structural path and factors using AMOS version 22.

RESULTS:

During EFA, 12 items with <0.40 factor loadings were trimmed sequentially. The remaining items with respective factors had a good internal consistency of ≥ 0.9. Comparative fit index (0.78), goodness-of-fit index (0.8), adjusted goodness of fit index (0.77), Tucker–Lewis index (0.7), standardized root mean square residual (0.06), and root mean square error of approximation (0.09) values showed that six-factor model fits to satisfactory. Pearson's correlation coefficient was found to be high between factors (>0.80). SEM for each item (observed) and factor (unobserved) illustrated the hypothesized model.

CONCLUSION:

The resultant 40-item model based on MAI designed by Schraw is a valid and reliable tool for assessing the metacognitive awareness of Indian students. Employing a valid and reliable tool in assessing the metacognitive awareness will help the academicians in incorporating appropriate curricular interventions.

Introduction

Metacognition is the term first used by the prominent Developmental Psychologist John Flavell in the field of Psychology where he refers to it as “thinking about thinking.”[1] The documented associations exist between metacognition, academic performance, and other related processes such as planning (P), assessing, and self-monitoring.[2] Metacognitive awareness includes one's own knowledge about cognition and the methods of regulation of cognition.[3] Metacognition involves several physiological mechanisms in the brain, especially the limbic system, which is the seat for learning, attention, memory and reticular activating system.[4] Studies have shown that students with higher metacognitive awareness perform better in academics, problem-based learning, language acquisition, teamwork, group learning, and communication all of which are very crucial for medical training.[5,6] Metacognitive process assumes significant importance in medical education as it aids in critical thinking, accurate judgment capabilities, minimizes diagnostic errors, and facilitates reflective practices.[7] Metacognitive skills can be enhanced using several teaching–learning methods such as Problem-Based Learning, reflection, feedback, Self Directed Learning (SDL) sessions and flip classroom method.[8,9] Assessment of metacognition periodically helps in implementing appropriate teaching strategies. There are several quantitative and qualitative tools available for assessment of metacognitive awareness, of which metacognitive awareness inventory (MAI) designed by Schraw and Dennison is a widely used established tool.[10] It is a type of verbal report method where all the aspects of thinking can be accessed which are not directly observable.

Convergent validity, divergent validity, and reliability of the 52-item MAI needs to be evaluated before using in different student population as suggested by the original authors.[10] Hence, this study was aimed to evaluate the validity and reliability of the MAI among the first-year Indian medical and dental students. The objectives of the study were to test the construct validity of MAI factor analysis and reliability of the MAI by applying internal consistency method.

Materials and Methods

Study design and setting

A cross-sectional study was conducted to evaluate the construct validity and reliability of the MAI among first-year medical and dental students, from a private medical university in India.

Study participants and sampling

Three batches (2015–2018) of undergraduate medical (250 students/batch) and dental students (100 students/batch) at a private medical university in South India were invited to participate in this study (n = 1050). After giving informed consent, 933 students participated and completed the MAI questionnaire. A convenient sampling technique was followed, and all the entries were included for analysis. All students who expressed willingness were included in the study. There were no exclusion criteria followed in this study.

Data collection tool and technique

MAI questionnaire containing 52 items was used, after getting permission from the author (Dr. Schraw). In Schraw's model, the items under “knowledge of cognition” were grouped under declarative knowledge (DK, knowledge about self and strategies, 8 items), procedural knowledge (PK, knowledge about the use of procedure, 4 items), and conditional knowledge (CK, knowledge on when and how to use the strategies 5 items), whereas “regulation of cognition” were grouped as P (setting the goal, 7 items), strategy (S, implementing strategies, 10 items), monitoring (M, self-assessment, 7 items), debugging (D, correction of errors, 5 items), and evaluation (E, performance analysis, 6 items).[10,11] The validity and reliability of MAI using different scoring models were documented by many researchers in different populations (Harrison G, 2018; Akin A 2007).

MAI questionnaire was administered as a hard copy to students during their physiology lecture class, after getting formal approval from the department of physiology. Dedicated time was allotted for the students to answer MAI.

Prior to the primary analyses, duplicate and invalid data were examined first. Next, normality using histogram and outliers using box plot were examined for the purpose of using factor analysis. There were no outliers identified and the distribution was approximately normal. Due to the large sample size, normality and no outliers met the assumption for factor analysis.

Exploratory factor analysis (EFA) with principal component analysis was performed to examine the factor structure. Items with factor loadings roughly >0.40 were considered for the model. Reliability using Cronbach's α was used to assess the internal consistency of each factor.

The resultant MAI was administered to 933 students and confirmatory factor analysis (CFA) was performed to assess goodness-of-fit model for the global fit of the model by R statistical version 4.0.2(R Foundation for Statistical Computing, Vienna, Austria). These indices include χ² and its subsequent ratio with degrees of freedom (χ²/df); comparative fit index (CFI), goodness-of-fit index (GFI), adjusted goodness-of-fit index (AGFI), Tucker–Lewis index (TLI), standardized root mean square residual (SRMR), and root mean square error of approximation (RMSEA). Structural equation modeling (SEM) was performed to evaluate the relationship between the structural path and factors using AMOS version 22 (IBM SPSS, Chicago).

Ethical consideration

This study was approved by the institutional ethics committee (IEC-NI/14/Dec/44/91). Informed consent was obtained from all the students. Permission was obtained from the author (Dr. Schraw) for using the 52-item MAI questionnaire.[10]

Results

Nine hundred and thirty-three students were included in the analysis. The participants belonged to the age group of 18 years–20 years and 68% were female students. Factor loadings from the EFA analysis with <0.40 were trimmed from the model sequentially. In total, 12 items were removed: five from CK (no. 5, 18, 26, 29, and 35), four from P (no. 22, 23, 42, and 45), one from S (no. 48), and two from M (no. 34 and 49). During CFA, the remaining items loaded under six factors, namely DK (7 items), PK (3 items), CK (4 items), P (13 items – few items under the original domain S also loaded here), M (8 items – few items under the original domain D also loaded here), and E (5 items), as represented in Table 1. All the extracted factors had a good internal consistency of ≥0.9.

Table 1.

Principle component analysis with factor loadings and communalities (h²) of each item

Factors	Items	Factor loading	h2
DK	DK5	0.498	0.341
	DK10	0.558	0.514
	DK12	0.497	0.586
	DK16	0.467	0.355
	DK17	0.452	0.535
	DK20	0.529	0.472
	DK32	0.597	0.409
PK	PK14	0.516	0.344
	PK27	0.652	0.409
	PK33	0.538	0.556
CK	CK18	0.553	0.40
	CK26	0.485	0.448
	CK29	0.405	0.627
	CK35	0.551	0.278
P	P6	0.60	0.44
	P8	0.663	0.435
	P22	0.46	0.518
	P23	0.478	0.606
	P42	0.555	0.432
	P45	0.56	0.473
	P30	0.489	0.64
	P31	0.59	0.442
	P37	0.479	0.636
	P39	0.488	0.458
	P43	0.513	0.431
	P47	0.537	0.412
	P48	0.493	0.351
E	E19	0.476	0.47
	E24	0.528	0.487
	E36	0.639	0.37
	E38	0.639	0.539
	E50	0.489	0.433
M	M1	0.625	0.474
	M2	0.424	0.502
	M11	0.406	0.422
	M21	0.513	0.439
	M28	0.587	0.487
	M49	0.608	0.55
	M25	0.399	0.427
	M44	0.539	0.526
Cronbach’s alpha		0.93
% variance		47.68
KMO		0.91
Bartlett’s test of Sphericity, approximate Chi-square		3644
df		579
Significant		0.001

Factors were extracted by maximum likelihood method and rotated by varimax rotation. h²=Communalities for each item, % variance=Percentage of variance explained, KMO=Kaiser-Meyer-Olkin, DK=Declarative knowledge, PK=Procedural knowledge, CK=Conditional knowledge, P=Planning, E=Evaluation, M=Monitoring

Pearson's correlation coefficients were calculated to explore the inter-relationships between factors [Table 2].

Table 2.

Correlation matrixes for the factors of metacognitive awareness inventory

	DK	PK	CK	P	E	M
DK	1
PK	0.884	1
CK	0.949	1.179	1
P	0.795	0.985	0.999	1
E	0.649	0.834	0.856	0.951	1
M	0.733	0.986	0.925	0.983	0.937	1

P=0.000. All the factors showed a relatively high correlation. DK=Declarative knowledge, PK=Procedural knowledge, CK=Conditional knowledge, P=Planning, E=Evaluation, M=Monitoring

A summary for the goodness-of-fit indices from the CFA is displayed in Table 3. CFI (0.82), GFI (0.86), AGFI (0.83), SRMR (0.05), and RMSEA (0.06) values represent that six-factor model fits to satisfactory.

Table 3.

Goodness-of-fit indices for the factors of metacognitive awareness inventory

Statistics	GFI	AGFI	NFI	CFI	RMSEA	SRMR	TLI	χ 2	df	χ2/df	P
Model fit for basic model with six domains	0.86	0.83	0.78	0.82	0.06	0.05	0.79	2547	544	4.8	0.001

GFI=Goodness-of-fit index, AGFI=Adjusted GFI, NFI=Normed fit index, CFI=Comparative fit index, RMSEA=Root mean square error of approximation, SRMR=Standardized root mean square residual, TLI=Tucker-Lewis index, χ²=Model Chi-square, df=Degrees of freedom

The results of CFA were accepted by AMOS and no notification messages were obtained from AMOS about analyzed parameters. This illustrates that the resultant factor model passed the essential stage for identification. Then, the items (observed) and factors (unobserved) were exemplified in the hypothesized model [Figures 1 and 2]. The factors were characterized as rectangles; ellipses represented the items; and the circles symbolized measurement errors. The regression paths were shown as arrows between the items and factors, whereas the numerical values on those lines represented standardized regression weight. The arrows connecting the small circles and items were nothing but a measurement of error term. The correlation of covariance of the model was demonstrated by the double-headed arrows running between any two factors. SEM path for knowledge of cognition [Figure 1] and regulation of cognition [Figure 2] also illustrated for each item (observed) and factor (unobserved) in the hypothesized model.

Figure 1.

SEM results of the confirmatory factor analysis for the Knowledge of cognitive factors

Figure 2.

SEM results of the confirmatory factor analysis for the Regulation of cognitive factors

The results of this study indicate that the MAI questionnaire is a valid and reliable tool for Indian medical and dental students as the reliability analysis indicates high internal consistency for all the scales (≥0.9). CFA was conducted to authorize the factor structure of the MAI. Based on the factor loading in the EFA analysis, 12 items were removed and the extracted factors had a good internal consistency and demonstrate excellent fit with a final set of 40 items. The lowest factor loading of 0.24 was observed for item 19 (I ask myself if there was an easier way to do things after I finish a task) followed by 0.29 for item 15 (I learn best when I know something about the topic).

Discussion

The results of construct validity and reliability of the MAI among first-year students of Indian Private University indicate that the MAI questionnaire is a valid and reliable tool with high internal consistency for all the scales.

The CFA analysis has provided valid and reliable results, similar to the analysis results of original author Schraw and Dennison, 1994.[10] and supports the applicability of the MAI for Indian health professional undergraduate students. Studies conducted by previous authors in the area of MAI validation indicated that it is an authentic tool to assess the metacognitive ability of students in different fields of undergraduate education.

Validation of MAI among Indonesian medical students showed that it complies to construct validity criteria, specifically content validity and internal consistency, and the author suggests MAI as a useful instrument to assess various aspects of metacognitive skills in the academic stage of undergraduate medical education.[11]

There are also other studies in literature which have examined the validity of MAI translated in different languages with the university students as study subjects with a mean age of 20 years. Validation of Turkish version of MAI,[12] Spanish version of MAI,[13] and Chinese version of MAI[14] among university students found the instrument to be valid and reliable with high internal consistency and recommended for usage in the field of education.

Health profession education requires training students at higher order thinking and performance which are essential to handle cognitive complexity in uncertain clinical situations in the field of health profession.[15] The development of metacognitive awareness is one of the fundamental skills to be achieved among health professional students during their undergraduate training to adopt critical thinking and problem-solving skills in lifelong professional practice.[16] Metacognitive skill can be acquired through training like any other psychomotor skills[17] and metacognitive awareness can be created among health profession students to bring about academic success.[18] Hence, validation of MAI among a specific group of students is essential to increase its applicability of its use among the specific population.

Metacognitive awareness increases with age and all our students belonged to the narrow age range of 18–20 years and all of them were in their first year of education; hence, we did not further stratify them into different age categories. About 68% of the students in our study were females and we did not group the data of males and females separately for analysis.

Studies have shown that there are no clear differences in metacognition between males and females, and also it is not related to age.[19,20] However, some authors could find a gender difference in few aspects of metacognitive awareness, the scores of knowledge of cognition and regulation were found to be more in females[21,22] compared to males in their study subjects.

MAI tool has been used for assessment among postgraduate Indian students.[23] However, there are apprehensions, about weak correlations between calculated metacognitive scores and academic success. This is partly explained by lesser reporting by low performers (as they rarely indulge in metacognition) and high performers (as they have already programmed their metacognition into default use) as well.[24] The added reason could be the lack of validation of the instrument applied for measuring metacognition, which is addressed in this study.[17]

Periodic assessment of metacognitive awareness among students is a useful exercise as it can aid teachers to incorporate appropriate teaching–learning interventions to augment the metacognition and, in turn, improve the academic performance of the students. Metacognitive awareness can be enhanced by innovative teaching methods such as CBL, flip classroom, and reflective practice.[8,9] MCI has included several innovative teaching–learning methods such as foundation course, early clinical exposure, SDL, and AETCOM in the recently revised undergraduate medical curriculum where students can be taught to reflect on their own thinking.[25] Developing metacognitive strategies is important for reaching the goal of an Indian medical graduate as envisioned by MCI.[25,26]

The limitation of the study is that the MAI was administered only to the first-year students, hence cannot be generalized to the students belonging to other years. Metacognitive awareness increases with age; hence, periodic assessment is required to demonstrate the ascendency of the higher order thinking. This validated questionnaire can be administered for the students belonging to advanced years as well as postgraduate students. Additional qualitative research can be conducted on this scale.

Conclusion

Metacognition is required for learner self-direction where learners can decide what, how, and when to discover their learning needs. The resultant 40-item model based on MAI designed by Schraw is a valid and reliable tool for assessing the metacognitive awareness of Indian students. Metacognition is closely linked with academic performance; hence, having a valid and reliable tool in assessing the metacognitive awareness will help the academicians in incorporating appropriate curricular interventions to augment the metacognition which, in turn, will enhance the academic performance and competency of the students.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.