Psychometric evaluation of Maastricht clinical teaching questionnaire based on the cognitive apprenticeship model

Abstract

BACKGROUND:

The Maastricht Clinical Teaching Questionnaire is a valid and reliable instrument for measuring the quality of clinical teaching. This study was aimed at translation and psychometric evaluation of the Persian version of this questionnaire to evaluate clinical teaching based on the cognitive apprenticeship model.

MATERIALS AND METHODS:

The translation of the questionnaire was performed according to Guillemins framework. Reliability was examined by calculating Cronbach's alpha coefficient. Confirmatory factor analysis was studied among 120 medical students. Content validity was assessed by calculating the content validity index and content validity ratio. Face validity was evaluated by conducting interviews with students using concurrent verbal probing and thinking aloud.

RESULTS:

Cronbach's alpha coefficient for the whole scale was 0.95. The content validity index was 0.92, and the content validity ratio was 0.82. Confirmatory factor analysis resulted in a seven-factor model and demonstrated an adequate fit with the data.

CONCLUSION:

The Persian version of the Maastricht Clinical Teaching Questionnaire with seven factors including modeling, coaching, scaffolding, articulation, reflection, exploration, and learning environment appears to be a valid and reliable instrument for the evaluation of clinical teaching in Iranian universities of medical sciences.

Introduction

Clinical teaching as the heart of medical education provides the opportunity for medical students to obtain clinical skills.[1] Clinical teaching is a process in which medical students gradually acquire competencies at the patient's bedside and get prepared for clinical care.[2] This type of education as the most appropriate way to teach medical students as adult learners is an active learning process providing an opportunity for students to turn their theoretical knowledge into the affective and psychomotor skills.[3] Clinical teaching improves the quality of clinical decision-making and provides the possibility of observing and applying theoretical learning in a real environment.[4]

Nevertheless, the main activity of clinical settings is providing medical services to patients, and a smaller amount of these activities deals with clinical teaching. In this situation, a clinical teacher, as a key factor in successful clinical education in a complex clinical environment, has a significant and positive impact on education.[5] Considering the amount of energy and time that clinical teachers spend teaching medical students in clinical settings, accurate evaluation of clinical teaching is necessary to identify the strengths and weaknesses and ways to improve this education.[6]

Several instruments have been designed and used to evaluate the clinical teaching. One of the most cited instruments is Stanford List, which has a clear theoretical foundation, but it emphasizes too much on a wide collection of teaching arrangements such as small groups. This focus makes the instrument less appropriate for clinical teaching evaluation.[7]

Another instrument is the Cleveland Clinical Teaching Effectiveness Instrument (CCTEI), which involves stakeholders in its design process, but the lack of clearly specified theoretical dimensions could decrease its effectiveness.[8]

An evaluation instrument should have a clear theoretical basis to provide a foundation to compare findings and use evaluation results in a way that improved clinical teaching practice.[9] The cognitive apprenticeship model is a fundamental theory in clinical education which is suitable for teaching a variety of tasks in complex situations and involves learning through guided experience rather than focusing on processes.[10] This model contains six dimensions including modeling, coaching, scaffolding, articulation, reflection, and exploration.[11] The theoretical constructs of cognitive apprenticeship model support the Maastricht Clinical Teaching Questionnaire (MCTQ). This instrument was developed by Stalmeijer et al. (2010) to afford clinical teachers with feedback about their teaching skills at the workplace. This questionnaire consists of 24 items evaluating the six dimensions in the cognitive apprenticeship model and the learning environment of the clinical teaching.[12]

In the modeling dimension, the student observes the teacher while performing daily clinical tasks and acts as a role model for students. In the coaching dimension, the teacher observes students performing a task, gives them feedback, and supervises the student's performance. In the scaffolding dimension, the clinical teacher supports students in performing tasks, and then the support is gradually removed so that the learners can continue their activities independently. In the articulation dimension, the clinical teacher involves the student in the discussion by asking questions and helps the student think about their activities and state the reasons for the things they do. This causes the learners to clearly articulate their knowledge, reasoning, or problem-solving processes. In reflection dimension, the clinical teacher stimulates students to reflect on their performances and compare them with those of experts and other students. In the exploration dimension, the teacher encourages autonomy in students by asking them to formulate and pursue their own personal learning goals.[13]

The MCTQ, by recognizing the weaknesses, strengths, and areas of improvement in clinical teaching, was reported as a valuable instrument to improve the quality of medical education and can also benefit faculty development programs to evaluate their efficacy. The psychometric evaluation of this questionnaire has been investigated in different medical contexts. Boerboom et al. (2011) studied the reliability and validity of the MCTQ as an instrument to evaluate individual clinical teachers during short clinical rotations in veterinary education. They reported that the content validity was supported by the cognitive apprenticeship model. A five-factor model showed a good fit with the data. They resulted in the fact that the MCTQ appears to be a valid and reliable instrument to evaluate clinical teachers' performance during clinical rotations.[14] Giannasi et al. (2019) performed a study for cross-cultural adaptation of the MCTQ by content and construct validation and reliability assessment. They reported that the content and answering process were validated. A five-factor model showed a good fit with the data. The Cronbach's alpha coefficient was 0.80. They identified that the Spanish version of the MCTQ with the Cronbach's alpha coefficient of 0.80 is a valid and reliable instrument for evaluating clinical teaching.[15]

Educational practices are context-specific; therefore, setting is a key component that can influence psychometric properties in instruments that evaluate the quality of teaching.[16] Therefore, psychometric evaluation of this questionnaire across many different contexts will reinforce its application as well as establish generalizability of the findings.

With the current international movement toward medical education improvement and Iranian medical universities being no exception, the ability to evaluate the clinical education is becoming a crucial component of the medical education enhancement. The results of the clinical educator's evaluation provide learning opportunities for educators to help them to enhance teaching abilities.[17]

Despite the widespread use of MCTQ in different contexts, this questionnaire has not been psychometrically evaluated and used in Iran. This study aimed to translate the MCTQ and evaluate psychometric properties of the Persian version. The findings may lead to application of this instrument to evaluate clinical teaching based on the cognitive apprenticeship model and as an ultimate goal help policy-makers of faculty development programs to recognize the current condition of the clinical teaching process and inform the future faculty development programs.

Materials and Methods

Study design and setting

The research was a cross-sectional study that was conducted at Kerman University of Medical Sciences in Iran between March 2022 and July 2022. The inclusion criteria were being interested in participating in the study, being a medical student, and having experience in clinical departments. The exclusion criteria were questionnaires with more than 10% of questions without answers.

Ethical consideration

The KMU's institutional review board approved the study (No. IR.KMU.REC.1400.703) on January 2022. The participants did not receive any incentives, and participation was voluntary. Verbal and written consent for participation was obtained based on the proposal approved by the ethics committee. The participants were also assured of the confidentiality of their information, and it was explained that the results would only be used for research objectives.

The original version of MCTQ is composed of 24 items in seven domains, which involve a set of items scored on a five-point Likert scale from fully disagree to fully agree. The modeling domain with four items evaluates whether the clinical teacher performs a task by acting as a role model and provides an opportunity for students to observe and complete the task. The coaching domain with three items measures whether the clinical teacher observes students while carrying out a task and gives them feedback. The articulation domain with four items measures whether the teacher asks students to explain their reasons for different actions and helps to be aware of gaps in their knowledge and skills. The scaffolding domain with four items assesses whether the clinical teacher provides support to the level of the student and gradually reduces that support as the student progresses. The reflection domain with two items evaluates if the teacher encourages students to become aware of their strengths and weaknesses and reflect what they can do to improve. The exploration domain with three items assesses whether the teacher asks students to formulate and pursue their learning objectives based on identifying strengths and weaknesses. The learning environment as the final domain with four items involves items that intend to measure if the tutor generates a safe learning environment and treats students with respect.

Translation Process

The translation process was performed using the Guillemin framework.[18] First, two English language experts separately translated the questionnaire into Persian. Second, two medical education faculty members and two clinical faculty members compared both translated versions and made the necessary corrections in terms of ambiguous words, sentence structure, and meaning of the sentence. Third, two English language experts investigated and confirmed the changes. Fourth, the translated versions were reviewed and compared with the original instrument by two medical education faculty members and two faculty members in terms of conceptual, semantic, and content equivalence. The instructions to answering the questionnaire were also checked. Finally, the pre-final version was compiled.

Psychometric evaluation

Content validation

The content validity of the initial MCTQ was investigated both quantitatively and qualitatively by expert opinion. Ten experts from faculty members were recruited based on their experience in clinical education and their expertise in medical education. They were selected within Kerman University of Medical Sciences. Experts were asked to consider each item of the MCTQ based upon the criteria of “essential,” “relevance,” “clarity,” and “simplicity.” Each item was assessed using Likert scales: A three-point scale for “essential” (1, unessential; 2, useful, but not essential; and 3, essential,) and four-point scales for “relevance” (1, not relevant; 2, rather relevant; 3, relevant; and 4, completely relevant) and “clarity” (1, not simple; 2, rather simple; 3, simple; and 4, completely simple) criteria. In addition, the experts were asked to provide comments about the “simplicity” of each item (fluency and using simple and understandable words) as well as the most appropriate placement and order of the items.

We examined content validity by computing the content validity ratio (CVR) and content validity index (CVI) using ratings of item relevancy that were highlighted by the content experts.[19]

Given the ten experts who evaluated the items, the minimum acceptable amount of CVR was 0.62 based on Lawshe table. The formula for calculating CVI in Waltz and Bausell method is the number of all the respondents in “relevancy,” “clarity,” and “simplicity” criteria divided by the number of experts who have scored 3 or 4 in the relevant question in that criterion. In this formula, if an item has a score more than 0.79, that item is retained in the questionnaire. If CVI is between 0.70 and 0.79, the item is questionable and needs correction and revision. Furthermore, if it is less than 0.70, the item is unacceptable and it must be deleted.

The corrective comments of experts about the wording of items, such as fluency, using simple and understandable words, and the suitable placement of the words, were used. Two items were revised to increase the ease of understanding the wordings. The overall CVR was 0.82, which was acceptable. The CVI for all items was 0.91 by using Waltz and Bausell method (in terms of relevance 0.95, clarity 0.90, and simplicity 0.90). Two items with CVI ˂ 0.70 were removed as they were identified as being vague or similar to other items. Five items were revised and accepted.

Face validation

Students' opinions were used to check the face validity. In this regard, interviews were conducted with ten medical students using concurrent verbal probing and thinking aloud. The questionnaire items were examined in terms of fluency, appropriate phrasing, avoiding specialized words, and potential ambiguity. Based on the students' opinions in the face validation process, item 19 was modified and accepted. This item was changed from (the clinical teacher encouraged me to learn new things) to (the clinical teacher encouraged me to learn new content).

Construct validation

The modified MCTQ based on content and face validation was sent to 150 medical students who were clerkships and interns and included in the study by census method. The sample size was chosen based on the recommendation for confirmatory factor analysis of 5–10 persons per parameter estimates in the measurement model.[20] The questionnaire was redistributed two more times at approximately 2 weeks intervals via E-mail and also followed up through the social media.

For investigating the construct validity, first, exploratory factor analysis (EFA) followed by a varimax rotation was applied to determine the factorial structure of the questionnaire. We applied the Kaiser–Meyer–Olkin (KMO) test and Bartlett's test to assess the sample adequacy and sphericity of the MCTQ, respectively. A KMO value equal to or above 0.70 and a significant Bartlett's test of sphericity were considered as acceptable criteria for sample adequacy and factorability of correlation matrix. The criteria for keeping the factor for this study were extraction values above 0.32 and Eigen values above 1.0.

In the next step, a confirmatory factor analysis (CFA) was performed to examine and verify the assumed seven-factor structure of the MCTQ with LISREL software (8.8 version. New Jersey). Several fit indices were carried out to assess the fit of the hypothesized model to the data: comparative fit index (CFI), goodness of fit index (GFI), and adjusted goodness of fit index (AGFI), with values of about 0.9 considered adequate, standardized mean square residual (SRMR), and root mean square error of approximation, which should approximately be equal to or less than 0.08 to be indicative of adequate fit of the model to the data.[21]

Reliability assessment

The internal consistency of the MCTQ was investigated by Cronbach's alpha. An internal consistency of more than 0.7 was considered suitable.

Results

Demographic data

All ten experts completed the content validation form. The majority of them (60%) were women, 50% were assistant professors, 30% were associate professors, and two participants were professors. The final number of participants who completed the questionnaire for investigating construct validity was 120 of the 150 recruited, yielding a response rate of 80%. Most participants (64.3%) were female. About half were interns (53%). Most of the respondents were affiliated with the internal medicine department (65%), and 19% were from the general surgery department. The pediatrics respondents (9%) were almost equal in number to the respondents from the obstetrics and gynecology department (7%).

Construct validity

EFA and sample size adequacy were examined using the SPSS software. The results of the KMO test and Bartlett's test indicated ample adequacy of the sample size and factorability of correlation matrix for conducting EFA (KMO index = 0.923, P < 0.001, Bartlett's test = 3645.222 and df = 210). Viewing of the scree plot revealed seven factors with eigen values greater than 1, and these factors explained 61.4% of the total variance.

The results of CFA showed an appropriate fitness for the seven-factor structure of the questionnaire (RSMEA: 0.07, GFI: 0.77, AGFI: 0.71, CFI: 0.97, NNFI: 0.97, and SRMR: 0.052). The results of CFA are shown in Figure 1.

Figure 1.

Confirmatory factor analysis results

Reliability assessment

The Cronbach's alpha coefficient for all items of the questionnaire was 0.95. The Cronbach alpha coefficients for “Modeling”, “Coaching”, “Articulation”, “Scaffolding”, “Reflection”, “Exploration”, and “Learning environment” were 0.84, 0.63, 0.82, 0.79, 0.70, 0,88, and 0.88, respectively, which were suitable.

Production of the final questionnaire

After investigating reliability and validity, the Persian version of MCTQ with 22 items in seven domains was finalized. These domains included “Modeling” with three items, “Coaching” with three items, “Articulation” with four items, “Scaffolding” with four items, “Reflection” with two items, “Exploration” with three items, and “ Learning environment” with three items. A copy of the final Persian version of MCTQ is included in Appendix 1.

Discussion

This research described the translation process and psychometric evaluation of the Persian version of the MCTQ to evaluate clinical teaching based on the cognitive apprenticeship model. The initial MCTQ included 24 items, and after validation, 22 items were retained. The removed items included item 4 (the clinical teacher was a role model as to the kind of health professional I wish to become) and item 23 (the clinical teacher was genuinely interested in me as a student). The results of the CFA indicated that the seven-factor model fits the data reasonably well. Two items were deleted through content validation. The MCTQ was established based on the cognitive apprenticeship model that was known as a flexible and logical model in rapidly changing clinical education environments and has been validated as a useful approach to design, implement, and evaluate educational practices in clinical teaching.[22] This model is effective in developing the knowledge, skills, and attitudes of medical students to acquire professional competencies, providing feedback to clinical teachers about their clinical teaching practice.[23]

Even though there are no studies reporting the validity evidence of this questionnaire in Persian, our results are closely aligned with the previous international published work on the psychometric evaluation of MCTQ. Giannasi et al. (2019) performed a study to investigate the reliability and validity of the Spanish version of the MCTQ. The content and answering process reported valid. By conducting a CFA, a five-factor model with 14 items showed a good fit with the data. The reflection domain was removed from the questionnaire. The Cronbach's alpha coefficient in this questionnaire was 0.80. They identified that the Spanish version of the MCTQ is a valid and reliable instrument for evaluating clinical teaching.[15]

Because of the small number of questions in this questionnaire, answering requires a short time, and it is possible to easily collect students' opinions about clinical teaching. As a result, it can be used as a suitable instrument to investigate the strengths and weaknesses of clinical teaching and provide keys to improvement.

Some limitations of this research should be considered. The study was performed in one organization; therefore, findings may not be generalizable to other contexts. Further research performed at other universities is recommended to determine if these findings generalize over one institution. Also, as a methodological defect, the opinions of students of different years about the evaluation of clinical teaching were not investigated because the effect of the cognitive apprenticeship model may be different in different stages of education. For example, students who have just started clinical rotations need more observation and support than more senior students. Future research should be conducted separately on students in diverse years of their education to show that this instrument can be used for medical students in different stages.

Conclusion

The current study supports the validity and reliability of the Persian version of the Clinical Teaching Questionnaire with seven factors including modeling, coaching, scaffolding, articulation, reflection, exploration, and learning environment as an instrument for evaluating the teaching skills of clinical teachers.

Financial support and sponsorship

The KMU's institutional review board approved the study (No. IR.KMU.REC.1400.703).

Conflicts of interest

There are no conflicts of interest.

Appendix 1

Health profession you are studying (e.g. nursing, physiotherapy, medicine etc): _______________

Your year of study: ________________

Please indicate your level of agreement with the following statements:	Fully agree	Agree	Neutral	Disagree	Fully disagree
This clinical supervisor:
1. consistently demonstrated how different tasks should be performed
2. clearly explained the important elements for the execution of a given task
3. created sufficient opportunities for me to observe them.
4. observed me multiple times during patient encounters
5. provided me with useful feedback during or following direct observation of patient encounters
6. helped me understand which aspects I needed to improve
7. Adjusted teaching activities to my level of experience
8. Offered me sufficient opportunities to perform activities independently
9. Supported me in activities I find difficult to perform
10. Gradually reduced the support given to allow me to perform certain activities more independently
11. Asked me to provide a rationale for my actions
12. Helped me to become aware of gaps in my knowledge and skills
13. Asked me questions aimed at increasing my understanding
14. Encouraged me to ask questions to increase my understanding
15. Stimulated me to explore my strengths and weaknesses
16. Stimulated me to consider how I might improve my strengths and weaknesses
17. Encouraged me to formulate learning goals.
18. Encouraged me to pursue my learning goals.
19. Encouraged me to learn new things.
20. Created a safe learning environment.
21. Took sufficient time to supervise me.
22. Showed me respect.