Assessment and Evaluation of Cardiology Residency Training in Türkiye: A National Survey

Abstract

Background:

Workplace-based assessment methods are essential in the assessment and evaluation of competency-based cardiology residency training. This study aims to determine the assessment and evaluation methods used in cardiology residency training in Türkiye and to reach the opinions of the institutions on the applicability of the workplace-based assessments.

Methods:

In this descriptive study, a Google Survey was sent to the heads/trainers of residency educational centers and their opinions about the currently used assessment and evaluation methods, applicability of cardiology competency exams, and the workplace-based assessments were asked.

Results:

Responses were received from 65 (76.5%) of 85 training centers. Of the centers, 89.2% reported using resident report cards, 78.5% case-based discussion, 78.5% direct observation of procedural skills, 69.2% multiple-choice questions, 60% traditional oral exams, and less commonly other exam types. About 74% of responders gave a positive opinion on the requirement of being successful in the Turkish Cardiology Competency knowledge exam before specialty. Case-based discussion was the most common workplace-based assessments that the centers think could be applied as suggested by the current literature. A common idea was the adaptation of workplace-based assessments based on international standards and our national norms. The trainers supported a nationwide examination for all training centers to ensure standardization.

Conclusion:

In Türkiye, it was promising to see that the trainers are positive about the applicability of workplace-based assessments, but they commonly thought that the proposed workplace-based assessments should be adapted before nationwide applicability. Medical educators and field experts need to work together on this issue.

Highlights

• In Türkiye, the most commonly used assessment and evaluation methods in cardiology residency training were resident report cards, case-based discussion, and direct observation of procedural skills, followed by multiple-choice questions, written and traditional oral exams.

• It was promising to see that trainers support the use of standardized objective joint examinations.

• About 74% of the trainers think that the residents should pass the Turkish Board of Accreditation in Cardiology knowledge exam before entering the specialty exam.

• Trainers commonly suggested that the proposed workplace-based assessments should be adapted to our local conditions before nationwide applicability.

Introduction

After the introduction of Tomorrow’s Doctors in 1993, medical education began the transition from a time- and process-based system to a competency-based training framework. Competency-based medical education (CBME) is an “outcome-based educational method” that aims to train physicians equipped with competencies to meet the needs of the society and the patients, using proficiency frameworks in planning, implementation, measurement, and evaluation.^1,2 Assessment and evaluation in CBME should be based on programmed formative workplace-based assessment (WBAs) that include multiple methods and assessors embedded within an effective educational system.^2,3 Competency-based medical education recommends frequently repeated formative assessment and evaluation methods to support learning instead of one-time (point in time) high-stake examinations for summative assessment. These assessment methods should include the opinions of different evaluators and feedback.⁴ Decisions regarding competence should be made when a wealth of data is reached due to multiple observations and scoring.

Education Section of the European Society of Cardiology (ESC) defined the curriculum about competency-based cardiology training. The knowledge, skills, and attitudes that should be acquired in cardiology residency training were defined under the title of “ESC Core Curriculum for General Cardiologist” in 2006, 2008, and 2013 and were last updated in 2020 based on Entrustable Professional Activity (EPA).^5,6 Entrustable Professional Activities are defined as units of professional activity or clinical competence that a trainee can be trusted to perform in a given healthcare context once an appropriate competence level has been demonstrated.⁷ In the 2020 update of European Society of Cardiology-Core Curriculum (ESC-CC), the definition, CanMEDS roles, knowledge, skills, attitude goals, assessment methods, and expected level (from 1 to 5) for the trainee are specified separately for each EPA. Diagnostic procedures and interventional skills are also listed.

The 2020 ESC-CC recommended to measure trainee’s knowledge with the “European Examination of Core Cardiology-EECC” exam, which consists of clinical case-based multiple-choice questions (MCQ). Formative WBAs and structured feedback are suggested to evaluate skills and attitudes.⁸ In postgraduate medical education and training programs, various combinations of WBAs are designed to address observation and feedback on practical, technical, communication, and judgment skills.⁹ The ESC-CC and current literature recommends case-based discussions (CBD), mini-clinical assessment (mini-CEX), direct observation of procedural skills (DOPS), and multisource feedback (MSF) (360-degree evaluation) in assessing these skills and attitudes. Patients’ participation in this evaluation, as well as other members of the team, is emphasized. The ESC-CC recommends formative assessments as a part of routine clinical care and that the proficiency decision should be made through a decision-making assessment consisting of their components.

The study for cardiology curriculum in Türkiye was first started in 2010 by the Ministry of Health, and a commission named “Medical Speciality Board-Curriculum Formation and Standard Setting System (TUK-MOS)” was founded. First draft of Turkish Cardiology Training Program was established with the joint effort of TUK-MOS, Cardiology Board of Accreditation, and field expertise’s and published in 2013 under the title of “Cardiology Specialization Education Core Curriculum” Medical Speciality Board-Curriculum Formation and Standard Setting System. This TUK-MOS Cardiology Training Program (TR-CC) was updated in 2016, 2019, and 2021.¹⁰ In the current TR-CC, cardiology training is 5 years, and the clinical competencies and skills aimed to be acquired during the training are presented in lists with the expected level of competence. Regulations on Speciality Education in Medicine and Dentistry defines resident report cards (RRC), written opinion of program director at 6 months intervals, preparation of thesis and speciality exam with theoretical and practical components as compulsory components for evaluation during residency.¹¹ The current TR-CC does not contain detailed information on the assessing and evaluating of the individual competencies.¹⁰ Turkish Board of Accreditation in Cardiology (TBA-C) carries a 2-step competency examination, with knowledge and skills components, compatible with the TR-CC. After passing both exams, candidates get a “personal accreditation certificate for cardiology.” There is no obligation to participate in these exams to become a cardiology specialist. Therefore, the popularity of the exam is lower than expected.¹²

There is an obvious need to use valid and reliable multidimensional evaluation methods in residency training.¹³ Clinical evaluation WBAs, such as mini-CEX, DOPS, CBD, or MSF, will meet these needs. The TR-CC is currently being updated, and the new update will include a more detailed description of assessment and evaluation methods about how procedural skills are achieved. Before this update, first we need to define the current situation in our country and figure out the problems we are expected to face in the implementation of the assessment methods and collate the best useful and valid solutions.

This study aimed to describe the current methods used in the assessment and evaluation of cardiology residency training in Türkiye, to take the opinions of trainers on the applicability of the recommended WBAs, to identify possible obstacles/difficulties in the implementation of these methods, and to figure out possible solutions to these obstacles/difficulties.

Methods

Participants

After approval from Hacettepe University Institutional Ethics Committee (E-35853172-900-00001679624), we conducted a descriptive survey among cardiology specialist training centers in Türkiye. The information of the residency training centers were obtained from secretariat of the Turkish Society of Cardiology (TSC), Accreditation Board. A total of 85 clinics (54 State University Hospitals, 25 Training and Research Hospitals affiliated with Health Science University, and 6 Foundation University Hospitals) providing active cardiology residency training were contacted by the relevant chief of the training center who were informed about the questionnaire via a phone call and verbal informed consents were obtained as a mandatory requirement for participation. Participants were informed about the aim of the study, the voluntary nature of participating in the study, and the data would be analyzed anonymously. The links to the survey (Google Form) with the written informed consents were sent via WhatsApp application or e-mail. Participants were also informed how to contact the researchers in the case of queries, concerns, or when they wanted to withdraw from the study. At least 70% of feedback is targeted. Data were gathered between December 2021 and March 2022.

Survey Instrument

A questionnaire was designed to answer our study questions, which included three main topics:

1. Which assessment and evaluation methods do cardiology specialization training centers currently use in Türkiye?

2. What are the views of the Turkish cardiology trainers regarding the applicability of the ESC-CC based on EPAs?

3. What are the opinions and recommendations of the cardiology trainers on the applicability of the WBAs recommended in CBME literature?

European Society of Cardiology-Core Curriculum and the current literature on measurement and evaluation in residency training were used to create the survey questions. After a pilot study, the link to the survey was sent to the heads of departments or education supervisors of universities and training and research hospitals.

The survey comprised 5 sections, and 18 questions including selection for a list, open-short answer, and open-long answer questions (Table 1). For questions 5, 6, and 7 multiple selections from a list of answers were possible. For questions 8-13, 15, and 17, only one selection from a list of answers was possible. Questions 6, 14, 16, and 18 were optional, all other questions in the survey were mandatory to be answered before completion of the survey. Brief explanations for each of the assessment methods, definitions of formative /summative, as well as links to TR-CC and ESC-CC were included in the survey. The list of these sections and questions is given in Table 1.

Table 1.

Sections of the Survey and Questions in Each Section

Section I	Demographic characteristics of the training center: 1. What is the name of the training center?1 2. How many years has your center been giving cardiology speciality training?1 3. What is the number and status of the trainers in your center?1 4. What is the number and status of the trainees in your center?1
Section II	Assessment and evaluation methods used by the training center: 5. What methods do you use to evaluate whether the trainees achieve the goals defined in your cardiology training programs during their training?2 (selection from a list including essay, MCQ, traditional oral examination, structured oral examination, progress test, OSCE, Mini-CEX, DOPS, MSF, CBD, EBD, RRC, portfolio), Please also define your purpose of the use of this method as summative, formative or both2 6. Please write if you use any method other than the list above2 7. What components does your specialization exam include other than thesis defense?2 (selection from a list including essay, MCQ, oral examination, case presentation / case discussion, others) 8. What is your opinion on the requirement to pass TBA-C competency examination ‘knowledge step’) before specialization exam?3 (selection from 5-Scale Likert) 9. What is your opinion on the requirement to pass TBA-C competency examination ‘Skills step’ before specialization exam?3 (selection from 5-Scale Likert)
Section III	TUK MOS Cardiology Training Program (TR-CC): 10. Please evaluate to what extent the cardiology core training program implemented in your institution is compatible with TR-CC? 3 (selection from scale ranging 0-100%) 11. What is your opinion on the applicability /application of the TR-CC in cardiology specialization training?3 (selection from a 5-Likert scale) 12. How much do you think the cardiology residents trained in your institution meet the competencies specified in the TR-CC when they become specialist?3 (selection from a scale ranging 1-10 for each competency) 13. Do you think that TR-CC needs to be updated?3 (selection from yes/no) 14. If yes, what are your suggestions for the update?4
Section IV	ESC Cardiology Core Curriculum Training Program (ESC-CC): 15. What is your opinion on the applicability/implementation of the ESC-CC defined in 2020?3 (selection from 3 scale Likert) 16. What are your suggestions for the implementation of the ESC-CC in cardiology residency training?4
Section V	WBAs in Cardiology Residency Training 17. What is your opinion on the applicability of the WBAs defined in ESC-CC?3 (selection from 3 scale-Likert for each WBAs including RRC, Portfolio, Mini-CEX, DOPS, MSF, CBD, EBD) 18. In your opinion, indicate the two or three most important obstacles to the application of WBAs recommended in the CBME. What are your suggestions for removing these obstacles? 4

¹Short-answer question

²Multiple selection is possible from a list of options

³Only one selection is possible from a list of options

⁴Long-answer question

CBD, case-based discussion; CBME, competency-based medical education; DOPS, direct observation of procedural skills; EBD, entrustment-based discussion; ESC-CC, ESC Core Cardiology Curriculum; MCQ, multiple choice question test; Mini-CEX, Mini Clinical Examination; MSF, multisource feedback; OSCE, Objective Structured Clinical Examination; RRC, resident report card; TBA-C, Turkish Board of Accreditation in Cardiology; TR-CC, TUK MOS Cardiology Training Program; WBA, Workplace-Based Assessments.

Statistical Analysis

The statistics of the study were carried out by using Statistical Package for Social Sciences SPSS v25 (SPSS Inc., Chicago, Ill, USA) statistical program. Values were expressed as mean, standard deviation, percentage values, or median/mode as appropriate.

Chi-square or Fisher’s exact test was used to compare categorical values. The survey results were compared between State University Hospitals versus Training and Research Hospitals affiliated with Health Science University and between accredited and nonaccredited centers. Independent samples t-test was used for the comparison of 2 groups for normally distributed continuous variables. A P value was accepted as significant at < .05 level.

Results

Section I: Demographic Characteristics

Sixty-five out of 85 training centers filled the survey. A map of training centers that responded to the survey is illustrated in Figure 1. Percentages of respondence to the survey were 75.9% for State University Hospitals, 84% for Training and Research Hospitals, and 50% for Foundation University Hospitals. Table 2 summarizes the characteristics of training centers included in the study. The number of trainers and residents for each type of training center was also presented. The mean number of total trainers was 10.02 ± 8.54 (min-max: 3-53), and the total number for 65 centers was 651. The total number of residents was 645; 364 (56.4%) of them were in the first and 281 (43.5%) of them were in the second half of their training period during the survey.

Figure 1.

A map of training centers that responded to the survey.

Table 2.

Characteristics of Training Centers Participating in the Survey (n = 65)

Distribution of centers (n, %):
State University Hospitals	41 (63.1)
Training and Research Hospitals	21 (32.3)
Foundation University Hospitals	3 (4.6)
Accreditation status of centers (n, %):
Current or past accredited	23 (35.4)
Not accredited	42 (64.6)
Duration of specialization training in years (mean ± SD, range):	20.09 ± 12.9 (1-60)
Total number of trainers (mean ± SD, range):	10.02 ± 8.54 (3-53)
State University Hospitals	8.6 ± 3.4 (3-9)
Training and Research Hospitals	13.1 ± 13.9 (3-53)
Foundation University Hospitals	7.0 ± 3.6 (3-10)
Total number of residents (mean ± SD, range):	9.92 ± 7.55 (0-44)
State University Hospitals	9.02 ± 4.64 (2-25)
Training and Research Hospitals	12.86 ± 10.84 (2-44)
Foundation University Hospitals	1.67 ± 2.89 (0-5)

Section 2: Assessment and Evaluation Methods Used by the Training Center

Table 3 summarizes the assessment and evaluation methods reported by the training centers that they were using during cardiology residency training. The most used method (89.2% of the centers) was the RRC. Objective Structured Clinical Examination (OSCE), MSF, and Mini-CEX were the least frequently reported methods by the training centers. The purpose of use of all exam types as formative, summative, or both are detailed in Table 3.

Table 3.

The used Assessment and Evaluation Methods in Cardiology Residency Training

Assessment and Evaluation Methods	Usage	Only Formative	Only Summative	Both Formative and Summative
Essay	37 (56.9%)	28 (43.1%)	3 (4.6%)	6(9.2%)
MCQ	45 (69.2%)	34(52.3%)	5 (7.7%)	6 (9.2%)
Traditional oral exam	39 (60%)	21 (32.3%)	13 (20%)	5 (7.7%)
Structured oral exam	19(29.2%)	10(15.4%)	6(9.2%)	3(4.6%)
Developmental exam	22 (33.8%)	16 (24.6%)	4 (6.2%)	2 (3.1%)
OSCE	12 (18.5%)	7 (10.8%)	4 (6.2%)	1 (1.5%)
Mini-CEX	16 (24.6%)	9 (13.8%)	4 (6.2%)	3 (4.6%)
DOPS	51(78.5%)	33 (50.8%)	10 (15.4%)	8 (12.3%)
MSF	13(20.0%)	10 (15.4%)	2(3.1%)	1(1.5%)
CBD	51 (78.5%)	38 (58.5%)	10 (15.4%)	3 (4.6%)
EBD	26 (40%)	14 (21.5%)	7 (10.8%)	5 (7.7%)
RRC	58 (89.2%)	33 (50.8%)	13 (20%)	12 (18.5%)
Portfolio	20(30.8%)	10 (15.4%)	8 (12.3%)	2 (3.1%)

% of centers among responders (n = 65) using that specific exam type

CBD, case-based discussion; DOPS, direct observation of procedural skills; EBD, entrustment-based discussion; MCQ, multiple choice question test; Mini-CEX, Mini Clinical Examination; MSF, multisource feedback; OSCE, Objective Structured Clinical Examination; RRC, resident report card.

The distribution of exam types used during cardiology residency training according to the type and accreditation status of the institutions was summarized in Tables 4 and 5. The exam types did not differ between the type of institutions (all P values > .05) (Table 4). The exam types were similar between accredited and non-accredited institutions (all P values > .05), except entrustment-based discussion (EBD), which was reported to be more commonly used in nonaccredited institutions (P = .001) (Table 5).

Table 4.

Distribution of Exam Types Used During Cardiology Residency According to the Institutions Status

Exam type	State University Hospitals (n = 41)	Training and Research Hospitals (n = 21)	P value
Essay	25 (60.9%)	10 (47.6%)	.315
MCQ	25 (60.9%)	18 (85.7%)	.046
Traditional oral exam	26 (63.4%)	12 (57.1%)	.631
Structured oral exam	14 (34.1%)	4 (19.0%)	.215
Developmental exam	15 (36.6%)	6 (28.6%)	.528
OSCE	9 (21.9%)	2 (9.5%)	.225
Mini-CEX	10 (24.4%)	5 (23.8%)	.960
DOPS	34 (82.9%)	16 (76.2%)	.525
MSF	10 (24.3%)	2 (9.5%)	.161
CBD	33 (80.5%)	17(77.3%)	.965
EBD	17 (41.4%)	8 (38.1%)	.798
RRC	38 (92.7)	18 (85.7%)	.380
Portfolio	12 (29.3%)	7 (33.3%)	.742

The 3 Foundation Universities were excluded from statistical analysis. P values are for comparison between State University Hospitals and Training and Research Hospitals.

Column percentages indicate the percentage of the exam type used within the related institution.

CBD, case-based discussion; EBD, entrustment-based discussion; DOPS, direct observation of procedural skills; MCQ, multiple choice question test; Mini-CEX, Mini Clinical Examination; MSF, multisource feedback; OSCE, Objective Structured Clinical Examination; RRC, Resident report card.

Table 5.

Distribution of Exam Types Used During Cardiology Residency According to Accreditation Status of the Institution

Exam type	Acr + (n = 23)	Acr – (n = 42)	P value
Essay	16 (69.5%)	21 (50.0%)	.128
MCQ	13 (56.5%)	32 (76.2%)	.100
Traditional oral exam	15 (65.2%)	24 (57.1%)	.525
Structured oral exam	6 (26%)	13 (30.1%)	.680
Developmental exam	8 (34.8%)	14 (33.3%)	.906
OSCE	5 (21.8%)	7 (16.7%)	.614
Mini-CEX	3 (13.0%)	13 (30.1%)	.190
DOPS	19 (82.6%)	33 (78.5%)	.697
MSF	4 (17.4%)	9 (21.4%)	.697
CBD	16 (69.6%)	35 (83.3%)	.197
EBD	3 (13.0%)	23 (54.8%)	.001
RRC	21 (91.3%)	37 (88.1%)	.690
Portfolio	8 (34.8%)	12 (28.6%)	.604

Column percentages, indicating, the percentage of exam type used in accredited and nonaccredited institution.

Acr+, accredited center; Acr-, nonaccredited center; CBD, case-based discussion; DOPS, direct observation of procedural skills; EBD, entrustment-based discussion; MCQ, multiple choice question test; Mini-CEX, Mini Clinical Examination; MSF, multisource feedback; OSCE, Objective Structured Clinical Examination; RRC, resident report card.

The summative exam after 5-year cardiology residency training period, namely the specialty exam in Türkiye, commonly consisted of an essay, MCQ, oral examination, and CBD. Among the 65 centers who responded to the survey, essay was a part of a specialty exam in 15 centers (23.1%), MCQ in 7 centers (10.8%), traditional oral examination in 63 centers (96.9%), and CBD in 50 centers (76.9%). Only one center reported that they also make an exam from updated guidelines. The comparison of the components of specialty examination did not differ between the types of institutions except essay, which was reported to be more commonly used in State University Hospitals (Figure 2). The comparison of the components of speciality examination did not differ based on the accreditation status of the institution (all P values > .05).

Figure 2.

The distribution of assessment and evaluation methods used in cardiology speciality exam according to the institution type. The three Foundation Universities were excluded from graph. MCQ, Multiple choice question test; CBD, Case-based discussion

The opinion of trainers about the requirement of being successful in the TBA-C knowledge (cognitive) and skill board exams before speciality were investigated in the survey. For the knowledge exam, 18 centers (27.7%) reported that they “completely agree” with the opinion of requirement of being successful in the TBA-C knowledge exam before speciality (Figure 3). Thirty centers (46.2%) marked the choice of “agree,” 9 centers (13.8%) reported to be “neutral,” and only 8 centers (12.3%) marked the choice of “disagree.” When the “completely agree” and “agree” options are combined as a total 48 (73.8%) of the responded training centers reported positive opinion to the knowledge exam. For the skill exam, 14 centers (21.5%) reported that they “completely agree” with the opinion of requirement of being successful in the TBA-C skill exam before speciality. Twenty-four centers (36.9%) marked the choice of “agree,” 17 centers (26.2%) reported to be “neutral,” and only 10 centers (15.4%) marked the choice of “disagree.” When the “completely agree” and “agree” options are combined as a total 38 (58.5%) of the responded training centers reported positive opinion to the skill exam. The opinion of centers about the requirement of being successful in the TBA-C knowledge and skill board exams before speciality did not differ with the type or accreditation status of the institutions (all P values >.05).

Figure 3.

The opinion of trainers about the requirement of being successful in the TBA-C board exams. TBA-C, Turkish Board of Accreditation in Cardiology.

Section 3: TUK MOS Cardiology Training Program

The centers were asked to select from a scale (extending from 0% to 100%) to what extent the cardiology training program implemented in their institution was compatible with TR-CC. The mean score of the 65 centers was reported to be 74.9 ± 14.3% (40-100 range). The mean score was similar between State University Hospitals (76.8 ± 11.9%) and Training and Research Hospitals (72.4 ± 17.3%) (P > .05).

The centers expressed their opinion about the applicability/application of the current TR-CC in specialization training by selecting from a 5-point Likert scale. Four centers (6.2%) reported that TR-CC program could be applied as it was, 51 centers (78.5%) reported that it could be applied to a great extent. Six centers (9.2%) reported that it could be partially applied. One center reported that it cannot be applied and a complete revision was needed.

The centers’ opinion about to what extent the cardiology residents trained in their institution met the competencies specified in the TR-CC when they become specialist were also asked. They selected from a scale ranging from 1 to 10 for the competencies included in the CanMEDS competency framework. As indicated in Table 6, the mods and medians of 65 centers who responded to the survey were similar for all competencies.

Table 6.

The Extent the Cardiology Residents Meet the Competencies Specified in the TR-CC when they Become Specialist

	Mod	Median	Range
Leader	8	7	2-10
Collaborator	8	8	2-10
Health advocate	8	8	2-10
Communicator	8	7	2-10
Professional	8	8	2-10
Scholar	8	8	2-10
Medical Expert – Basic Clinical	8	8	2-10
Medical Expert – Interventional	8	8	2-10

TR-CC, TUK-MOS Cardiology Core Training Program.

For the question about the requirement of an update for TR-CC, 33 centers (50.8%) reported that an update was needed. The suggestions of the training centers from open-ended questions about the update of TR-CC included the following 5 main opinions:

• The minimum required procedural goals needed to be revised. To reach these goals, national or international rotations could be planned.

• Basic cardiac magnetic resonance imaging (MRI) and cardiac computerized tomography (CT) education could be included.

• Arrhythmia education including implanted device control and follow-up should be more structured and standardized among educational centers.

• Subspeciality in certain areas of cardiology including invasive cardiology, arrhythmias and electrophysiology, cardiac imaging, and heart failure could be considered. Expert centers could be defined by TSC for these fields.

• A structured educational program for communication skills and evidence-based medicine could be planned.

Section 4: European Society of Cardiology-Cardiology Core Curriculum Training Program

The centers’ opinion on the applicability/implementation of the EPA-based ESC-CC defined by ESC in 2020 was asked in the survey. Forty-five centers (69.2%) responded that it could be applied after certain modification; 1 center (1.5%) responded that it could not be applied at all. None of the centers answered the choice of “it can be applied as it is,” 19 centers (29.2%) declared that they had no idea.

The answers of the training centers to the open-ended question of their suggestions for the implementation of the EPA-based ESC-CC were gathered in the following main opinions:

• To what extent the current TR-CC was applied should be documented first, then the differences between both curriculum and their applicability should be discussed in a broad-based meeting. Awareness should be raised in both educators and trainees.

• The EPA-based ESC-CC needed certain modifications and adaptations before application in Türkiye based on our local conditions. The educators and health authorities should be convinced. Some obligations/rules may be put forward and regularly followed by the Ministry of Health.

• The trainers should concentrate more on educational activities and should not be under the pressure of time constraints or routine daily practice. They should be paid for educational activities. The trainees also should focus on their own education, and routine care of the patients should not be their first responsibility.

Section 5: Workplace-Based Assessments in Cardiology Residency Training

The centers’ opinion on the applicability of the WBAs defined in ESC-CC is shown in Figure 4. The centers were asked to select from 3 options “can be applied as it is,” “can be applied after modification,” or “cannot be applied at all.” Case-based discussion was the most common WBAs that the centers think could be applied as suggested (63.1%). About 35.4% of the centers reported that CBD method could be applied after modification. The percentages of DOPS, RCC, Mini-CEX, EBD, Portfolio, and MSF were 52.3%, 43.1%, 27.6%, 26.2%, 16.9%, and 12.3%, respectively, for the “can be applied as it is” choice. Of the centers, 78.5% reported that MSF could be applied after modification. This percentage for the “can be applied after modification” choice was followed by portfolio (76.9%), EBD (70.8%), mini-CEX (69.2%), RCC (56.9%), DOPS (47.7%), and lastly CBD (35.4%). The centers reporting that the listed WBAs “cannot be applied at all” was minority (Figure 4).

Figure 4.

The opinion of cardiology trainers about the applicability of WBAs. WBAs, workplace-based assessments. WBAs, Work-based assessment methods; RRC, Resident report card; Mini-CEX, Mini Clinical Examination; DOPS, Direct Observation of Procedural Skills; MSF, Multisource feedback; CBD, Case-based discussion; EBD, Entrustment based discussion.

The opinions of the training centers about the most important obstacles to the application of WBAs and their suggestions for removing these obstacles were asked with open-ended questions. The main topics reported by the centers were the following:

• Working conditions and working load of the trainers were the main reported problems. Training centers also reported the lack of trainers specifically dealing with educational activities and the lack of experience and motivation about the use of WBAs by the trainers.

• Performance system about the income of physicians and the minor effect of educational activities on this income was another main concern both for Training and Research Hospitals and University Hospitals.

• The centers suggested that the speciality examination should be revised. Success on a general knowledge examination, like cardiology board examination, before speciality examination was suggested by the centers. Centers suggested an obligation for abstract presentation or paper publication before thesis defense examination related to the thesis subject.

• Some centers stated that a new format for WBAs should be developed based on international standards and our national norms. The trainers should be convinced, and the implementation of this format should be supported and motivated.

Discussion

This study is the first to evaluate the current methods used in the assessment and evaluation of cardiology residency training in Türkiye and to get the opinions of trainers about applicability WBAs, as well as their suggestions for implementation. Main results of this nationwide descriptive study indicated that the commonly used assessment and evaluation methods in cardiology residency training were RRC, CBD, and DOPS, followed by MCQ, written and traditional oral exams. It was promising to see that trainers support the use of standardized objective joint examinations during and at the end of specialization training, but trainers commonly suggested that the proposed WBAs should be adapted before nationwide implementation.

This study was realized by an online survey sent to the head of all cardiology residency training centers after informing and getting their consent. In our protocol, we aimed to reach at least 70% of the centers, and in the end, we got the opinions of 65 centers (76.5%), which is more than our expectations. The training centers included State University Hospitals, Training and Research Hospitals affiliated to Health Science University, and Foundation Universities from different regions of the country. Among the centers, the number of trainers and residents showed a huge diversity. The experience of the centers on cardiology residency training was also very different with a duration of specialization training ranging from 1 year to 60 years. Therefore, we think that our study sample is a good presenter of Türkiye.

Competency-based cardiology training focuses on graduate competency frameworks reflecting effective professional practice in the specific working environment. Frequently repeated formative assessment and evaluation methods are very important to support learning instead of summative assessment.² In our survey, we checked the frequency of the use of assessment and evaluation methods used by the training centers. Our results indicated that during the training period, the most commonly used method was RRC, being used by about 89% of the centers. Although this number was quite high, since it was an obligation to be used by during residency training we expected it to be 100%.¹¹ Whether or not the desired competency has been achieved is expected to be checked on the RRC. Completion of the RRC with predefined competencies and procedures is an obligation before speciality examination. However, in practice, RRC serve as a list of skills to be done (what, how much?) and do not contain feedback on how competent the skills were done. Another important problem with the RRC is that it is usually not filled daily and regularly. Therefore, the use of RRC should be followed more carefully by the trainee and checked more commonly and precisely by trainers (or mentors). Electronic versions of RRCs have been developed to overcome some of the problems, and it was reported to be useful for both assessment and feedback purposes.¹⁴ In our survey, the use of RRC in frequency was followed by DOPS and CBD, which were reported to be used by 78.5% by the centers including both formative and/or summative purposes. In the survey, although these measurement methods were briefly defined in questionnaire for trainers not very familiar with it, we did not specifically ask whether or not the evaluator fill a structured form for each assessment. Therefore, answers to these questions might have included the informal use in daily clinical works of these two methods.

The least frequently used methods were OSCE, MSF, and Mini-CEX reported to be used by the training centers by 18.5%, 20%, and 24.6% consecutively. Objective Structured Clinical Examination evaluates the third step of Miller’s pyramid, namely “shows how to do” step.¹⁵ The rare use of OSCE could be related to the number of residents in training centers since this method is time consuming and not cost effective for small number of participants. Joining of training centers for OSCE exams might increase its use in residency training and evaluation. MSF and Mini-CEX aim to assess trainees at the top of Miller’s pyramid and are very suitable during residency training.¹⁶ These 2 methods promote the assessment of clinical skills along with professional attitudes and higher cognitive skills that are essential in high-quality patient care. In mini-CEX, a single faculty member observes and evaluates a resident while he/she conducts a thorough history and physical examination and after asking the resident for the differential diagnosis and treatment plan gives direct feedback. The resident can be evaluated on several occasions and by various faculty members. A recent study evaluating the impact of mini-CEX on learning in a cardiology residency program indicated that residents found mini-CEX to be a useful assessment tool with a favorable impact on a constructive approach to study and learning.¹⁷ Therefore, their use in cardiology residency training should be encouraged. The reason behind the low incidence of their use might be related to the fact of the trainers’ unfamiliarity with these methods. Similarly, the “ESC cardiovascular education roadmap” in 2019 emphasized that measurement methods such as MSF were also rarely used throughout Europe.¹⁸ Obviously, there will be difficulties to implement these tools in residency training at the beginning. Adapting to WBAs of CBME mandates a change in process and approach, ultimately a change in institutional culture.¹⁹ Supports from the regulatory authorities, help of local leaders, faculty education, and development of electronic platforms are very critical steps for this change.¹⁹ Initiating these methods during undergraduate medical education in clinical clerkships and internships periods will contribute to the development of WBAs culture.

Formative WBAs are very important for post-graduate residency training. It is directed at steering and fostering learning by providing feedback to the learner. Therefore, facilitating its implementation in postgraduate training is crucial, but engaging stakeholders in formative assessment in daily practice is quite complex.¹⁶ The utility of formative assessment, defined as learning as a result of assessment process, is mainly dependent on how the instrument is employed in practice.¹⁶ In our survey, in addition to the use of WBAs such as DOPS, Mini-CEX, MSF, or CBD, we also asked the purpose of these assessment tools as formative or summative. The survey results indicated that the most commonly used formative assessment tools were reported to be RRC, CBD, and DOPS. Resident report cards was reported to be used by 50.8% of the center only for formative and 18.5% of the centers for both formative and summative purposes, as a total 68.5% of centers use RCC for formative evaluation. This number, although the highest among the training centers, is still low, since it is obligatory during residency training. The second most commonly used formative assessment methods were CBD and DOPS (both 63.1%). Considering the answers to open-ended questions, this number is most probably belonging to the informal use of DOPS and CBD, without filling a structured form.

The results indicated that in Türkiye the main components of speciality examination are oral examination and CBD in addition to thesis defense. Few centers use essays or MCQ, which mainly focus on evaluation of cognitive domain. Answers from the open-ended questions suggested that speciality exam is unfortunately far from being objective; the result of exam is clear before taking the exam. Trainers think that they need a more objective and externally directed assessment before speciality. Cardiology Accreditation Board exam may be a solution for that. Turkish Board of Accreditation in Cardiology is conducting a 2-step competency examination since 2005, with knowledge and skills components, compatible with TR-CC. The knowledge exam is open for cardiology residents in the last 2 years of their training or for cardiology specialist. However, the skill component is open only for cardiology specialists. After passing both exams, candidates get “personal accreditation certificate for cardiology,” which is valid for 5 years. However, it is not compulsory to become a cardiology specialist; therefore, the rate of candidates taking these exams and their success rate are lower than expected.¹² Nearly 74% of the trainers think that the residents should pass the TBA-C knowledge exam before entering the speciality exam. This opinion is in parallel with the TBA-C Board report in 2019, suggesting an obligation for 4th and 5th year cardiology resident to take and succeed the board knowledge exam before speciality.¹² This article also draws attention to the reduction of the popularity of Board Exams after the new criteria of becoming “associate professor” and offers that “Board Certification” should be obligatory before becoming associate professor. The positive opinions of educators of cardiology training centers about Cardiology Board exams are very promising for the future.

The last update of TR-CC was in 2021. In the section of survey related to TR-CC, the centers reported that their program is about 75% compatible with TR-CC. This number needs to increase to the aim of 100%. Only 4 centers (6.2%) reported that current TR-CC can be applied as it is, and the great majority had the idea that it can be applied to some extent but not totally. About half of the centers thought that TR-CC should be updated. The update suggestions of the trainers included addition of some newly developed areas such as cardiac MR or CT to the curriculum, more structured educational program for some fields such as arrthymias, communication skills, or evidence-based cardiology, and addition of some national or international rotations to reach procedural goals. An important suggestion from the training centers is about the development of speciality areas in certain fields of cardiology, such as invasive cardiology, cardiac imaging, arrhythmias, or heart failure. In many countries in Europe and US, these fields are subspeciality areas based on a structured educational program. In Türkiye, by law, a cardiology specialist is expected to perform and had the right to perform every interventional procedure and treat every kind of complicated patient, which is not either possible or fair. Therefore, the suggestions of trainers about the development of subspeciality areas and certification programs should be planned.

The educators’ opinion about to what extent the cardiology residents trained in their institution meet the competencies specified in the TR-CC when they become specialist were quite optimistic. They ranked the residents’ competency as 8 (mod) out of 10. However, the range of ranking from 2 to 10 draws attention to the heterogeneity between training centers. In a previous study, one-third of the residents reported that they had sufficient training in invasive and clinical competencies,²¹ reflecting that the educators and trainees think differently about the achievement of required competencies. We investigated in a previous study, the residents’ self-evaluation levels of clinical competencies on all competency areas of TR-CC.²² Our results showed that the self-reported level of competency at certain fields was lower than expected and differed between universities and training and research hospitals. In the future update plan of TR-CC, special attention should be given to the areas with lower self-reported competency such as congenital heart disease, peripheral artery disease, or sudden cardiac death. Furthermore, a homogeneity in the achievement of competencies should be provided and made certain with assessment and evaluation methods among different type of training centers throughout the country.

The practice of cardiology has changed substantially for the past 20 years. In order to reflect modern cardiology, ESC-CC was updated in 2020 with a joint effort between the Education Section of ESC and the European Union of Medical Specialists. Besides the updated definition of the core areas needed for the practice of Cardiology, one of the main changes in this 2020 curriculum is the implementation of EPAs to describe clinical competencies. In this newly updated EPA-based curriculum, when a trainee can be trusted to perform a professional activity at the expected level of independence, the EPA is completed. This concept has been increasingly used by medical educators throughout the world, and the name arises from the fact that trust is a central aspect of learning. During training, trainees progressively acquire knowledge, skills, and attitudes while, at the same time, trainers develop increasing trust in their trainees.⁷ This process is often subconscious, but EPAs make it conscious and formalize it. In our survey, the reported high use of EPA-related WBAs such as CBD or DOPS might be related to this subconscious use of these methods. Our educators reported mainly positive opinions about an EPA-based training program, but common idea suggested that an adaptation is needed before implementation of such a program. About one-third of the chief of the training centers reported that they did not have any idea about such a program. Therefore, as suggested, awareness should be raised in both educators and trainees, and these training programs should be discussed in broad-based meetings. Convincement and support of health authorities and some rules/regulations put forward by the Ministry of Health services may help implementation of national residency training programs.

European Society of Cardiology-Core Curriculum integrated many WBAs to formalize the achievement of EPA into daily practice such as mini-CEX, DOPS, MSF, or EBD. These should be done by multiple evaluator faculty members for different skills in different places and conditions during the residency education. It is also very important that the faculty members are ready to apply these assessment methods. In a previous report, one of the most stubborn problems with WBAs was reported as the absence of standardized observation of trainees and the lack of feedback based on observations.²³ Faculty members rarely use a criterion-based standard to benchmark residents against best practices, and even criterion-based competence judgments may be influenced by relative comparisons of trainees. It was concluded in the report that educators and supervisors need to be trained in how to apply WBAs. The factors important for acceptability, effectiveness, and utility of WBAs in postgraduate medical education were evaluated in a hermeneutic review.²⁴ The authors presented 12 lessons to improve the WBAs related to the user and tool, but most fundamentally user–tool–context interaction, particularly trainee–assessor relationship. They suggested that assessor’s assessment literacy should be combined with cultural and administrative factors in the organization and broader medical systems. Stakeholders’ engagement in the development and review of WBAs was reported to be critical.²⁴ In our survey, our educators reported that they use commonly DOPS and CBD, rarely Mini-CEX or MSF. Entrustment-based discussion was reported to be used in between. Our educators generally reported positive opinions about the use of these WBAs in cardiology residency training. However, to implement these methods, they reported that an adaptation is needed based on our local conditions. Before such an adaptation, first we should fight with the main obstacles against their implication. The main reported obstacles are working load of both trainers and trainees, performance system about the income of physicians, and lack of motivation and education about WBAs. Trainers suggested that a new format for WBAs based on international standards and our national norm should be developed. Documentation of assessments with electronic aids such as apps is particularly important. On one hand, this will help the trainees to document their activities in an electronic logbook. On the other hand, trainers would not get bothered with additional administrative work due to the documentation of assessments. For the development of such a format, all of the stakeholders of the system including TBA-C, board exam coordinators, educators, and trainees should work together with the support of health authorities.

Study Limitations

This study is subject to some limitations. The numbers of residents and trainers were representing the study period; there may be changes in these numbers because of dynamic changes between institutions. The results of the study are based on the answers of chief/educator of the training centers. It is a well-known limitation of survey studies that, sometimes it may be difficult to convince the people to get reliable answers. Even the chief of the training center may not be very familiar with the curriculums or assessment methods, or may understand the question differently. Some WBAs reported to be used by the center might be used informally without any documentation, or the reported trust process of our educators may be mainly subconscious without filling a defined form. The study should be read by considering these limitations. To overcome these limitations after getting the picture of current condition with this paper, in the second part of our study, we planned a qualitative study to analyze the research questions in more detail. This study is planned on volunteer-focused groups including residents at different levels of their training period as well as educators with different levels of experience with semi-structured interview. This interview will also focus on feedback in formative assessment. The combined results of these quantitative and qualitative analyses will define our research questions in more detail.

Conclusion

In conclusion, the most commonly used assessment and evaluation methods in cardiology residency training are seen as RRS, CBD, and DOPS, followed by MCQ, written and traditional oral exams. On the other hand, the specialization exam often consists of oral and CBD components. Trainers support the use of standardized objective joint examinations throughout the country during and at the end of specialization training. It is promising to see that the trainers are positive about the applicability of WBAs, but they commonly think that the proposed WBAs should be adapted before nationwide applicability. Collaborative effort of medical educators and field experts to work together on this issue is essential.