Abstract
Purpose
This study examines the implementation of the competency-based curriculum (CBC), which has become the primary educational model in Korean medical schools. It also presents valid evaluation criteria developed through expert consensus to support ongoing improvements in curriculum quality.
Methods
This study examines the implementation of the CBC, which has become the primary educational model in Korean medical schools. It also presents valid evaluation criteria developed through expert consensus to support ongoing improvements in curriculum quality.
Results
Three rounds of Delphi surveys were completed to validate the valid evaluation criteria. Criteria with lower content validity ratios were revised and supplemented based on expert feedback.
Ultimately, the process resulted in the identification of five evaluation areas, 16 evaluation items, and 51 evaluation indicators. Among these, the indicators focusing on learner achievement were found to be the most significant.
Conclusion
This study developed an evaluation model and valid evaluation criteria specifically designed for the curricula of Korean medical schools. These were created based on expert consensus, which adds to the strength and relevance of the proposed framework. The establishment of valid evaluation criteria is expected to enhance evaluation practices, promote educational quality, and support the continuous improvement of medical education.
Keywords: Curriculum evaluation, Evaluation criteria, Competency-based curriculum, Delphi, Analytic hierarchy process, Delphi hierarchy process
Introduction
To prepare medical students with the competencies required by society, improving the quality of education is essential. As both societal demands and student characteristics evolve, educational needs continue to change, increasing the demand for curriculum enhancement [1]. In response to these shifts, many medical schools in Korea have implemented competency-based curricula, which now serve as a core component of medical education [2-4]. These curricula specify the competencies students must demonstrate in practice and inform the development of educational objectives, content, and assessment strategies [5-8].
Evaluating curricula to determine their value and areas for improvement is crucial for enhancing educational quality [9]. However, existing studies have primarily focused on partial assessments without a systematic framework, lacking research that reflects competency-based curricula [9-12]. This makes it necessary to conduct thorough evaluations that provide clear information on improvement needs.
To achieve reliable and actionable insights for curriculum enhancement, valid evaluative criteria must be established. The importance of these criteria cannot be overstated, as they significantly influence the success of evaluations [13-15]. While there have been some studies on evaluative criteria in domestic medical education, such as those by Hwang [16] in 2015 and the National, Teacher Training Center for Health Personnel, Seoul National University [17] in 2019, they are limited in scope and applicability to competency-based curricula.
This study aims to validate the evaluative criteria derived from the research by Lee et al. [18] in 2023. The aforementioned study proposed a systematic curriculum evaluation model by integrating the context, input, process and product (CIPP) and Kirkpatrick evaluation models through an extensive literature review, establishing relevant evaluation criteria. A preliminary draft of evaluation criteria was first developed to assess competency-based medical education. The initial version was informed by interviews with domestic experts. As a result, the integration of the two evaluation models led to a structure encompassing context, input, process, and product, with the product domain further subdivided into reaction, learning, behavior, and results (Table 1).
Table 1.
Evaluation Criteria Derived from a Literature Review and Expert Interviews
| CIPP model | Kirkpatrick model | Evaluation areas | Evaluation items | Evaluation indicators |
|---|---|---|---|---|
| Context | - | 1. Context | 1.1. Competences and educational goals | 1.1.1. Clarity of educational competences and educational goals |
| 1.1.2. Appropriateness of analyzing stakeholders’ demands in educational competences and education | ||||
| 1.1.3. Sufficiency of reflecting essential medical education competences to educational competences an education goal | ||||
| 1.1.4. Clarity of recognition of educational competences and educational goals | ||||
| 1.2. Curriculum design principles | 1.2.1. Sufficiency of reflecting curriculum design principles | |||
| 1.3. Teachers and learners | 1.3.1. Integrity of learner analysis | |||
| 1.3.2. Integrity of teacher analysis | ||||
| 1.4. Organization | 1.4.1. Sufficiency of executive leadership commitment and willingness | |||
| 1.4.2 Integrity of finance and regulations for curriculum implementation | ||||
| 1.4.3. Sufficiency of member consensus on curriculum introduction | ||||
| 1.4.4. Appropriateness of procedures for identifying and addressing barriers | ||||
| Input | - | 2. Input | 2.1. Educational contents and method | 2.1.1. Appropriateness of educational contents |
| 2.1.2. Appropriateness of education methods | ||||
| 2.2. Educational strategy | 2.2.1. Appropriateness of educational programming in consideration of clinical performance capabilities | |||
| 2.2.2. Appropriateness of clinical educational programming in consideration of learner’s level and education after graduation | ||||
| 2.2.3. Sufficiency of providing education opportunities through meetings with patients | ||||
| 2.2.4. Appropriateness of basic medicine-medical humanities-clinical medicine education connection | ||||
| 2.2.5. Practice reinforcement | ||||
| 2.2.6. Appropriateness of balance coordination in curriculum and subjects | ||||
| 2.2.7. Sufficiency of repeated learning opportunities to promote mastery | ||||
| 2.3. syllabus | 2.3.1. Scrupulosity of syllabus | |||
| 2.3.2. Appropriateness of amount of learning goals | ||||
| 2.4. Learner evaluation | 2.4.1. Clarity of establishing basic principles of learner evaluation | |||
| 2.4.2. Appropriateness of learner evaluation methods | ||||
| 2.4.3. Evaluation focused on performance | ||||
| 2.4.4. Clarity of competence achievement evaluation criteria | ||||
| 2.4.5. Appropriateness of implementing progress test | ||||
| 2.4.6. Appropriateness of implementing criterion-referenced evaluation methods | ||||
| 2.4.7. Appropriateness of providing remediation and reevaluation | ||||
| 2.4.8. Appropriateness of portfolio evaluation | ||||
| 2.4.9. Appropriateness of formative evaluation and summative evaluation | ||||
| 2.5. Human resources | 2.5.1. Clarity of granting authorities and roles to curriculum supervising professor | |||
| 2.5.2. System of organization in charge of medical education | ||||
| 2.5.3. System of organization in charge of evaluation | ||||
| 2.5.4. Appropriateness of support for teaching capability development | ||||
| 2.6. Material resources | 2.6.1. Sufficiency of facilities and equipment | |||
| 2.6.2. Organic connection between medical schools and hospitals for practical training | ||||
| 2.7. Learner support | 2.7.1. Appropriateness of learning support for learners | |||
| Process | - | 3. Process | 3.1. Teaching-learning process | 3.1.1. Sufficiency of opportunities for competence acquisition during clinical practice |
| 3.1.2. Integrity of education plan implementation | ||||
| 3.1.3. Smoothness of conducting education | ||||
| 3.1.4. Appropriateness of feedback | ||||
| 3.2. Achievement evaluation process | 3.2.1. Integrity of evaluation plan implementation | |||
| 3.2.2. Appropriateness of remediation and reevaluation implementation | ||||
| 3.3. Educational monitoring and improvement | 3.3.1. Appropriateness of educational evaluation | |||
| 3.3.2. Sufficiency of collecting and utilizing evaluation results | ||||
| 3.3.3. Sustainability of competence review committee activities | ||||
| 3.3.4. Appropriateness of curriculum supervising professor activities | ||||
| Product | Level 1 reaction | 4. Learning outcome | 4.1. Degree of satisfaction | 4.1.1. Sufficiency of learner’s satisfaction with education |
| 4.1.2. Sufficiency of investigation factors to learner’s satisfaction | ||||
| Level 2 learning | 4.2. Competence achievement | 4.2.1. Sufficiency of competence achievement in learner’s knowledge, skills, and attitude | ||
| 4.2.2. Appropriateness of unintended learning outcome | ||||
| Level 3 behavior | 4.3. Transfer of learning | 4.3.1. Sufficiency of applying basic medical education to clinical medical education | ||
| 4.3.2. Sufficiency of applying previous learning contents to clinical practice | ||||
| 4.3.3. Sufficiency of application to medical sites after graduation | ||||
| 4.3.4. Identification of factors influencing the application of competencies | ||||
| Level 4 results | 4.4. Impact of competence | 4.4.1. Changes in postgraduate training courses | ||
| 4.4.2. Impact on patient care and clinical practice | ||||
| 4.4.3. Impact on society and scientific advancement | ||||
| - | 5. Continuous quality improvement of curriculum | 5.1. Ensuring sustainability and feedback for continuous curriculum improvement | 5.1.1. Sustainability of success factors of curriculum | |
| 5.1.2. Feedback of curriculum evaluation results |
The present study expanded upon this work by adopting and validating the evaluation criteria. A combination of the Delphi method and the analytic hierarchy process (AHP) was used. This integrated approach, known as the Delphi hierarchy process (DHP), involved three rounds of surveys to assign weights to the evaluation criteria.
The primary aim of this research is to develop valid evaluation criteria for evaluating competency-based medical education. Two key questions were addressed: (1) What are the valid evaluation criteria for assessing competency-based medical education? (2) What is the relative importance of each evaluation criterion?
The study also seeks to encourage the use of these evaluation criteria in practical educational settings. Applying the criteria is expected to support the continuous development of competency-based curricula and improve the overall quality of medical education.
Methods
1. Validation of evaluation criteria
In the research by Lee et al. [18], evaluation criteria developed based on literature review and expert interviews were utilized as a draft for validation (Table 1). This study conducted three rounds of the Delphi survey to validate the evaluation criteria, employing a combined method of the Delphi technique and the AHP, referred to as the DHP. According to Khorramshahgol and Moustakis [19], the DHP method was used to validate the evaluation criteria through the first and second Delphi surveys, while the third Delphi survey determined the relative importance, or weights, of these criteria.
To minimize error and ensure reliability, it is recommended to select a Delphi expert panel consisting of 10–15 members [20-22]. As shown in Table 2, a panel of 20 experts in medical education, education, and medicine was selected. The Delphi panel comprised 20 experts, of whom seven held medical degrees (Doctor of Medicine [MD]) with substantial clinical experience. The educational experts (Doctor of Philosophy [PhD]) also possessed a minimum of 4 years of experience working in medical education environments. Notably, all MD participants were board-certified specialists actively engaged in clinical practice, selected to ensure the validity and practical relevance of the responses. All 20 panel members (100%) participated in the first and second rounds of surveys conducted from September 21, 2020, to November 15, 2020, while 17 members (85%) participated in the third round.
Table 2.
Status of the Delphi Panel Group
| Panel group | No. | Degree | Affiliation | Total experience in medical education (yr) | Delphi survey participation |
|||
|---|---|---|---|---|---|---|---|---|
| 1st | 2nd | 3rd | ||||||
| Medical education major | 1 | PhD | OO | University | 11 | O | O | O |
| 2 | MD, PhD | OO | University | 10 | O | O | O | |
| Education major | 3 | PhD | OO | University | 3 | O | O | O |
| 4 | PhD | OO | University | 15 | O | O | - | |
| 5 | PhD (in progress) | OO | University | 15 | O | O | O | |
| 6 | PhD | OO | University | 7 | O | O | O | |
| 7 | PhD | OO | University | 20 | O | O | O | |
| 8 | PhD | OO | University | 5 | O | O | O | |
| 9 | PhD | OO | University | 9 | O | O | O | |
| 10 | PhD | OO | University | 4 | O | O | O | |
| 11 | PhD | OO | University | 20 | O | O | - | |
| 12 | PhD | OO | University | 8 | O | O | O | |
| Medical major | 13 | MD, PhD (in progress) | OO | University | 9 | O | O | O |
| 14 | MD, PhD | OO | University | 8 | O | O | O | |
| 15 | MD, PhD | OO | University | 12 | O | O | O | |
| 16 | MD, PhD | OO | University | 8.5 | O | O | O | |
| 17 | MD, PhD | OO | University | 20 | O | O | O | |
| 18 | MD, PhD | OO | University | 4.5 | O | O | O | |
| Medical and education major | 19 | MD, PhD | OO | University | 13 | O | O | O |
| Other major | 20 | PhD | OO | University | 10 | O | O | - |
PhD: Doctor of Philosophy, MD: Doctor of Medicine.
Two Delphi surveys were implemented for the validation of the evaluation criteria. The expert panel was asked to respond in a 5-point Likert scale (1: very invalid, 2: invalid, 3: moderate, 4: valid, and 5: very valid) about the validity of the evaluation areas, evaluation items and evaluation indicators in the draft evaluation criteria derived in the evaluation criteria drafting stage. The judgment of the validity was based on the review of the basic statistics including the means, standard deviations, and quartiles. In particular, the judgment was made according to the content validity ratio (CVR), which is an essential variable in the validity judgment in Delphi surveys, by deciding the CVR is higher than 0.42 as suggested by Lawshe [23] in 1975 for an expert panel consisting of 20 members.
In addition, since the validity in the Delphi method may be presented by analyzing the levels of opinion convergence and consensus by the expert panel [24-26], a degree of consensus of 0.75 or higher and a degree of opinion convergence of 0.5 or lower were considered as a high validity.
In the Delphi method, the stopping criterion that determines the rounds was the coefficient of variation (CV), which was considered to examine the stability. The CV is the standard deviation divided by the arithmetic mean. A CV of 0.5 or lower was considered as requiring no additional round, 0.5 to 0.8 was considered as a relatively stable, and 0.8 or higher as requiring an additional survey [25,27]. The data from the first and second Delphi survey were analyzed by calculating the frequency, percentage, mean, standard deviation, median, quartile, degree of consensus, degree of convergence and CV by using Excel 2016 software program (Microsoft Corp., Redmond, USA).
In addition, the Delphi panel was asked to freely describe the parts of the individual questions that required correction, addition, or removal. After completing a round, the opinions from the Delphi panel were summarized to modify the questions, especially those that were pointed out by at least two experts. The questions that were pointed out by at least one expert were modified through the consultation with the experts who reviewed the draft evaluation criteria.
2. Relative importance (weights)
The process of determining relative importance involved calculating the weights of the evaluation criteria finalized during the validation phase, using the AHP. To assign these weights, the evaluation criteria confirmed through the Delphi survey were presented to the panel using a specially developed survey tool based on the AHP methodology. Since AHP requires a hierarchical structure for comparisons, the criteria were organized into evaluation areas, items, and indicators from the outset. A paired comparison scale ranging from 1 to 9 and their reciprocals was applied, and only odd-numbered values were used in this study to clarify the differences between the scales more distinctly [22].
To minimize the potential influence of subjective judgment inherent in expert assessments, the consistency ratio (CR) was calculated for each response set. A CR value of ≤0.1 was used as the threshold for acceptable consistency, as recommended by Saaty [22], ensuring logical coherence in the paired comparisons. In the third Delphi survey, local weights, composite weights, and CR values were calculated using I Make It (https://imakeit.kr/info/index.html) and Excel 2016. The I Make It software facilitated data entry for pairwise comparisons and automatically computed CR values for each participant, while Excel 2016 was used to cross-verify the computed weights and CR results. Responses with CR values exceeding the threshold were reviewed, and where necessary, adjustments were made or responses excluded to maintain the reliability and consistency of the weighting process.
Results
1. Validation of evaluation criteria (Delphi): results of first Delphi survey
1) Results of first Delphi surveys on the evaluation areas (Appendix 1)
The evaluation criteria encompassed five areas: 1. Context, 2. Input, 3. Process, 4. Learning outcomes, and 5. Curriculum outcomes. In the first Delphi survey, all areas received a mean score of 4.0 or higher, with standard deviations ranging from 0.5 to 0.74. The median and mode were both 5, indicating very high validity. The overall positive response rate was 95% to 100%, and the CVR ranged from 0.90 to 1.00, exceeding the validity level of 0.42 suggested by Lawshe [23] in 1975. Consequently, all evaluation areas in the draft criteria were deemed valid. Based on expert panel feedback, “1. Context” was modified to “1. Educational environment & context,” and “5. Outcome of curriculum” to “5. Continuous quality improvement.”
2) Results of first Delphi surveys on the evaluation items (Appendix 2)
Most evaluation items (17 out of 18) scored 4 or higher, with the exception of “1.2. Design principles,” which was removed as it did not exceed the CVR threshold of 0.42. The mode was 5 for all but two items, confirming high validity. The item “1.2. Design principles,” which did not meet the validity criteria, was reassigned to “2.1.” Items with a CVR above 0.42 but lower than expected were revised based on expert opinions, including renaming and reclassifying several items for clarity and specificity.
3) Results of first Delphi surveys on the evaluation indicators (Appendix 3)
Indicators with a CVR of 0.42 or lower were excluded to maintain the robustness of the evaluation criteria. For example, indicators such as “1.4.1. Sufficiency of leadership will” and “2.3.6. Appropriateness of absolute evaluation” were removed due to their low CVR values.
Additionally, the terminology of the remaining indicators was refined based on feedback from the expert panel to improve clarity and precision. For example, “3.1.1. Appropriateness of competence acquisition opportunities” was revised to improve clarity.
To avoid redundancy, overlapping indicators were combined, creating a more concise and coherent set of evaluation indicators. As an illustration, “3.3.3. Sustainability of activities by the competence review committee” and “3.3.4. Appropriateness of curriculum supervising professor” were merged into a single indicator.
2. Validation of evaluation criteria (Delphi): results of second Delphi survey
1) Results of second Delphi surveys on the evaluation areas (Appendix 1)
The second Delphi survey aimed to verify the validity of the modified evaluation areas, items, and indicators from the first survey. All evaluation areas achieved a mean validity score of 4.0 or higher, with median and mode scores of 5, confirming their high validity. The CVR for all areas was between 0.90 and 1.00, surpassing the 0.42 threshold. Consensus levels exceeded 0.75, and convergence scores were 0.5 or lower, confirming the results without needing further rounds. Following expert feedback, “1. Educational environment & context” was reverted to “1. Context.”
2) Results of second Delphi surveys on the evaluation items (Appendix 2)
In the second survey, 16 evaluation items scored 4 or higher, with the exception of “2.1. Curriculum design principles” and “4.4. Performance accomplishments by graduates after graduation.” Fifteen items had a mode and median of 5, indicating high validity, while “4.4. Performance accomplishments by graduates after graduation” was removed for not meeting the CVR criterion of 0.42. All items exceeded the consensus criterion of 0.75 and the convergence criterion of 0.5, validating the results. The CV value was 0.5 or less, allowing the results to be finalized without further rounds of Delphi survey.
Additionally, based on Delphi panel feedback, several names and positions of the evaluation criteria were updated. The item “3.3. Education monitoring status and monitoring system” was removed due to its similarity to “5.1. Continuous curriculum quality assurance and improvement.” Other evaluation items were also renamed for clarity, including changes from “1.2. Teacher and learner analysis” to “1.2. Teacher and learner,” and from “4.3. Performance transfer of learning” to “4.3. Transfer of learning.”
3) Results of second Delphi surveys on the evaluation indicators (Appendix 3)
As presented in Appendix 3, out of the 52 evaluation indicators, 47 received a mean score of 4.0 or higher, while the remaining 5 scored above 3.5, demonstrating the overall high validity of the indicators. Both the mode and median ranged from 4 to 5, and all indicators met the CVR criterion, confirming their validity. Each indicator also surpassed the consensus threshold of 0.75 and the convergence criterion of 0.5, allowing the survey results to be finalized without the need for additional Delphi rounds (Appendix 4).
Unnatural wording in some indicators was revised following feedback from the expert panel. For instance, phrases like “1.2.2. Appropriateness of analyzing stakeholders’ demands” and “2.3.4. Sufficiency of providing education opportunities” were adjusted for greater clarity.
In addition, to improve understanding, the evaluation item 3.3 4. was removed, and its content was integrated into “5.1.2. Appropriateness of curriculum evaluation method.” Certain indicators were also repositioned to improve the clarity and consistency of the evaluation criteria.
As a result of these revisions, a total of five evaluation areas, 16 evaluation items, and 51 evaluation indicators were finalized (Table 3).
Table 3.
Evaluation Criteria Finally Derived by Reflecting the Results of the Second Delphi Survey
| Evaluation areas | Evaluation items | Evaluation indicators |
|---|---|---|
| 1. Context | 1.1. Competences and educational goals | 1.1.1. Clarity of educational competences and educational goals |
| 1.1.2. Appropriateness of analyzing stakeholders’ demands in educational competences and education goals | ||
| 1.1.3. Sufficiency of reflecting essential medical education competences to educational competences an education goal | ||
| 1.1.4. Clarity of recognition of educational competences and educational goals | ||
| 1.2. Teachers and learners | 1.2.1. Integrity of learner analysis | |
| 1.2.2. Integrity of teacher analysis | ||
| 1.3. Administration and organization | 1.3.1. Integrity of finance and regulations for curriculum implementation | |
| 1.3.2. Sufficiency of member consensus on curriculum introduction | ||
| 1.3.3. System of organization in charge of medical education | ||
| 1.3.4. System of organization in charge of evaluation | ||
| 2. Input | 2.1. Curriculum design principles | 2.1.1. Sufficiency of reflecting curriculum design principles |
| 2.1.2. Scrupulosity of syllabus | ||
| 2.1.3. Appropriateness of amount of learning goals | ||
| 2.2. Educational contents | 2.2.1. Appropriateness of educational contents | |
| 2.2.2. Appropriateness of basic medicine-medical humanities-clinical medicine education connection | ||
| 2.2.3. Appropriateness of balance coordination in curriculum and subjects | ||
| 2.3. Educational methods | 2.3.1. Appropriateness of education methods | |
| 2.3.2. Appropriateness of educational programming in consideration of clinical performance capabilities | ||
| 2.3.3. Appropriateness of clinical educational programming in consideration of learner’s level and education after graduation | ||
| 2.3.4. Sufficiency of providing education opportunities through meetings with patients | ||
| 2.3.5. Practice reinforcement | ||
| 2.4. Learner evaluation | 2.4.1. Clarity of establishing basic principles of learner evaluation | |
| 2.4.2. Appropriateness of learner evaluation methods | ||
| 2.4.3. Evaluation focused on performance | ||
| 2.4.4. Clarity of competence achievement evaluation criteria | ||
| 2.4.5. Appropriateness of implementing progress test | ||
| 2.4.6. Appropriateness of providing remediation and reevaluation | ||
| 2.4.7. Appropriateness of portfolio evaluation | ||
| 2.4.8. Appropriateness of formative evaluation and summative evaluation | ||
| 2.5. Teacher support | 2.5.1. Clarity of granting authorities and roles to curriculum supervising professor | |
| 2.5.2. Appropriateness of support for teaching capability development | ||
| 2.6. Learner support | 2.6.1. Appropriateness of learning support for learners | |
| 2.7. Material resources | 2.7.1. Sufficiency of facilities and equipment | |
| 2.7.2. Organic connection between medical schools and hospitals for practical training | ||
| 3. Process | 3.1. Teaching-learning process | 3.1.1. Integrity of education plan implementation |
| 3.1.2. Smoothness of conducting education | ||
| 3.1.3. Appropriateness of feedback | ||
| 3.2. Achievement evaluation process | 3.2.1. Integrity of evaluation plan implementation | |
| 3.2.2. Appropriateness of remediation and reevaluation implementation | ||
| 4. Learning outcome | 4.1. Degree of satisfaction | 4.1.1. Sufficiency of learner’s satisfaction with education |
| 4.1.2. Sufficiency of investigation factors to learner’s satisfaction with education | ||
| 4.2. Competence achievement | 4.2.1. Sufficiency of competence achievement in learner’s knowledge, skills, and attitude | |
| 4.2.2. Appropriateness of unintended learning outcome | ||
| 4.3. Transfer of learning | 4.3.1. Sufficiency of applying basic medical education to clinical medical education | |
| 4.3.2. Sufficiency of applying previous learning contents to clinical practice | ||
| 4.3.3. Sufficiency of application to medical sites after graduation | ||
| 4.3.4. Change in training course | ||
| 5. Continuous quality improvement of curriculum | 5.1. Continuous curriculum quality assurance and improvement | 5.1.1. Appropriateness of evaluation monitoring system for curriculum improvement |
| 5.1.2. Appropriateness of curriculum evaluation method | ||
| 5.1.3. Sustainability of success factors of curriculum | ||
| 5.1.4. Feedback of curriculum evaluation results |
3. Estimation of relative importance of evaluation criteria
The third Delphi survey conducted an analysis of the relative importance of the evaluation criteria. Out of the 20 expert panel members, 17 responded regarding the evaluation criteria confirmed in the previous two Delphi surveys, and the analysis was performed using the AHP.
The composite weights of the evaluation indicators reflect the relative importance of the indicators, incorporating the weights of the evaluation areas, local weights of the evaluation items, and local weights of the evaluation indicators. Table 4 presents the composite weights of the evaluation indicators. The analysis revealed that the highest weights were assigned to the following indicators: “4.2.1. Sufficiency of competence achievement in learner’s knowledge, skills, and attitudes,” “5.1.4. Feedback of curriculum evaluation results,” “3.2.1. Integrity of evaluation plan implementation,” “5.1.2. Appropriateness of evaluation method,” “5.1.1. Appropriateness of evaluation monitoring system for curriculum improvement,” “5.1.3. Sustainability of success factors of curriculum,” “4.3.3. Sufficiency of application to medical sites after graduation,” “3.2.2. Appropriateness of remediation and reevaluation implementation,” “4.3.1. Sufficiency of applying basic medical education to clinical medical education,” and “3.1.3. Appropriateness of feedback.”
Table 4.
Composite Weight Analysis of Evaluation Indicators
| Evaluation areas | Evaluation items | Evaluation indicators | Composite weight | Rank |
|---|---|---|---|---|
| 1. Context | 1.1. Competences and educational goals | 1.1.1. Clarity of educational competences and educational goals | 0.0188 | 18 |
| 1.1.2. Appropriateness of analyzing stakeholders’ demands in educational competences and education goals | 0.0092 | 27 | ||
| 1.1.3. Sufficiency of reflecting essential medical education competences to educational competences an education goal | 0.0207 | 16 | ||
| 1.1.4. Clarity of recognition of educational competences and educational goals | 0.0111 | 25 | ||
| 1.2. Teachers and learners | 1.2.1. Integrity of learner analysis | 0.0271 | 14 | |
| 1.2.2. Integrity of teacher analysis | 0.0140 | 22 | ||
| 1.3.1. Integrity of finance and regulations for curriculum implementation | 0.0046 | 42 | ||
| 1.3.2. Sufficiency of member consensus on curriculum introduction | 0.0033 | 50 | ||
| 1.3.3. System of organization in charge of medical education | 0.0038 | 46 | ||
| 1.3.4. System of organization in charge of evaluation | 0.0040 | 45 | ||
| 2. Input | 2.1. Curriculum design principles | 2.1.1. Sufficiency of reflecting curriculum design principles | 0.0121 | 23 |
| 2.1.2. Scrupulosity of syllabus | 0.0069 | 33 | ||
| 2.1.3. Appropriateness of amount of learning goals | 0.0070 | 31 | ||
| 2.2. Educational contents | 2.2.1. Appropriateness of educational contents | 0.0117 | 24 | |
| 2.2.2. Appropriateness of basic medicine-medical humanities-clinical medicine education connection | 0.0093 | 26 | ||
| 2.2.3. Appropriateness of balance coordination in curriculum and subjects | 0.0070 | 32 | ||
| 2.3. Educational methods | 2.3.1. Appropriateness of education methods | 0.0073 | 29 | |
| 2.3.2. Appropriateness of educational programming in consideration of clinical performance capabilities | 0.0045 | 43 | ||
| 2.3.3. Appropriateness of clinical educational programming in consideration of learner’s level and education after graduation | 0.0054 | 40 | ||
| 2.3.4. Sufficiency of providing education opportunities through meetings with patients | 0.0069 | 34 | ||
| 2.3.5. Practice reinforcement | 0.0073 | 30 | ||
| 2.4. Learner evaluation | 2.4.1. Clarity of establishing basic principles of learner evaluation | 0.0063 | 37 | |
| 2.4.2. Appropriateness of learner evaluation methods | 0.0057 | 39 | ||
| 2.4.3. Evaluation focused on performance | 0.0051 | 41 | ||
| 2.4.4. Clarity of competence achievement evaluation criteria | 0.0066 | 35 | ||
| 2.4.5. Appropriateness of implementing progress test | 0.0033 | 49 | ||
| 2.4.6. Appropriateness of providing remediation and reevaluation | 0.0035 | 48 | ||
| 2.4.7. Appropriateness of portfolio evaluation | 0.0030 | 51 | ||
| 2.4.8. Appropriateness of formative evaluation and summative evaluation | 0.0036 | 47 | ||
| 2.5. Teacher support | 2.5.1. Clarity of granting authorities and roles to curriculum supervising professor | 0.0059 | 38 | |
| 2.5.2. Appropriateness of support for teaching capability development | 0.0090 | 28 | ||
| 2.6. Learner support | 2.6.1. Appropriateness of learning support for learners | 0.0174 | 20 | |
| 2.7. Material resources | 2.7.1. Sufficiency of facilities and equipment | 0.0042 | 44 | |
| 2.7.2. Organic connection between medical schools and hospitals for practical training | 0.0063 | 36 | ||
| 3. Process | 3.1. Teaching-learning process | 3.1.1. Integrity of education plan implementation | 0.0312 | 12 |
| 3.1.2 Smoothness of conducting education | 0.0228 | 15 | ||
| 3.1.3. Appropriateness of feedback | 0.0326 | 10 | ||
| 3.2. Achievement evaluation process | 3.2.1. Integrity of evaluation plan implementation | 0.0633 | 3 | |
| 3.2.2. Appropriateness of remediation and reevaluation implementation | 0.0383 | 8 | ||
| 4. Learning outcome | 4.1. Degree of satisfaction | 4.1.1. Sufficiency of learner’s satisfaction with education | 0.0204 | 17 |
| 4.1.2. Sufficiency of investigation factors to learner’s satisfaction with education | 0.0174 | 21 | ||
| 4.2. Competence achievement | 4.2.1. Sufficiency of competence achievement in learner’s knowledge, skills, and attitude | 0.1082 | 1 | |
| 4.2.2. Appropriateness of unintended learning outcome | 0.0301 | 13 | ||
| 4.3. Transfer of learning | 4.3.1. Sufficiency of applying basic medical education to clinical medical education | 0.0370 | 9 | |
| 4.3.2. Sufficiency of applying previous learning contents to clinical practice | 0.0313 | 11 | ||
| 4.3.3. Sufficiency of application to medical sites after graduation | 0.0411 | 7 | ||
| 4.3.4. Change in training course | 0.0184 | 19 | ||
| 5. Continuous quality improvement of curriculum | 5.1. Continuous curriculum quality assurance and improvement | 5.1.1. Appropriateness of evaluation monitoring system for curriculum improvement | 0.0517 | 5 |
| 5.1.2. Appropriateness of curriculum evaluation method | 0.0598 | 4 | ||
| 5.1.3. Sustainability of success factors of curriculum | 0.0428 | 6 | ||
| 5.1.4. Feedback of curriculum evaluation results | 0.0718 | 2 |
The evaluation indicators that received the lowest weights included: “2.4.7. Appropriateness of portfolio evaluation,” “1.3.2. Sufficiency of member consensus on curriculum introduction,” “2.4.5. Appropriateness of conducting progress tests,” “2.4.6. Appropriateness of providing remediation and reevaluation,” and “2.4.8. Appropriateness of formative and summative evaluations.”
Discussion
This study involved experts from nearly half of the medical schools in South Korea to develop and validate the evaluation criteria. This approach emphasized the value of creating an assessment tool supported by a broad, nationwide panel of experts.
The evaluation criteria were developed based on prior research and interviews with experts. Their validity was confirmed using the Delphi method, which showed that most of the criteria were reliable. In the second round of the Delphi process, there were noticeable improvements in consensus, convergence, and CV values compared to the first round. This suggested increasing agreement among the panel members.
These findings indicate that the Delphi method was applied effectively, especially through its repeated rounds, structured feedback, and the anonymity maintained for participants.
The validation of most of the proposed evaluation criteria also highlighted the importance of evaluating competency-based medical education using a broad framework. This framework incorporates context, input, process, learning outcomes, and continuous quality improvement. Based on the results, integrating the CIPP model with the Kirkpatrick model appeared to be an appropriate approach.
Among the evaluation areas, “2. Input” and “1. Context” demonstrated the highest levels of validity. These categories included a substantial number of valid indicators, suggesting that they should receive particular focus in future assessments.
Beyond confirming the evaluation criteria, the third round of the Delphi survey provided valuable information on the relative importance of each criterion. Through the use of the AHP, the expert panel assigned composite weights to the different evaluation indicators. The indicators given the highest weights were those related to the achievement of competencies by learners and the effectiveness of curriculum evaluation methods.
Notably, evaluation indicator “4.2.1. Sufficiency of competence achievement in learner’s knowledge, skills, and attitudes” received the highest weight, indicating that experts regarded this criterion as highly important. This reflects the growing emphasis on verifying actual competence acquisition, which is the core goal of competency-based medical education, and aligns with the global trend of placing learner outcomes and demonstrated competencies at the center of curriculum evaluation frameworks.
Additionally, this indicator showed strong correlations with other highly weighted criteria, including “5.1.4. Feedback of curriculum evaluation results” and “3.2.1. Integrity of evaluation plan implementation,” suggesting that not only competence achievement but also the mechanisms for curriculum evaluation and feedback are considered essential for ensuring educational quality and accountability.
Thus, systematically evaluating learner competence and applying the results to continuous quality improvement emerges as an effective strategy.
Meanwhile, the “1.3. Administration and organization” domain received relatively lower weights. Although competency-based curricula naturally emphasize learner competence, the importance of administrative and organizational elements—which are critical for curriculum management and quality assurance—appears to have been underestimated. Future curriculum evaluations should appropriately reflect the significance of these supporting structures [18].
Examining the criteria that were excluded from the validation process, it became apparent that many faced challenges regarding practical application. Indicators such as “4.4. Performance accomplishments by graduates after graduation,” “4.4.2. Changes in patients and treatments,” and “4.4.3. Changes in society and science” correspond to the “Outcomes” stage of the Kirkpatrick model, and numerous prior studies have pointed out limitations in actual evaluations of these aspects [28,29]. Although these criteria did not demonstrate validity, their initial formulation during the development phase indicates potential for future evaluation.
Additionally, the indicator “3.3.3. Appropriateness of opportunities for competence acquisition during clinical practice” reflects long-standing debates in domestic medical education, such as how to provide medical students with opportunities to engage with patients during clinical practice [30].
This study has significant implications; however, it also has the following research limitations. First, the evaluation criteria were derived based on previous research, and a Delphi survey with 20 participants. Therefore, it is difficult to claim that the presented evaluation criteria fully reflect the realities of competency-based curricula in domestic medical schools.
Second, while this study gathered evaluation criteria from experts, it did not include the perspectives of various stakeholders in medical education. As a result, one limitation of the findings is that they do not include the perspectives of a wider range of stakeholders, such as students and administrative staff.
Given these limitations, future research should focus on validating the evaluation criteria by including perspectives from a broader range of stakeholders involved in medical education.
Furthermore, additional studies are necessary to examine the practical applicability of the developed criteria to ensure they effectively meet the needs of the educational community.
Appendix 1. Results of first and Second Delphi Surveys on the Evaluation Areas
| Evaluation area | Mean | CVR | Degree of consensus | Degree of convergence | CV |
|---|---|---|---|---|---|
| First survey | |||||
| 1. Context | 4.55 | 0.9 | 0.8 | 0.5 | 0.13 |
| 2. Input | 4.55 | 1 | 0.8 | 0.5 | 0.11 |
| 3. Process | 4.6 | 0.9 | 0.8 | 0.5 | 0.13 |
| 4. Learning outcome | 4.55 | 0.9 | 0.8 | 0.5 | 0.13 |
| 5. Outcome of curriculum | 4.45 | 0.9 | 0.8 | 0.5 | 0.17 |
| Second survey | |||||
| 1. Educational environment & context | 4.75 | 1 | 1 | 0 | 0.09 |
| 2. Input | 4.85 | 1 | 1 | 0 | 0.08 |
| 3. Process | 4.7 | 0.9 | 0.85 | 0.38 | 0.12 |
| 4. Learning outcome | 4.75 | 0.9 | 1 | 0 | 0.12 |
| 5. Continuous quality improvement | 4.8 | 0.9 | 1 | 0 | 0.11 |
CVR: Content validity ratio, CV: Coefficient of variation.
Appendix 2. Results of First and Second Delphi Surveys on the Evaluation Items
| Evaluation item | Mean | CVR | Degree of consensus | Degree of convergence | CV | |
|---|---|---|---|---|---|---|
| First survey | ||||||
| 1.1. | Competences and goals | 4.6 | 1 | 0.80 | 0.50 | 0.11 |
| 1.2. | Design principles | 3.8 | 0.2 | 0.69 | 0.63 | 0.24 |
| 1.3. | Teachers and learners | 4.55 | 0.9 | 0.80 | 0.50 | 0.13 |
| 1.4. | Organization | 4.2 | 0.5 | 0.72 | 0.63 | 0.23 |
| 2.1. | Educational contents and educational methods | 4.35 | 0.5 | 0.75 | 0.63 | 0.23 |
| 2.2. | Educational strategy | 4.25 | 0.6 | 0.80 | 0.50 | 0.26 |
| 2.3. | Evaluation | 4.3 | 0.7 | 0.78 | 0.50 | 0.20 |
| 2.4. | Human resources | 4.5 | 0.8 | 0.80 | 0.50 | 0.15 |
| 2.5. | Material resources and facilities | 4.4 | 0.7 | 0.80 | 0.50 | 0.17 |
| 2.6. | Educational support | 4.4 | 0.8 | 0.80 | 0.50 | 0.19 |
| 3.1. | Teaching-learning process’ | 4.4 | 0.8 | 0.80 | 0.50 | 0.21 |
| 3.2. | Evaluation of learners | 4.7 | 1 | 0.80 | 0.50 | 0.10 |
| 3.3. | Education monitoring and improvement | 4.45 | 0.8 | 0.80 | 0.50 | 0.19 |
| 4.1. | Degree of satisfaction | 4.35 | 0.7 | 0.80 | 0.50 | 0.20 |
| 4.2. | Competence acquisition | 4.4 | 0.8 | 0.80 | 0.50 | 0.19 |
| 4.3. | Application of acquired competences | 4.1 | 0.5 | 0.69 | 0.63 | 0.22 |
| 4.4. | Performance accomplishments by graduates | 4 | 0.5 | 0.69 | 0.63 | 0.21 |
| 5.1. | Sustainability and feedback | 4.45 | 0.8 | 0.80 | 0.50 | 0.15 |
| Second survey | ||||||
| 1.1. | Educational competences and educational goals | 4.8 | 0.9 | 1.00 | 0.00 | 0.11 |
| 1.2. | Analysis of teachers and learners | 4.75 | 0.9 | 1.00 | 0.00 | 0.12 |
| 1.3. | Administrative organization | 4.5 | 0.8 | 0.80 | 0.50 | 0.15 |
| 2.1. | Curriculum design principles | 3.85 | 0.7 | 1.00 | 0.00 | 0.21 |
| 2.2. | Educational contents | 4.8 | 1 | 1.00 | 0.00 | 0.09 |
| 2.3. | Educational methods | 4.8 | 1 | 1.00 | 0.00 | 0.09 |
| 2.4. | Learner evaluation design | 4.75 | 0.9 | 1.00 | 0.00 | 0.12 |
| 2.5. | Teacher support | 4.8 | 1 | 1.00 | 0.00 | 0.09 |
| 2.6. | Learner support | 4.7 | 0.9 | 1.00 | 0.00 | 0.12 |
| 2.7. | Material resources | 4.4 | 0.8 | 0.80 | 0.50 | 0.23 |
| 3.1. | Teaching-learning process | 4.7 | 0.9 | 1.00 | 0.00 | 0.12 |
| 3.2. | Achievement evaluation | 4.6 | 0.7 | 1.00 | 0.00 | 0.19 |
| 3.3. | Education monitoring status and monitoring system | 4.65 | 0.8 | 1.00 | 0.00 | 0.14 |
| 4.1. | Degree of satisfaction | 4.45 | 0.9 | 0.80 | 0.50 | 0.21 |
| 4.2. | Competence achievement | 4.84 | 1 | 1.00 | 0.00 | 0.08 |
| 4.3. | Performance transfer of learning | 4.15 | 0.6 | 0.75 | 0.50 | 0.21 |
| 4.4. | Performance accomplishments by graduates after graduation | 3.9 | 0.4 | 0.50 | 1.00 | 0.26 |
| 5.1. | Continuous curriculum quality assurance and improvement | 4.75 | 0.9 | 1.00 | 0.00 | 0.12 |
CVR: Content validity ratio, CV: Coefficient of variation.
Appendix 3. Results of the Second Delphi Evaluation Indicator Analysis
| Evaluation indicators | Descriptive statistics |
Measures of central tendency |
CVR | |||
|---|---|---|---|---|---|---|
| Mean±SD | Positive rate (%) | Median | Mode | Interquartile range | ||
| 1.1.1. | 4.70±0.47 | 100.00 | 5 | 5 | 4–5 | 1.00 |
| 1.1.2. | 4.40±0.60 | 95.00 | 4 | 4 | 4–5 | 0.90 |
| 1.1.3. | 4.70±0.47 | 100.00 | 5 | 5 | 4.25–5 | 1.00 |
| 1.1.4. | 4.10±0.64 | 85.00 | 4 | 4 | 4–4.25 | 0.70 |
| 1.2.1. | 4.70±0.57 | 95.00 | 5 | 5 | 5–5 | 0.90 |
| 1.2.2. | 4.70±0.47 | 100.00 | 5 | 5 | 4.25–5 | 1.00 |
| 1.3.1. | 4.74±0.45 | 100.00 | 5 | 5 | 5–5 | 1.00 |
| 1.3.2. | 4.50±0.76 | 95.00 | 5 | 5 | 4–5 | 0.90 |
| 2.1.1. | 4.50±0.51 | 100.00 | 4.5 | 4 | 4–5 | 1.00 |
| 2.1.2. | 4.25±0.44 | 100.00 | 4 | 4 | 4–4.25 | 1.00 |
| 2.1.3. | 3.90±0.79 | 90.00 | 4 | 4 | 4–4 | 0.80 |
| 2.2.1. | 4.65±0.59 | 95.00 | 5 | 5 | 4.25–5 | 0.90 |
| 2.2.2. | 4.60±0.60 | 95.00 | 5 | 5 | 4–5 | 0.90 |
| 2.2.3. | 4.15±0.88 | 95.00 | 4 | 4 | 4–5 | 0.90 |
| 2.3.1. | 4.60±0.60 | 95.00 | 5 | 5 | 4–5 | 0.90 |
| 2.3.2. | 4.05±0.94 | 85.00 | 4 | 4 | 4–5 | 0.70 |
| 2.3.3. | 3.85±0.81 | 85.00 | 4 | 4 | 4–4 | 0.70 |
| 2.3.4. | 4.20±0.52 | 95.00 | 4 | 4 | 4–4.25 | 0.90 |
| 2.3.5. | 4.25±0.97 | 90.00 | 4 | 4 | 4–5 | 0.80 |
| 2.4.1. | 4.85±0.37 | 100.00 | 5 | 5 | 5–5 | 1.00 |
| 2.4.2. | 4.65±0.59 | 95.00 | 5 | 5 | 4.25–5 | 0.90 |
| 2.4.3. | 4.45±0.94 | 95.00 | 5 | 5 | 4–5 | 0.90 |
| 2.4.4. | 4.45±0.51 | 100.00 | 4 | 4 | 4–5 | 1.00 |
| 2.4.5. | 4.00±0.79 | 95.00 | 4 | 4 | 4–4 | 0.90 |
| 2.4.6. | 4.40±0.60 | 95.00 | 4 | 4 | 4–5 | 0.90 |
| 2.4.7. | 3.95±0.39 | 90.00 | 4 | 4 | 4–4 | 0.80 |
| 2.4.8. | 4.70±0.47 | 100.00 | 5 | 5 | 4.25–5 | 1.00 |
| 2.5.1. | 4.60±0.50 | 100.00 | 5 | 5 | 4–5 | 1.00 |
| 2.5.2. | 4.30±0.47 | 100.00 | 4 | 4 | 4–5 | 1.00 |
| 2.5.3. | 4.45±0.51 | 100.00 | 4 | 4 | 4–5 | 1.00 |
| 2.5.4. | 4.60±0.60 | 95.00 | 5 | 5 | 4–5 | 0.90 |
| 2.6.1. | 4.37±0.50 | 100.00 | 4 | 4 | 4–5 | 1.00 |
| 2.7.1. | 4.55±0.60 | 95.00 | 5 | 5 | 4–5 | 0.90 |
| 2.7.2. | 4.25±0.91 | 95.00 | 4 | 4 | 4–5 | 0.90 |
| 3.1.1. | 4.26±0.45 | 100.00 | 4 | 4 | 4–4.5 | 1.00 |
| 3.1.2. | 4.21±0.42 | 100.00 | 4 | 4 | 4–4 | 1.00 |
| 3.1.3. | 4.68±0.48 | 100.00 | 5 | 5 | 4–5 | 1.00 |
| 3.2.1. | 4.74±0.45 | 100.00 | 5 | 5 | 5–5 | 1.00 |
| 3.2.2. | 4.53±0.61 | 94.74 | 5 | 5 | 4–5 | 0.89 |
| 3.3.1 | 4.74±0.56 | 94.74 | 5 | 5 | 5–5 | 0.89 |
| 3.3.2. | 4.63±0.60 | 94.74 | 5 | 5 | 4–5 | 0.89 |
| 3.3.3 | 4.32±0.58 | 94.74 | 4 | 4 | 4–5 | 0.89 |
| 4.1.1. | 4.15±0.88 | 95.00 | 4 | 4 | 4–5 | 0.90 |
| 4.1.2. | 3.95±0.94 | 90.00 | 4 | 4 | 4–4 | 0.80 |
| 4.2.1. | 4.75±0.44 | 100.00 | 5 | 5 | 5–5 | 1.00 |
| 4.2.2. | 4.05±0.39 | 95.00 | 4 | 4 | 4–4 | 0.90 |
| 4.3.1. | 4.15±0.59 | 90.00 | 4 | 4 | 4–4.25 | 0.80 |
| 4.3.2. | 4.20±0.62 | 90.00 | 4 | 4 | 4–5 | 0.80 |
| 4.3.3. | 4.15±0.93 | 90.00 | 4 | 4 | 4–5 | 0.80 |
| 4.4.1. | 3.80±0.83 | 80.00 | 4 | 4 | 4–4 | 0.60 |
| 5.1.1. | 4.00±0.86 | 90.00 | 4 | 4 | 4–4 | 0.80 |
| 5.1.2. | 4.85±0.37 | 100.00 | 5 | 5 | 5–5 | 1.00 |
1.1.1. Clarity of educational competences and educational goals; 1.1.2. Appropriateness of analyzing stakeholders’ demands in educational competences and education goals; 1.1.3. Sufficiency of reflecting essential medical education competences to educational competences an education goal; 1.1.4. Clarity of recognition of educational competences and educational goals; 1.2.1. Integrity of learner analysis; 1.2.2. Integrity of teacher analysis; 1.3.1 Integrity of finance and regulations for curriculum implementation; 1.3.2. Sufficiency of member consensus on curriculum introduction; 1.3.3. System of organization in charge of medical education; 1.3.4. System of organization in charge of evaluation; 2.1.1. Sufficiency of reflecting curriculum design principles; 2.1.2 Scrupulosity of syllabus; 2.1.3 Appropriateness of amount of learning goals; 2.2.1. Appropriateness of educational contents; 2.2.2 Appropriateness of basic medicine-medical humanities-clinical medicine education connection; 2.2.3 Appropriateness of balance coordination in curriculum and subjects; 2.3.1. Appropriateness of education methods; 2.3.2 Appropriateness of educational programming in consideration of clinical performance capabilities; 2.3.3. Appropriateness of clinical educational programming in consideration of learner’s level and education after graduation; 2.3.4. Sufficiency of providing education opportunities through meetings with patients; 2.3.5. Practice reinforcement; 2.4.1. Clarity of establishing basic principles of learner evaluation; 2.4.2. Appropriateness of learner evaluation methods; 2.4.3. Evaluation focused on performance; 2.4.4. Clarity of competence achievement evaluation criteria; 2.4.5. Appropriateness of implementing progress test; 2.4.6. Appropriateness of providing remediation and reevaluation; 2.4.7. Appropriateness of portfolio evaluation; 2.4.8. Appropriateness of formative evaluation and summative evaluation; 2.5.1. Clarity of granting authorities and roles to curriculum supervising professor; 2.5.2. Appropriateness of support for teaching capability development; 2.5.3. Systematic structure of the evaluation authority; 2.5.4. Appropriateness of support for faculty development; 2.6.1. Appropriateness of learning support for learners; 2.7.1. Sufficiency of facilities and equipment; 2.7.2. Organic connection between medical schools and hospitals for practical training; 3.1.1 Integrity of education plan implementation; 3.1.2 Smoothness of conducting education; 3.1.3. Appropriateness of feedback; 3.2.1. Integrity of evaluation plan implementation; 3.2.2. Appropriateness of remediation and reevaluation implementation; 3.3.1. Appropriateness of evaluation methods for education; 3.3.2. Convergence of evaluation results for education; 3.3.3 Appropriateness of opportunities for competence acquisition during clinical practice; 3.3.4 Appropriateness of curriculum supervising professor; 4.1.1. Sufficiency of learner’s satisfaction with education; 4.1.2. Sufficiency of investigation factors to learner’s satisfaction with education; 4.2.1. Sufficiency of competence achievement in learner’s knowledge, skills and attitude; 4.2.2. Appropriateness of unintended learning outcome; 4.3.1. Sufficiency of applying basic medical education to clinical medical education; 4.3.2. Sufficiency of applying previous learning contents to clinical practice; 4.3.3. Sufficiency of application to medical sites after graduation; 4.3.4. Change in training course; 5.1.1. Appropriateness of evaluation monitoring system for curriculum improvement; 5.1.2. Appropriateness of curriculum evaluation method; 5.1.3. Sustainability of success factors of curriculum; 5.1.4. Feedback of curriculum evaluation results. Validity Scale: 1 (not valid at all), 2 (not valid), 3 (neutral), 4 (valid), and 5 (highly valid). According to Lawshe (1975), when the expert panel consists of 20 or more members, a CVR value of at least 0.42 indicates validity.
SD: Standard deviation, CVR: Content validity ratio.
Appendix 4. Consensus and Convergence Verification Results among Delphi Panelists for First and Second Evaluation Indicators
| Evaluation indicators | First Delphi survey |
Second Delphi survey |
||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Q1 | Q3 | Consensus level | Convergence score | CV | Q1 | Q3 | Consensus level | Convergence score | CV | |
| 1.1.1. | 4 | 5 | 0.80 | 0.50 | 0.11 | 4 | 5 | 0.80 | 0.50 | 0.10 |
| 1.1.2. | 4 | 5 | 0.75 | 0.50 | 0.20 | 4 | 5 | 0.75 | 0.50 | 0.14 |
| 1.1.3. | 4 | 5 | 0.80 | 0.50 | 0.17 | 4.25 | 5 | 0.85 | 0.38 | 0.10 |
| 1.1.4. | 4 | 5 | 0.75 | 0.50 | 0.20 | 4 | 4.25 | 0.94 | 0.13 | 0.16 |
| 1.2.1. | 4 | 5 | 0.80 | 0.50 | 0.17 | 5 | 5 | 1.00 | 0.00 | 0.12 |
| 1.2.2. | 4 | 5 | 0.78 | 0.50 | 0.15 | 4.25 | 5 | 0.85 | 0.38 | 0.10 |
| 1.3.1. | 4 | 5 | 0.80 | 0.50 | 0.13 | 5 | 5 | 1.00 | 0.00 | 0.10 |
| 1.3.2. | 4 | 5 | 0.75 | 0.50 | 0.21 | 4 | 5 | 0.80 | 0.50 | 0.17 |
| 2.1.1. | 4 | 5 | 0.75 | 0.50 | 0.21 | 4 | 5 | 0.78 | 0.50 | 0.11 |
| 2.1.2. | 4 | 5 | 0.75 | 0.50 | 0.17 | 4 | 4.25 | 0.94 | 0.13 | 0.10 |
| 2.1.3. | 4 | 5 | 0.75 | 0.50 | 0.20 | 4 | 4 | 1.00 | 0.00 | 0.20 |
| 2.2.1. | 4 | 5 | 0.80 | 0.50 | 0.11 | 4.25 | 5 | 0.85 | 0.38 | 0.13 |
| 2.2.2. | 4 | 5 | 0.80 | 0.50 | 0.20 | 4 | 5 | 0.80 | 0.50 | 0.13 |
| 2.2.3. | 3.75 | 4.25 | 0.88 | 0.25 | 0.23 | 4 | 5 | 0.75 | 0.50 | 0.21 |
| 2.3.1. | 4 | 5 | 0.80 | 0.50 | 0.11 | 4 | 5 | 0.80 | 0.50 | 0.13 |
| 2.3.2 | 4 | 5 | 0.75 | 0.50 | 0.20 | 4 | 5 | 0.75 | 0.50 | 0.23 |
| 2.3.3. | 3.75 | 5 | 0.69 | 0.63 | 0.22 | 4 | 4 | 1.00 | 0.00 | 0.21 |
| 2.3.4. | 4 | 5 | 0.75 | 0.50 | 0.17 | 4 | 4.25 | 0.94 | 0.13 | 0.12 |
| 2.3.5. | 4 | 5 | 0.78 | 0.50 | 0.24 | 4 | 5 | 0.75 | 0.50 | 0.23 |
| 2.4.1. | 4 | 5 | 0.80 | 0.50 | 0.15 | 5 | 5 | 1.00 | 0.00 | 0.08 |
| 2.4.2. | 4 | 5 | 0.80 | 0.50 | 0.10 | 4.25 | 5 | 0.85 | 0.38 | 0.13 |
| 2.4.3. | 4 | 5 | 0.78 | 0.50 | 0.17 | 4 | 5 | 0.80 | 0.50 | 0.21 |
| 2.4.4. | 4 | 5 | 0.75 | 0.50 | 0.14 | 4 | 5 | 0.75 | 0.50 | 0.11 |
| 2.4.5. | 3.75 | 4.25 | 0.88 | 0.25 | 0.21 | 4 | 4 | 1.00 | 0.00 | 0.20 |
| 2.4.6. | 4 | 5 | 0.75 | 0.50 | 0.14 | 4 | 5 | 0.75 | 0.50 | 0.14 |
| 2.4.7. | 3.75 | 4 | 0.94 | 0.13 | 0.27 | 4 | 4 | 1.00 | 0.00 | 0.10 |
| 2.4.8. | 4 | 5 | 0.78 | 0.50 | 0.19 | 4.25 | 5 | 0.85 | 0.38 | 0.10 |
| 2.5.1. | 4 | 5 | 0.75 | 0.50 | 0.17 | 4 | 5 | 0.80 | 0.50 | 0.11 |
| 2.5.2. | 4 | 5 | 0.75 | 0.50 | 0.19 | 4 | 5 | 0.75 | 0.50 | 0.11 |
| 2.5.3. | 4 | 5 | 0.75 | 0.50 | 0.17 | 4 | 5 | 0.75 | 0.50 | 0.11 |
| 2.5.4. | 4 | 5 | 0.80 | 0.50 | 0.17 | 4 | 5 | 0.80 | 0.50 | 0.13 |
| 2.6.1. | 4 | 5 | 0.75 | 0.50 | 0.13 | 4 | 5 | 0.75 | 0.50 | 0.11 |
| 2.7.1. | 4 | 5 | 0.78 | 0.50 | 0.15 | 4 | 5 | 0.80 | 0.50 | 0.13 |
| 2.7.2. | 3.75 | 5 | 0.69 | 0.63 | 0.22 | 4 | 5 | 0.75 | 0.50 | 0.21 |
| 3.1.1. | 4 | 5 | 0.75 | 0.50 | 0.16 | 4 | 4.5 | 0.88 | 0.25 | 0.11 |
| 3.1.2. | 4 | 5 | 0.75 | 0.50 | 0.17 | 4 | 4 | 1.00 | 0.00 | 0.10 |
| 3.1.3. | 4 | 5 | 0.80 | 0.50 | 0.18 | 4 | 5 | 0.80 | 0.50 | 0.10 |
| 3.2.1. | 4 | 5 | 0.80 | 0.50 | 0.20 | 5 | 5 | 1.00 | 0.00 | 0.10 |
| 3.2.2. | 4 | 5 | 0.80 | 0.50 | 0.17 | 4 | 5 | 0.80 | 0.50 | 0.14 |
| 3.3.1. | 4 | 5 | 0.80 | 0.50 | 0.17 | 5 | 5 | 1.00 | 0.00 | 0.12 |
| 3.3.2. | 4 | 5 | 0.80 | 0.50 | 0.19 | 4 | 5 | 0.80 | 0.50 | 0.13 |
| 3.3.3. | 3.5 | 4.5 | 0.75 | 0.50 | 0.14 | 4 | 5 | 0.75 | 0.50 | 0.13 |
| 4.1.1. | 4 | 5 | 0.75 | 0.50 | 0.17 | 4 | 5 | 0.75 | 0.50 | 0.21 |
| 4.1.2. | 4 | 5 | 0.75 | 0.50 | 0.19 | 4 | 4 | 1.00 | 0.00 | 0.24 |
| 4.2.1. | 4 | 5 | 0.80 | 0.50 | 0.11 | 5 | 5 | 1.00 | 0.00 | 0.09 |
| 4.2.2. | 4 | 4 | 1.00 | 0.00 | 0.21 | 4 | 4 | 1.00 | 0.00 | 0.10 |
| 4.3.1. | 4 | 5 | 0.75 | 0.50 | 0.18 | 4 | 4.25 | 0.94 | 0.13 | 0.14 |
| 4.3.2. | 4 | 5 | 0.75 | 0.50 | 0.21 | 4 | 5 | 0.75 | 0.50 | 0.15 |
| 4.3.3. | 4 | 5 | 0.75 | 0.50 | 0.19 | 4 | 5 | 0.75 | 0.50 | 0.22 |
| 4.4.1. | 3.75 | 4 | 0.94 | 0.13 | 0.19 | 4 | 4 | 1.00 | 0.00 | 0.22 |
| 5.1.1. | 4 | 4.25 | 0.94 | 0.13 | 0.19 | 4 | 4 | 1.00 | 0.00 | 0.21 |
| 5.1.2. | 4 | 5 | 0.78 | 0.50 | 0.11 | 5 | 5 | 1.00 | 0.00 | 0.08 |
1.1.1. Clarity of educational competences and educational goals; 1.1.2. Appropriateness of analyzing stakeholders’ demands in educational competences and education goals; 1.1.3. Sufficiency of reflecting essential medical education competences to educational competences an education goal; 1.1.4. Clarity of recognition of educational competences and educational goals; 1.2.1. Integrity of learner analysis,1.2.2. Integrity of teacher analysis; 1.3.1. Integrity of finance and regulations for curriculum implementation; 1.3.2. Sufficiency of member consensus on curriculum introduction; 2.1.1. Sufficiency of reflecting curriculum design principles; 2.1.2. Scrupulosity of syllabus; 2.1.3. Appropriateness of amount of learning goals; 2.2.1. Appropriateness of educational contents; 2.2.2. Appropriateness of basic medicine-medical humanities-clinical medicine education connection; 2.2.3. Appropriateness of balance coordination in curriculum and subjects; 2.3.1. Appropriateness of education methods; 2.3.2. Appropriateness of educational programming in consideration of clinical performance capabilities; 2.3.3. Appropriateness of clinical educational programming in consideration of learner’s level and education after graduation; 2.3.4. Sufficiency of providing education opportunities through meetings with patients; 2.3.5. Practice reinforcement; 2.4.1. Clarity of establishing basic principles of learner evaluation; 2.4.2. Appropriateness of learner evaluation methods; 2.4.3. Evaluation focused on performance; 2.4.4. Clarity of competence achievement evaluation criteria; 2.4.5. Appropriateness of implementing progress test; 2.4.6. Appropriateness of providing remediation and reevaluation; 2.4.7. Appropriateness of portfolio evaluation; 2.4.8. Appropriateness of formative evaluation and summative evaluation; 2.5.1. Clarity of granting authorities and roles to curriculum supervising professor; 2.5.2. Appropriateness of support for teaching capability development; 2.5.3. Systematic structure of the evaluation authority; 2.5.4. Appropriateness of support for faculty development; 2.6.1. Appropriateness of learning support for learners; 2.7.1. Sufficiency of facilities and equipment; 2.7.2. Organic connection between medical schools and hospitals for practical training; 3.1.1. Integrity of education plan implementation; 3.1.2. Smoothness of conducting education; 3.1.3. Appropriateness of feedback; 3.2.1. Integrity of evaluation plan implementation; 3.2.2. Appropriateness of remediation and reevaluation implementation; 3.3.1. Appropriateness of evaluation methods for education; 3.3.2. Convergence of evaluation results for education; 3.3.3. Appropriateness of opportunities for competence acquisition during clinical practice; 3.3.4. Appropriateness of curriculum supervising professor; 4.1.1. Sufficiency of learner’s satisfaction with education; 4.1.2. Sufficiency of investigation factors to learner’s satisfaction with education; 4.2.1. Sufficiency of competence achievement in learner’s knowledge, skills and attitude; 4.2.2. Appropriateness of unintended learning outcome; 4.3.1. Sufficiency of applying basic medical education to clinical medical education; 4.3.2. Sufficiency of applying previous learning contents to clinical practice; 4.3.3. Sufficiency of application to medical sites after graduation; 5.1.1. Appropriateness of evaluation monitoring system for curriculum improvement; 5.1.2. Appropriateness of curriculum evaluation method. Validity Scale: 1 (not valid at all), 2 (not valid), 3 (neutral), 4 (valid), and 5 (highly valid). A consensus level of 0.75 or higher indicates high validity. A convergence score of 0.5 or lower indicates high validity. A CV value of 0.5 or lower indicates that no additional rounds are needed, while a CV value between 0.5 and 0.8 suggests relative stability.
CV: Coefficient of variation.
Footnotes
Acknowledgements
None.
Funding
This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2020S1A5B5A17089960).
Conflicts of interest
No potential conflict of interest relevant to this article was reported.
Author contributions
SYL and SHL conceptualized the study design, and drafted the survey questionnaire. SYL. reviewed the draft and questionnaire. SYL acquired the data and wrote the initial manuscript draft. SHL critically reviewed initial manuscript draft. According to the reviews, SYL revised the draft. All authors read and approved the final submitted manuscript.
References
- 1.Lee BH. Development of curriculum theory and practice: international comparison, interdisciplinary discourse, and exchange between research and field practice. Paper presented at: the Korean Society for Curriculum Studies Conference 2006; April 25, 2006; Seoul, Korea.
- 2.Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002;287(2):226–235. doi: 10.1001/jama.287.2.226. [DOI] [PubMed] [Google Scholar]
- 3.Jung HY, Kim JW, Lee S, et al. A study of core humanistic competency for developing humanism education for medical students. J Korean Med Sci. 2016;31(6):829–835. doi: 10.3346/jkms.2016.31.6.829. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Kim SC, Ro YS, Shin SD, et al. Assessment of competence in emergency medicine among healthcare professionals in Cameroon. J Korean Med Sci. 2017;32(12):1931–1937. doi: 10.3346/jkms.2017.32.12.1931. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Magnusson K, Osborne J. The rise of competency-based education: a deconstructionist analysis. J Educ Thought. 1990;24(1):5–13. [Google Scholar]
- 6.Harden RM. AMEE Guide No. 14: outcome-based education: Part 1-an introduction to outcome-based education. Med Teach. 1999;21(1):7–14. doi: 10.1080/01421599978951. [DOI] [PubMed] [Google Scholar]
- 7.Frank JR, Mungroo R, Ahmad Y, et al. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teach. 2010;32(8):631–637. doi: 10.3109/0142159X.2010.500898. [DOI] [PubMed] [Google Scholar]
- 8.Lurie SJ. History and practice of competency-based assessment. Med Educ. 2012;46(1):49–57. doi: 10.1111/j.1365-2923.2011.04142.x. [DOI] [PubMed] [Google Scholar]
- 9.Park HR. Research and development of evaluation criteria for general curriculum of the 4-year colleges. J Curric Stud. 2007;25(3):109–132. doi: 10.15708/kscs.25.3.200709.005. [DOI] [Google Scholar]
- 10.Lee YM, So YH, Ahn DS, Park SH. Evaluating clinical teaching in medicine. Korean J Med Educ. 2002;14(2):293–300. [Google Scholar]
- 11.An JH, Han JJ, Kim NJ, et al. Analytical case study of evaluation of curriculum at a medical school. Korean J Med Educ. 2010;22(1):57–64. doi: 10.3946/kjme.2010.22.1.57. [DOI] [PubMed] [Google Scholar]
- 12.Im EJ, Chang BH. Analysis on teaching evaluation of clinical clerkship at Korean medical schools. Korean J Med Educ. 2011;23(2):119–126. doi: 10.3946/kjme.2011.23.2.119. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Jun JS, Oh HJ. Criteria for evaluating the quality of adult education programs. J Lifelong Educ. 2003;9(1):29–52. Accessed December 20, 2024. https://m.riss.kr/link?id=A76390998. [Google Scholar]
- 14.Ahn ML, Kim MR. Development of quality assurance model and guiding principles for effective cyber education. J Korean Assoc Comput Educ. 2001;4(1):1–10. [Google Scholar]
- 15.Fitzpatrick JL, Sanders JR, Worthen BR. Program evaluation: alternative approaches and practical guidelines. 4th ed. Upper Saddle River, USA: Pearson; 2011. [Google Scholar]
- 16.Hwang JY. Research and development of curriculum evaluation criteria for premedical curriculum [master’s thesis] Seoul, Korea: Seoul National University; 2015. [Google Scholar]
- 17.Seoul National University National, Teacher Training Center for Health Personnel . A guide to evaluation of educational programs for change in medical school education. Seoul, Korea: Seoul National University National, Teacher Training Center for Health Personnel; 2019. [Google Scholar]
- 18.Lee SY, Lee SH, Shin JS. Development of evaluation criteria for competency-based medical education curriculum in medical schools. J Learn Cent Curric Instr. 2023;23(15):727–755. doi: 10.22251/jlcci.2023.23.15.727. [DOI] [Google Scholar]
- 19.Khorramshahgol R, Moustakis VS. Delphic hierarchy process (DHP): a methodology for priority setting derived from the Delphi method and analytical hierarchy process. Eur J Oper Res. 1988;37(3):347–354. [Google Scholar]
- 20.Delbecq AL, Van de Ven AH, Gustafson DH. Group techniques for program planning: a guide to nominal group and Delphi processes. Glenview, USA: Scott, Foresman and Company; 1975. [Google Scholar]
- 21.Dalkey NC. Information systems in management science: a case study of a decision analysis: Hamlet’s soliloquy. Interfaces. 1981;11(5):45–49. [Google Scholar]
- 22.Saaty TL. Fundamentals of decision making and priority theory with the analytic hierarchy process. Pittsburgh, USA: RWS Publications; 2000. [Google Scholar]
- 23.Lawshe CH. A quantitative approach to content validity. Pers Psychol. 1975;28(4):563–575. [Google Scholar]
- 24.Lee JS. The Delphi method. Seoul, Korea: Kyoyookwahaksa; 2001. [Google Scholar]
- 25.Jeon YJ. A study on the job performance competencies & criterions of technology teacher related curriculum education. Korean J Technol Educ. 2006;6(1):121–155. Accessed December 20, 2024. https://kiss.kstudy.com/Detail/Ar?key=2516635. [Google Scholar]
- 26.Kang YJ. Understanding and application of Delphi technique. Seongnam, Korea: Employment Development Institute; 2008. [Google Scholar]
- 27.Khang YH, Yoon SJ, Kang GW, Kim CY, Yoo KY, Shin YS. An application of Delphi method to the assessment of current status of cancer research. Korean J Prev Med. 1998;31(4):844–856. [Google Scholar]
- 28.Jaesam J. Education program evaluation. Seoul, Korea: Kyowookbook; 2004. [Google Scholar]
- 29.Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs: the four levels. San Francisco, USA: Berrett-Koehler Publishers; 2006. [Google Scholar]
- 30.Kim HJ, Huh JS. The right of the clinical training for the medical students and privacy of the patients. Korean J Med Law. 2013;21(2):107–123. Accessed December 20, 2024. http://www.kaml.kr/journal/thesis.asp. [Google Scholar]