Structured oral clinical assessment for pharmacotherapy competencies in medical education: a study of validity and reliability analyses of seven domains

Abstract

Purpose

To develop and validate a structured oral clinical assessment (SOCA) tool to evaluate pharmacotherapy competencies among medical students during clinical clerkships.

Methods

The instrument was designed around seven core pharmacotherapy domains. A total of 30 pharmacology experts participated in the face and content validity assessments. The cognitive complexity was evaluated using Bloom’s taxonomy. Nine trained examiners assessed 77 clerkship students using the SOCA tool. Construct validity was tested using Spearman correlation, whereas inter-rater reliability was analyzed using Cohen’s kappa and Krippendorff’s alpha. Internal consistency was assessed using Cronbach’s alpha.

Results

All items showed strong content validity (item-level content validity index and face validity index=1.00). Most questions reflected moderate to high cognitive complexity (Bloom’s C2–C5). Construct validity was supported by significant domain-total score correlations (r=0.406–0.750; p<0.05). Inter-rater reliability was substantial (kappa=0.651–0.830; Krippendorff’s alpha=0.639–0.834), and internal consistency was acceptable (Cronbach’s alpha=0.759).

Conclusion

The SOCA tool has strong validity and reliability for evaluating pharmacotherapy competence through oral clinical examination. It offers a structured, feasible alternative to existing formats and has the potential for broader use following external validation.

Introduction

Safe and effective prescription of medications is a fundamental responsibility for medical graduates. However, several studies have reported that newly qualified doctors often lack adequate pharmacotherapy skills, including drug selection, dosage calculation, and recognition of adverse drug interactions [1]. These deficiencies not only undermine patient care quality but also contribute to medication errors, which are a significant cause of preventable harm in healthcare settings [2]. The gap between theoretical pharmacological knowledge and its clinical application underscores the need for effective educational and assessment strategies in undergraduate medical curricula [3].

One approach that has gained attention for its potential to bridge this gap is the structured oral clinical assessment (SOCA). Combining oral examination with structured clinical scenarios requires students to apply pharmacological knowledge in real-world decision-making contexts [4]. Unlike multiple-choice tests, oral examination promotes active reasoning, communication, and professional judgment, offering a more authentic and formative evaluation of clinical competence in pharmacotherapy [3]. Despite its educational promise, SOCA remains underutilized, partly because of the lack of standardized implementation and validated assessment tools.

Educators are required to consider various preparations, including the assessment instrument, administration, time management, exam room arrangements, and detailed planning [5]. It is also essential to recognize that subjectivity may influence assessment because some examiners might ask more questions than others. Furthermore, student anxiety can greatly affect performance and assessment results [6]. Any assessment method must undergo rigorous psychometric validation to ensure its effectiveness and credibility. This is particularly important in high-stakes and formative evaluations where the results influence learning and progression. Validation ensures that the instrument accurately measures the intended competencies, is applied consistently, and provides meaningful feedback to learners and educators [7].

In response to the recognized gaps in pharmacotherapy education and the potential of SOCA as a structured assessment tool, this study aims to evaluate its validity and reliability in assessing pharmacotherapy competencies among medical students during clinical clerkships. This study provide insight into its educational value and potential for broader application in clinical education by identifying how SOCA contributes to learning through its assessment process, feedback, and reflective components.

Methods

1. Study design

We performed a quantitative approach with a cross-sectional design. This study took place at the Division of Pharmacology and Therapy, the Faculty of Medicine, Universitas Airlangga, from July 2024 to January 2025. The study was approved by the research ethics committee, Universitas Airlangga Hospital, with ethical approval number 174/KEP/2024. The development of the SOCA instrument followed a structured multi-phase process (Fig. 1). Initially, we designed a rubric based on the Indonesian National Medical Doctor Competencies. This rubric guided the creation of the pharmacotherapy SOCA instrument across seven domains [8]. These domains included drug classification and efficacy, mechanism of action, potential side effects, interactions, dosage, administration, prescription interpretation, and rational drug use. In the first phase, SOCA Model 1 underwent expert review by 30 pharmacology and education specialists to assess its content and face validity. Informed by these reviews, a revised instrument was developed through focus group discussions (FGDs) and in-depth interviews with faculty members, yielding SOCA Model 2. A second validity analysis was conducted with the same expert group, focusing on content validity, face validity, construct validity, and cognitive complexity. Finally, reliability testing was performed through a 3-station rotation design, in which students completed SOCA based on three prescriptions (A, B, and C) across different desks according to the rubric blueprint (Supplement 1). Each student was assessed by three independent examiners. Inter-rater reliability and internal consistency were evaluated to determine the robustness of the instrument across the seven domains.

Fig. 1. Flowchart of the Development, Validation, and Reliability Analysis of the SOCA Instrument.

The process includes initial instrument design aligned with national competencies, two phases of expert review for validity analyses, qualitative refinement based on focus group and interview feedback, and final implementation in a three-station rotation with structured assessments and reliability testing. SOCA: Structured oral clinical assessment.

We analyzed the qualitative data from FGDs and in-depth interviews using thematic analysis. Audio recordings were transcribed verbatim and reviewed for accuracy. Two independent researchers (A.K.R.P. and D.S.P.) performed open coding to identify meaningful units related to the clarity, relevance, and clinical applicability of SOCA questions. They then compared, grouped, and refined these codes into subthemes. These themes directly informed targeted revisions to the SOCA items. To ensure analyst triangulation, a third reviewer (A.M.) independently reviewed coding outputs and theme structure. Discrepancies were discussed until consensus was reached. The final themes were mapped back to the SOCA model to ensure content validity and are reflected in Supplement 2.

2. Study participants

All final-year medical students scheduled for the clinical pharmacology rotation during the semester were invited to participate in the study. Participation in the SOCA was integrated into their standard clinical assessment process. However, the results were used solely for research purposes and did not affect students’ course grades. Recruitment was conducted through verbal and written invitations. All students participating in this study had already completed tutorials on prescription writing related to medical issues across different organ-systems in a two-week clerkship rotation in the Pharmacology Department. In addition, we employed a target Cronbach’s alpha of 0.80, with an acceptable minimum of 0.70, a significance level of 0.05, and a statistical power of 80%. The minimum sample size calculated using Bonett’s formula was 32 to ensure a reliable assessment of internal consistency for seven items, or 65 to establish construct validity with a moderate correlation coefficient (r-table) of 0.329. Consequently, we extended invitations to 85 students, who were evaluated by three raters.

3. Analysis of instrument validity

The validation analysis of the instrument involved a panel of pharmacology experts selected through purposive sampling. The validity analyses undertaken in this study included face validity, content validity, cognitive complexity validity, and construct validity analyses. Face validity pertained to the clarity of instructions and language within the questions or rubrics, evaluated using the face validity index (FVI), with an accepted threshold item-level FVI (I-FVI) of 0.83 [9]. The index demonstrating universal agreement among panelists was referred to as FVI/UA, along with its average, denoted as FVI/Ave. In addition, we performed an expert panel to review the content through FGD and in-depth interviews. We used the item-level content validity index (I-CVI) based on expert consensus, as outlined by Lawshe [10], to determine universal agreement (CVI/UA) and its average score (CVI/Ave). If an I-CVI score fell between 0.70 and 0.79, it indicated that the item required revision, while a score below 0.70 suggested it should be excluded [10]. Overall, the face and content were considerably acceptable when absolute agreement in consensus was achieved, or the Index/UA reached one across all domains.

We identified the cognitive complexity of each item using Bloom’s taxonomy, which consists of six distinct levels (C1–C6) [11]. Level C1 is the fundamental level at which students are expected to recall or recognize information, whereas C2 pertains to understanding and explaining ideas or concepts. Level C3 relates to the application of information, while Level C4 is associated with analytical thinking. At C5, students are required to make judgments through evaluating and critiquing all evidence or processes. Furthermore, C6 is the highest level, which involves the integration of various elements together to create original work based on prior knowledge and skills. Cognitive complexity was assessed by asking 30 pharmacology experts to independently rate each item in the SOCA tool using a structured rubric based on Bloom’s taxonomy (C1 to C6). Experts selected the cognitive level they believed each item targeted. The final cognitive level assigned to each item was determined by the proportion of experts who assigned each Bloom level to a given domain, demonstrating how consensus varied across domains.

This cognitive framework was further mapped to the four core national competencies for medical practitioners in Indonesia, ensuring that each domain of the SOCA aligns with expected levels of clinical reasoning and patient management. Competency 1 necessitates the recognition and explication of case, which corresponds with lower-order cognitive skills (C2–C3), while Competency 4 permits diagnosis and independent management to tackle accomplishing the treatment, which aligns with higher-order levels (C4–C5). This alignment reinforces the relevance of SOCA in evaluating real-world clinical competence [8]. In addition, we evaluated the construct validity for the SOCA instrument by using r-table and r-count. The minimum validity threshold was derived from the r-table, r-count, a 95% confidence interval (CI), and a significant p-value. An item is considered valid if the correlation coefficient of the r-count exceeds the r-table value or if the p-value is less than 0.05.

4. Analysis of instrument reliability

We analyzed inter-rater reliability to evaluate how well the experts agreed on each indicator within the instrument. This analysis employed the Cohen kappa agreement coefficient to ascertain the extent of agreement among raters [12]. Three raters evaluated pharmacotherapy competence using Cohen’s weighted kappa and Krippendorff’s alpha, which were considered reliable at both values of 0.60 or above. Notably, we performed an internal consistency reliability analysis using a value of Cronbach’s alpha. Based on this value, we categorized reliability as very good (0.90 and above), high (0.70–0.89), moderate (0.50–0.69), and low (below 0.50) [13]. Furthermore, we conducted inter-item correlation to address the degree of correlation between each pair of items within a set of questions. It was accepted when the average inter-item correlation fell between 0.15 and 0.50 [14].

5. Statistical analysis

The analyses of FVI, CVI, and cognitive complexity were summarized in descriptive statistics. Construct validity was evaluated via Spearman’s Rho correlation utilizing JASP software ver. 0.18.3 (Apple Silicon; JASP Team, Amsterdam, The Netherlands). Reliability analysis employed Cohen kappa and Cronbach alpha correlations with the same version of JASP. We observed some notable statistical differences, with a p-value of less than 0.05.

Results

The characteristics of the 30 experts and nine examiners involved in the study are presented in Table 1. The median age was 44 years (interquartile range [IQR], 40–46 years) for experts and 40 years (IQR, 36–44 years) for examiners. Among the experts, 40% were male and 60% female. All held an MD (Doctor of Medicine) degree with advanced educational backgrounds, including master’s degree (30%), medical specialists (26.7%), and PhD (Doctor of Philosophy) in medicine (43.3%). Most experts (60%) served as medical teachers, while 40% were both medical teachers and practitioners. Both experts and examiners had a median of 16 years (IQR, 5.3–18.0 years) of experience in medical education. For reliability analysis, eight out of 85 invited students were unable to attend. Therefore, 77 students contributed to the study and were rated.

Table 1.

Characteristics of the Experts Involved in the Study

Characteristic	Experts (n=30)	Examiners (n=9)
Age (yr)	44 (40–46)	40 (36–44)
Gender
Man	12 (40.0)	3 (33.3)
Woman	18 (60.0)	6 (66.7)
Holding an MD degree	30 (100.0)	9 (100.0)
Education background
Master’s degree in medicine	9 (30.0)	3 (33.3)
Specialist doctor	8 (26.7)	2 (22.2)
PhD in medicine	13 (43.3)	4 (44.4)
Position
Medical teacher	18 (60.0)	7 (77.8)
Medical teacher and practitioner	12 (40.0)	2 (22.2)
Experience years as medical teacher (yr)	16.0 (5.3–18.0)	16.0 (5.3–18.0)

Data are presented as median (25% percentile–75% percentile) or number (%).

MD: Doctor of Medicine, PhD: Doctor of Philosophy.

1. Face and content validity analysis of the first model

In the face validity analysis, SOCA Model 1 showed I-FVI ranging from 0.63 (Domain 2) to 0.93 (Domain 1 and 5), with all items exhibiting the Index/UA value of 0, indicating a lack of consensus among experts. Conversely, SOCA Model 2 displayed a consistent I-FVI of 1.00 alongside the Index/UA of 1 for FVI, reflecting universal agreement. In terms of content validity analysis, all domains in SOCA Model 1, with the exception of Domain 5, exhibited an I-CVI of 1.00. The Index/AVE for CVI in SOCA Model 1 was 0.98. Notably, it is important to note that Model 1 lacked universal agreement (CVI/UA=0).

2. Adjusting the SOCA instrument

Qualitative analysis of data from FGDs and in-depth interviews informed critical refinements to the SOCA instrument. Through open coding of the transcripts, we identified recurring issues grouped into three overarching themes: (1) cognitive alignment with Bloom’s taxonomy, (2) contextual clarity for clinical application, and (3) appropriateness of question format and wording. Specifically, participants emphasized the need for more precise prompts when assessing drug side effects, which directly led to the refinement of Domain 3 questions. Feedback also highlighted the use of vague terminology, such as “signa,” and the limited applicability of some drug interaction questions; these insights prompted targeted revisions in Domains 4 and 6 to improve contextual clarity. Furthermore, concerns about the complexity of pediatric dosage calculations and the absence of rationale-based prescribing tasks informed enhancements in Domains 5 and 7, ensuring better cognitive alignment and practical relevance. These thematic insights were derived through consensus between two primary coders and validated via analyst triangulation with a third reviewer. The finalized SOCA model, reflecting these modifications, is presented in Table 2, while representative excerpts from the qualitative data and the detailed analytical process are summarized in Supplement 2, providing a clear link between feedback and revision.

Table 2.

Develop SOCA Model 2 from SOCA Model 1 on the Seven Domains of Pharmacotherapy Competencies

Domain	Pharmacotherapy competency	SOCA Model 1 questions	Model 1		SOCA Model 2 questions	Model 2
			I-FVI	I-CVI		I-FVI	I-CVI
1	Explaining the classification and efficacy of drug	Explain the classification and pharmacological effects of all medications in the prescription?	0.93	1.00	What is the classification and pharmacological effects of the druga)inthiscase?	1.00	1.00
2	Explaining the mechanism of action of the drug	Explain the mechanism of action of the drug in the prescription?	0.63	1.00	How is the mechanism of action of the druga)inthispatient?	1.00	1.00
3	Explaining the side effects of the drug	What are the side effects of the medication in this prescription?	0.80	1.00	What are the side effects of the druga)?	1.00	1.00
					Why does the drug cause the side effectb)?
4	Understanding the application in clinical settings where potential drug interactions may occur	How do drug interactions occur in this prescription?	0.70	1.00	How does the interaction mechanism between two drugsc)?	1.00	1.00
					Pharmacokinetics like absorption, distribution, metabolism, excretion, or pharmacodynamic interaction
5	Analyzing drug dosage	What is the dosage of the medication for adults?	0.93	0.89	What is the dosage regimen for the drug “amoxicillin” in adults?	1.00	1.00
		What is the dosage of the medicine for children?			If the pediatric patient weighs 15 kg, how many mL of the syrup or mg of the medication should be given in a single administration?
6	Explaining and interpretation prescription	Read m.f.l.a and signa on a prescription? Explain their meaning?	0.90	1.00	What is the meaning of Latin language from the prescription of “m.f.l.a” and “signa 3 d.d. cth I p.r.n.”?	1.00	1.00
					What does it mean?
7	Evaluating the rational use of drugs and health education	Why is a tablet, capsule, powder, syrup, IV, IM, and SC given to the patients?	0.90	1.00	Is the use of the druga)intheprescription,correct?	1.00	1.00
					Are there any other drugs that can be used in this case?
		When is the right time to take the medicine?			What are the contraindications for administering the druga)?
Index/Ave			0.83	0.98		1	1
Index/UA			0	0		1	1

SOCA: Structured oral clinical assessment, I-FVI: Item-level face validity index, I-CVI: Item-level content validity index, m.f.l.a: Misce fiat lege artis, 3 d.d.: Ter die, cth: Cum theca, p.r.n.: Pro re nata, IV: Intravenous, IM: Intramuscular, SC: Subcutaneous, Index/Ave: Average of validity index, Index/UA: Validity index in universal agreement method.

^a)Every drug’s name must be mentioned clearly, e.g., amoxicillin.

^b)Choose one side effect, e.g., allergy or hypersensitivity.

^c)The two drugs must be mentioned, e.g., the interaction between simvastatin and gemfibrozil.

3. Validity analysis of cognitive complexity

The SOCA items demonstrated a range of cognitive levels, with most domains aligning with Bloom’s C2 (understanding) through C5 (evaluating). As shown in Fig. 2, the highest consensus was observed for domains related to mechanism of action, clinical indication, and adverse effects, with over 60% of experts rating these items at C4 or C5, reflecting application and evaluative reasoning. In contrast, domains such as prescription format and terminology had greater variation, with some items rated at C2 or C3, suggesting lower but still meaningful cognitive engagement. The diversity in ratings indicates that the SOCA tool assesses a spectrum of cognitive complexity, consistent with its design intent to scaffold clinical reasoning across levels.

Fig. 2. Cognitive Complexity Mapping of SOCA Pharmacotherapy Domains Based on Expert Panel Consensus.

Each cell represents the proportion (percentage) of experts who assigned a specific Bloom’s taxonomy cognitive level (C1 to C6) to each given domain (1–7). The intensity of the color within each cell indicates the level of expert agreement. This visualization demonstrates the distribution of cognitive demands, with Domains 1, 2, 3, and 6 primarily aligning with understanding (C2), Domain 4 with application (C3), Domain 5 with analysis (C4), and Domain 7 with evaluation (C5), thereby supporting the cognitive validity of the structured oral clinical assessment (SOCA) instrument.

4. Construct validity analysis

The analysis of construct validity showed that all domains exhibited a p-value below 0.05 and an r-count exceeding the r-table value of 0.329 (Table 3). Domain 5, regarding competency in analyzing drug dosage, had a strong correlation with the r-count of 0.75 (95% CI, 0.581–0.861).

Table 3.

Analysis of r-table and r-count in the Construct Validity Analyses and Inter-rater and Internal Consistency in Reliability Analyses

Domain	r table	r count (95% CI)	p-value	Validity	Average Cohen’s weighted κ	Krippendorff’s α	Cronbach’s αa)	Item-rest correlationb)	Reliability
1	0.329	0.718 (0.538–0.834)	<0.001	Valid	0.793	0.755	0.727	0.718	Reliable
2	0.329	0.739 (0.560–0.850)	<0.001	Valid	0.830	0.787	0.716	0.658	Reliable
3	0.329	0.693 (0.443–0.858)	<0.001	Valid	0.770	0.772	0.724	0.673	Reliable
4	0.329	0.484 (0.183–0.730)	0.003	Valid	0.651	0.639	0.750	0.399	Reliable
5	0.329	0.750 (0.581–0.861)	<0.001	Valid	0.690	0.692	0.721	0.689	Reliable
6	0.329	0.599 (0.345–0.765)	<0.001	Valid	0.775	0.834	0.736	0.624	Reliable
7	0.329	0.406 (0.060–0.676)	0.014	Valid	0.693	0.746	0.751	0.360	Reliable

CI: Confidence interval.

^a)Overall Cronbach’s α=0.759 (95% CI, 0.735–0.793).

^b)Average interitem correlation=0.419.

5. Reliability analysis

All domains were found to be reliable (Table 3), with kappa values ranging from 0.651 (Domain 4) to 0.830 (Domain 2) and Krippendorff’s alpha values ranging from 0.639 (Domain 4) and 0.834 (Domain 6). The overall Cronbach’s alpha for the developed instrument was 0.759 (95% CI, 0.735–0.793), and the average inter-item correlation was 0.419 (min–max, 0.399–0.718). This indicates a commendable level of internal consistency in assessing pharmacotherapy competence.

Discussion

This study confirmed the SOCA instrument, encompassing seven pharmacotherapy competency domains, to be a valid and reliable tool with C2–C5 cognitive levels for clinical clerkship assessments. The construct validity was confirmed by statistically significant correlations between individual domain scores and overall performance (r=0.406–0.750), exceeding the critical r-table value. Internal consistency was supported by a Cronbach’s alpha of 0.759, while inter-rater agreement was strong, with Cohen’s kappa and Krippendorff’s alpha values consistently above 0.60 across domains. These results affirm the tool’s psychometric robustness. The findings contribute to the growing emphasis on competency-based education in medical curricula. This is achieved by providing a structured and reproducible tool that supports reliable assessment of pharmacotherapy skills. The SOCA format allows examiners to assess not only factual knowledge but also students’ clinical reasoning through oral responses aligned with Bloom’s taxonomy levels. Its design facilitates fair evaluation across multiple raters and offers a feasible alternative to high-resource assessments, particularly in limited infrastructure settings. These findings underscore the value of the SOCA instrument in clinical education. The achievement of C5-level cognitive complexity, for instance, illustrates the tool’s ability to assess advanced clinical reasoning, such as evaluating treatment options and making patient-centered prescribing decisions. The unanimous expert agreement on item relevance and clarity (I-CVI/FVI=1.00) further confirms the tool’s alignment with professional standards and educational goals.

The achievement of C5-level cognitive complexity illustrates the tool’s ability to assess advanced clinical reasoning, such as evaluating treatment options and making patient-centered prescribing decisions. The unanimous expert agreement on item relevance and clarity (I-CVI/FVI=1.00) confirmed the tool’s alignment with professional standards and educational goals. Furthermore, Domain 7 exhibited a comparatively lower r-value (0.406). This may reflect the complexity of assessing rational drug use and health education. This domain aligns with higher-order cognitive skills (Bloom’s C5), requiring students to integrate pharmacological knowledge with clinical reasoning. Such demands may introduce variability in performance, thereby contributing to a lower correlation coefficient. The substantial inter-rater reliability further supports its feasibility, fairness, and reproducibility of the SOCA tool in real-world assessment settings, where different faculty members may serve as examiners. Collectively, these outcomes affirm SOCA’s potential as a robust, competency-based assessment tool in pharmacotherapy education.

The validity of the SOCA oral examination instrument encompasses several crucial factors. First, having a clear definition of the SOCA construct is essential, reflecting the necessity of agreement among examiners in exploring students’ capabilities in rational drug use [15]. Second, well-defined assessment criteria ensure objectivity and mitigate variability among assessors, thereby enhancing inter-rater reliability [16]. Third, training examiners on the use of the instruments fosters a shared understanding and minimizes discrepancies [17]. Furthermore, involving multiple examiners reduces assessment biases [18]. The SOCA oral examination encounters challenges, including inconsistent ratings among examiners. Implementing standardized rubrics and question formats can facilitate uniform evaluations, thereby decreasing reliance on personal judgments and improving overall reliability [19]. By adopting these strategies, educators can establish a fair framework for assessing pharmacotherapy competencies of students.

Assessment plays a vital role in curriculum evaluation, as it measures the quality and effectiveness of students’ learning [20]. In the context of medical education, pharmacotherapy competence encompasses the skills, knowledge, and behaviors that healthcare professionals require to manage drug therapy confidently. This competence ensures the safe and effective use of medications in clinical settings [21]. It involves thoughtfully assessing patient needs, selecting appropriate medications, monitoring outcomes, and educating patients regarding their care plans. Achieving this level of competence requires a profound understanding of pharmacology, including an awareness of drug mechanisms, interactions, side effects, and patient-specific factors such as age, gender, genetics, and comorbidities [22].

The SOCA tool, while grounded in oral assessment traditions, offers several innovations that distinguish it from existing approaches. Unlike objective structured clinical examinations, which require physical stations, standardized patients, and extensive logistical planning, SOCA can be implemented with minimal resource demands within regular clerkship rotations. Unlike P-scribe and other computer-based prescribing tools, SOCA integrates examiner interaction, enabling real-time probing of students’ reasoning processes. Furthermore, its structured format, based on seven validated domains and explicitly linked to Bloom’s taxonomy (C2–C5) [23], allows educators to evaluate pharmacotherapy competence across multiple cognitive levels, not just task completion. These features contribute to the practical value and originality of SOCA, particularly in clinical education settings that seek scalable, competency-aligned assessment models.

The examiner plays a crucial role in shaping student performance. An examiner who offers support and constructive feedback can reduce anxiety and improve results. By providing criticism following practice sessions, students can recognize areas that require improvement. Regular feedback facilitates adjustments to study strategies and improves assessment preparedness [24]. However, certain students may exhibit bias in their self-evaluations, frequently overestimating their performance compared with the actual results. This discrepancy can result in inadequate preparation and subsequently detract from performance [25]. Understanding the classification, efficacy, and interactions of drugs is crucial for safe prescribing. Integrating theoretical knowledge with practical application enhances the assessment of students’ pharmacology proficiency during clinical clerkships. The performance of medical students in oral examinations is influenced by cognitive abilities, emotional states, motivation, interactions, and the environment. Targeted education strategies that address these factors can improve the preparedness of students and evaluator in the evaluation process.

This study represents the first step in the development of a SOCA tool designed to evaluate pharmacotherapy competency during clinical clerkships in Indonesia. We obtained evidence in academic settings through the oral assessment of SOCA, which was intended to assess clinical skills and problem-solving capabilities. However, it exhibits methodological limitations. The assessment does not incorporate direct patient interaction, which is crucial for medical education. Students’ stress levels during the oral examination may detrimentally influence their performance during the assessment and subsequently affect the outcomes. Moreover, the characteristics of the participants in this study were locally representative and require evaluation at either the national or international level. Several strategies are recommended to address these limitations. To address these limitations, several strategies are recommended. First, integrating high-fidelity simulation or standardized patients into the SOCA format could increase clinical realism and better capture students’ patient interaction and communication skills. Second, pilot designs for multi-institutional implementation of SOCA would allow for the evaluation of its generalizability across different curricula, learner populations, and cultural contexts. Finally, incorporating structured examiner training programs, including calibration exercises and the use of scoring anchors, could further enhance scoring consistency and reduce potential rater bias. These future directions provide a clear pathway for refining the tool and strengthening its applicability in diverse medical education environments.

In conclusion, the SOCA instrument demonstrates strong evidence of validity and reliability as a structured oral assessment of pharmacotherapy competence within a clinical clerkship context. Its alignment with cognitive complexity, substantial inter-rater agreement, and high expert consensus support its use in medical education. However, given that this study was conducted in a single institutional setting, further external validation across multiple sites is necessary before considering broader implementation. Future studies should explore its adaptability to diverse educational and cultural contexts.

Supplementary materials

Supplementary files are available from https://doi.org/10.3946/kjme.2025.345

Supplement 1.

Blueprint of the SOCA Rubric Used in Assessing Pharmacotherapy Knowledge according to the MD Competency Level.

Supplement 2.

The Example of the Results from Open Coding of FGDs and Interviews.