Correlation between MMI performance and OSCE performance – a pilot study

Abstract

Objective

The multiple mini-interview (MMI) has been shown to have a positive correlation with early medical school performance, clerkship evaluations, and national licensing examinations. There is limited data on its predictive validity at the postgraduate level.

Methods

Six hundred and nineteen internship candidates were interviewed using the MMI format by the internal medicine residency program of The Reading Health System, between September 2011 and February 2014. Fifty-two interns were recruited. Each intern participated in an objective structured clinical examination (OSCE) 3–4 months after the start of the program. The OSCE score of each intern was used as the independent variable to test the relationship with both the MMI interpersonal score and the MMI overall score.

Results

There was a moderate positive correlation between the average MMI interpersonal score and the communication score on the OSCE, r=0.384, n=52, p=0.005, and a negligible relationship between the average MMI overall score and the communication score on the OSCE, r=0.175, n=52, p=0.214.

Conclusion

The MMI is a useful tool for residency programs to assess interpersonal and communication skills prior to matriculation into residency training. This study provides evidence for its validity in assessing these competencies.

Increasing numbers of undergraduate and graduate medical education programs have adopted the multiple mini-interview (MMI) format for admission interviews (1–12). The MMI has been shown to predict medical student performance in clinical clerkships, an objective structured clinical examination (OSCE), and on licensing examinations (5, 13–15). However, there is little data on the relationship between MMI results and the future performance of residents recruited into postgraduate positions through this methodology. One of the six core competencies of the Accreditation Council for Graduate Medical Education (ACGME) is interpersonal and communication (IPC) skills, which is directly related to health outcomes (16). The MMI is designed specifically to assess IPC skills, which are important to the role of a physician. Although the reliability and acceptability of the MMI in postgraduate admissions has been demonstrated, evidence supporting its predictive validity for assessing specific components of IPC skills including emotional intelligence and ethical reasoning is lacking (7, 12, 17).

Residency training programs seek to recruit medical students with qualities that predict high performance during training and in practice afterward (18). Strong and effective communication skills have been directly associated with patient satisfaction and a physician's ability to provide high-quality care (19, 20). The OSCE in clinical training is designed to assess communication as well as clinical skills, and has been recommended by the ACGME as a tool for evaluating this critical skill (21). We hypothesize that MMI scores in IPC skills should correlate with OSCE scores in the same competency, providing evidence to support the validity of the MMI for postgraduate recruitment.

Methods

This was a retrospective, non-randomized cohort study evaluating the scores of the IPC skills domain of medical interns recruited with an MMI-based selection process and subsequent OSCEs done within their first 4 months of internship.

Participants

In total 619 internship candidates were interviewed by the internal medicine residency program of The Reading Health System, a single, independent academic medical center, between September 2011 and February 2014. During this period, 52 interns matched into the program. As part of an in-training evaluation of all interns, each participated in an OCSE 3–4 months after the start of the program. The study was exempted by the Reading Hospital Institutional Review Board as a component of an ongoing quality improvement project with review of existing data. Therefore, informed consent was not obtained.

MMI

The details of the rationale, design, and process of the MMI at our institution have been published in detail elsewhere (12). To summarize, the MMI was adapted to consist of five MMI stations to last 10 min each, allowing 2 min for candidates to read the scenarios and 8 min to review their thinking regarding the scenario with an MMI trained interviewer. A traditional 20-min interview with the program director was also conducted to ensure that program- and candidate-specific information could be exchanged. At the end of each MMI station, candidates were rated on two characteristics: 1) aggregated IPC skills and 2) overall performance. Ratings were calculated on a seven-point behaviorally anchored Likert scale. Interviewers were also allowed to note any ‘red flag’ issues in a free-text commentary field. Summative scoring in the MMI process was determined by averaging the scores in each of the five stations to create a final score in each of the two characteristics.

OSCE

The OSCE in this study was a 10-station standardized clinical experience constructed by faculty of the Department of Medicine at Sidney Kimmel Medical College at Thomas Jefferson University to assess interpersonal, communication, and clinical skills. Standardized patients were trained to simulate medical histories for purposes of assessing medical interviewing and instructed to record physical examination maneuvers related to their particular simulated complaint. Some stations involved performing procedures on medical mannequins designed for educational purposes. Time allotted at each station was 10 min with 5 min between stations. During these 5 min, examinees left the patient stations to record their responses to three general items: 1) history of present illness, 2) evaluation of the problem, including diagnosis and risk factors, and 3) plan for treatment. During this time, the simulated patients completed their performance checklists of 30–40 items in each evaluated domain. The faculty case content committee provided a scoring protocol with points allocated for complete and for partially correct answers. Scoring of each examination was based on a maximum of 100% in each of four general scoring areas: 1) counseling and patient education, 2) history-taking, 3) physical examination, and 4) communication skills. A total score on the whole OSCE was an average of the four component scores, with a maximum of 100%. The OSCE communication skills subdomains included clarity of questions, listening skills, professional manner, organization of data collection, giving information, body language, affect, empathy and support, and closing the visit. The OSCE was given to one quarter of the intern class on four separate occasions, at a centralized testing center. The same standardized patients were used for each session.

Statistical analysis

Two Pearson correlation coefficients were computed. One was done to assess the relationship between the average MMI interpersonal score and the communication score on the OSCE. The second was done to assess the average MMI overall score and the communication score on the OSCE. The OSCE score was used as the independent variable on the x-axis to test the relationship (association), if any, with both the y-axis variables (MMI interpersonal score and overall). The statistical analyses were performed using SPSS v22.0 (IBM, Armonk, New York, USA). The figures were generated by the same software (Figs. 1 and 2).

Fig. 1.

Scatter plot of intern average score with average MMI interpersonal score.

Fig. 2.

Scatter plot of intern average score with average MMI overall score.

Results

There was a moderate positive correlation between the average MMI interpersonal score and the communication score on the OSCE, r=0.384, n=52, p=0.005, and a negligible relationship between the average MMI overall score and the communication score on the OSCE, r=0.175, n=52, p=0.214.

Discussion

The strong relationship between effective IPC skills and healthcare outcomes has been well established (16, 22). Poor patient – physician communication scores have also been reported to be statistically significant predictors of future complaints to medical regulatory authorities (23). Health care institutions are being challenged to develop innovative ways of teaching and assessing these critical skills at all levels of medical education (24–26). The OSCE has been found to be a reliable tool in assessing communication skills in graduate medical education (21, 27, 28). The MMI as a recruiting tool was implemented to enhance our assessment of communication and interpersonal skills during interviews. Both tests evaluate similar verbal and non-verbal domains of communication that are essential to the practice of medicine. Although the correlation we obtained was moderate, the results support the validity of the MMI in assessing the IPC skills of trainees at the time of recruitment for residency training.

Competence in both cognitive and non-cognitive skills predicts postgraduate clinical performance. Medical schools and residency programs aim to recruit the most competitive candidates in both domains (29–31). The United States Medical Licensing Examination Step 2 Clinical Skills investigates similar domains for IPC but provides only a pass or fail score to residency programs (32). The MMI provides a numeric score, which allows for more dispersion in evaluating the strength of the interviewees’ communication skills.

The performance of applicants on the MMI factors directly into their position on our National Residency Matching Program rank list and has led to a considerable progression of our recruitment class characteristics. Subjective feedback from volunteer faculty has revealed trends toward improved professionalism, observed bedside manner, and communication skills demonstrated by resident physicians since implementing this selection process.

One limitation of our study was the lack of access to OSCE results of trainees recruited prior to implementing the MMI for comparison. This restriction was due to changes in the OSCE center protocols and station content in the years preceding the implementation. As such, our results can only confirm a positive correlation but cannot determine the effect of the MMI on improving performance on the OSCE between pre-intervention and postintervention cohorts.

Although evidence for the reliability, validity, and acceptability of the MMI for postgraduate recruitment is growing, data on the predictive validity of the MMI in graduate medical education are sparse (1, 12). This study provides early evidence supporting the MMI for residency recruitment. However, further research is needed to investigate the predictive validity of the MMI at the postgraduate level.

Conclusions

The MMI is a useful tool for residency programs to assess IPC skills prior to matriculation into residency training. This study provides evidence for its validity in assessing these competencies.

Conflict of interest and funding

The authors have not received any funding or benefits from industry or elsewhere to conduct this study.