Abstract
Objectives
To determine the impact of a pregraduation diabetes certificate program on PharmD students' knowledge and skills.
Methods
A comprehensive elective in diabetes was created and implemented in the third-professional year of the PharmD curriculum. A nonrandomized, single-blinded, controlled, 2-year study was conducted to determine the impact of the elective. Written and oral examinations were administered to the participants and students in a control group.
Results
The certificate students' (N = 25) and control students' (N = 12) average oral examination grades were 88.5% ± 13.4% and 89.5% ± 15.8%, respectively (p = 0.58). The certificate students' average grade on the counseling section of the oral examination was 9% higher than that of the students in the control group (p = 0.01). The certificate students' and control students' grades on the written examination were 80.9 ± 11.1% and 61.1 ± 17.4% (p = 0.0062), respectively.
Conclusions
A diabetes certificate program improved students' knowledge of diabetes disease state management and patient education skills compared to students who did not take the elective.
Keywords: diabetes certificate program, diabetes, assessment, certificate
INTRODUCTION
The burden of diabetes and its complication on the US population is enormous. The Center for Disease Control and Prevention projects that the incidence of diabetes in the United States will increase from 13 million in 2002 to 29 million by 2050.1 The complications of diabetes can be reduced by maintaining euglycemia.2,3 Currently, only 38.5% of patients with diabetes achieve glycemic control.4 Pharmacists' capacity to improve the glycemic control of patients in a variety of settings has been well documented.5-18 Certificate programs are a common way to train pharmacists in diabetes education.19-23 Plake et al found that completing a postgraduate diabetes certificate program for pharmacists increased the extent of/quality of pharmaceutical care the pharmacists provided, number of patient interactions they conducted, and the amount they were reimbursed for services.22 Another study showed that completing a diabetes certificate program improved pharmacists' performance test grades and their confidence in providing diabetes care services.21 Two studies showed that completing a diabetes certificate program improved pharmacists' grades on an oral examination using standardized patients.19,23 Only one study that assessed a diabetes certificate program used a control group of pharmacists for comparison.24 Pharmacists who completed the intensive pharmaceutical care training program gained significant knowledge and problem-solving ability compared to the control group.
Certificate or concentration training for pharmacy students in diabetes care and immunization administration has been described.25-27 The number of US pharmacy schools that have a certificate program incorporated into their curricula has not been reported. There is one report of the incorporation of the American Pharmacists Association's immunization certificate program into a pharmacy curriculum.25 Students who completed the program received an average grade of 94% ± 5% on the posttest. There is little published information on the effectiveness of pregraduation diabetes certificate programs on student learning. In 2001, the University of Washington School of Pharmacy initiated its diabetes education program for third-professional year students. This program integrated lectures, problem-base cases, role-play, assessment laboratories, web-based exercises, and a dispensing laboratory. The authors reported increased student knowledge, improved confidence, and positive program evaluations.27
Johnson et al described a 3-credit hour advanced diabetes care elective course that was part of a 24-credit hour diabetes concentration.26 The requirements of this concentration included 10-credit hours of the following pharmacy electives: the advanced diabetes care elective, a nutrition support elective, a course on the psychosocial aspects of chronic illness, and a course entitled Integrated Diabetes Cases. Eight-credit hours of experiential courses that focused on diabetes care were also required, as well as 6 credit hours of electives outside the pharmacy program. The Advanced Diabetes Care course was designed to increase student knowledge of medical and psychosocial issues in diabetes, pharmacotherapy of diabetes, diabetes medical nutrition management, marketing diabetes care services, and diabetes patient education. Preintervention and postintervention survey instruments indicated that this course increased students' skills and knowledge.
The objective of this study was to determine whether completing a diabetes certificate program would affect students' performance on written and oral diabetes examinations. After completion of the course students should be able to:
(1) Assist patients in the management of diet, exercise, glucose monitoring, and medication and insulin administration;
(2) Describe the requirements for implementing pharmaceutical care for patients with diabetes in a community pharmacy setting;
(3) Describe and address psychosocial issues that may affect diabetes care;
(4) Identify and describe important issues in the diabetes management in geriatric, pediatric, and pregnant patients; and
(5) Describe ongoing research in diabetes and new diabetic drugs and technology.
The curriculum at Mercer University College of Pharmacy and Health Sciences is a 4-year professional modular block program. The first-professional year is primarily basic science courses. Students started 3- to 4-week disease state modules in the fall semester of their second-professional year. All students in the pharmacy program received 18 lecture hours on the pathophysiology, pharmacology, medicinal chemistry, pharmacotherapy, complications, and medical nutrition management of diabetes in the endocrine module during the fall semester of the third-professional year. In addition, all students completed a 3-hour laboratory on glucose monitors and a 2-hour laboratory on insulin administration and foot care.
METHODS
To increase our graduates' ability to provide diabetes disease state management, a diabetes certificate program was developed and implemented at Mercer University College of Pharmacy and Health Sciences. The diabetes care elective was developed by a committee that included an ambulatory care PharmD who is a certified diabetes educator, 2 community practice-based PharmDs, and 1 registered dietitian. The course content was derived from the Mercer University's Diabetes Certificate Program, Grady Health System/Emory University's Modern Methods of Diagnosing and Treating Diabetes Mellitus and Its Complications Mini Residency, and the Core Curriculum for Diabetes Education.28 The 2-credit hour elective course met 2 hours a week in a 16-week semester (Appendix 1).
The requirements for a Diabetes Care Certificate included completion of 3 components: (1) a 32-hour didactic elective in diabetes care in the third-professional year (Appendix 1) (2) an advanced pharmacy practice experience (APPE) at a site that focused on diabetes care; and (3) a written examination and an oral examination with a minimum combined grade of 75%. The written and oral examinations were administered to all students after they completed their advance practice experiences in the fourth-professional year. Not all students who took the diabetes didactic elective completed the certificate program.
The elective was offered for the first time in spring 2004 and the final group was tested in spring 2006. The course had 1 primary coordinator who was a PharmD and a certified diabetes educator. Guest lecturers included a pediatric pharmacist, 2 community pharmacists, a long-term care pharmacist, and a registered dietitian. The course assignments included interviewing a patient with diabetes, taking 4 unannounced quizzes based on reading assignments, completing a personal diet analysis, and participating in a mock patient role-play assignment. During the last 2 weeks of the course, students were assigned to role-play either a patient or a pharmacist. Each mock “patient” was assigned his/her own mock “pharmacist” for a week. Case patients presented a new scenario to the case pharmacist every other day for a week. After 1 week, each pair of students was assigned a new patient and the students switched roles. When acting as the case patient, the student had to check blood glucose, mimic insulin injections with normal saline, and administer candy to simulate medication as described in their case. Case pharmacists were required to write their recommendations for treating/addressing patient problems. The purpose of this assignment was to improve student empathy, increase knowledge of diabetes self-care activities, and review and apply course material.
There were 10 different APPE's that qualified students to receive a diabetes certificate upon completion. Five of the sites were community pharmacies and 5 were ambulatory care practices. The sites were chosen because the pharmacists at these sites had completed a diabetes certificate program or were certified in diabetes education or diabetes management. In addition, the practice sites chosen provided a significant amount of diabetes care.
The control group was recruited from the classes of 2005 and 2006 and given a $50 stipend for participating. Both certificate students and control students were given the examination after they completed all their requirements for graduation, including all their APPEs. Students who were in the diabetes elective but did not complete the certificate program were not eligible for inclusion in the study. Students who had diabetes were also excluded from participation. All subjects signed an informed consent form. The grader of the written examination was blinded to the students' name, and oral examination evaluators (N = 5) did not know whether the student was a control or a certificate student. This study was approved by the Mercer University Institutional Review Board and all subjects signed an informed consent.
The oral examination was divided into 2 domain skill sets: skill demonstration and patient counseling. The examination was written by a panel of health care professionals, which included an ambulatory care PharmD who is a certified diabetes educator, 2 community practice-based PharmDs, and 1 registered dietitian. The examination content was based on information from the required curriculum, Diabetes Care Elective, and the clinical experience of the examination authors. The oral examination was structured according to suggestions made by Davis and Karunathilake29 including: structuring the examination on clinical scenarios, asking all the candidates the same questions, and using a grading rubric. To maximize interrater reliability, examiners were trained and the test questions assessed objective skills. It was not possible to formally assess interrater reliability because 2 of the faculty members who were examiners resigned between the first and second year of the study.
The primary outcome measured was the difference between examination grades. A sample size of 26 subjects was needed to detect a 10% examination score difference with an 80% power and a two-sided level of significance of 5%, assuming a 9% standard deviation in examination scores. The assumption regarding standard deviation was based on the fact that in a previous study of the effect of a diabetes certificate program on pharmacists' examination performance the standard deviation of examination score was 8.33%.19 Differences between baseline categorical variables were assess with either a Fisher's exact test or χ2 test. The relationships between examination performance and age, sex, race, and grade point average (GPA) were assessed through multiple linear regression and a Pearson correlation coefficient.
RESULTS
Between 2005 and 2006, 25 graduates were awarded a certificate in diabetes care. The course evaluations for the diabetes didactic elective are listed in Table 1. In 2004, there were 30 students enrolled in the elective and 11 completed evaluations. In 2005, 26 of the 29 students enrolled completed evaluations. Over the 2 years, there were a total of 30 written comments. The most common written comments (submitted by 76% of the respondents) were complaints regarding the large volume of required reading material.
Table 1.
PharmD Students' Responses on a Diabetes Care Course Evaluation*
Responses based on a 1-5 scale on which 5 = “strongly agree”
The demographic information for the 37 students who received certificates and the control students is listed in Table 2. There were no significant differences in any of the baseline demographics. The certificate program had a significant impact on the combined grade for the oral and written examinations (p = 0.01) after adjustments for age, race, sex, and GPA (Figure 1). Students who earned a certificate scored significantly higher in the counseling section of the oral examination (p = 0.01), but the results of the demonstration section of the oral examination were not statistically significantly different (p = 0.92) (Figure 2). After adjusting for age, race, sex, and GPA (p = 0.002) the certificate program (p = 0.0007) had a significant impact on the grades on the written section of the examination. However, age, race, and sex did not affect performance on any part of the examination. The course evaluation results are listed in Table 2.
Table 2.
Study Student Demographics (N = 37)
Figure 1.
Student Performance on Post-APPE Examination.
Figure 2.
Student Performance on Oral Examination by Section.
DISCUSSION
This study demonstrates that participation in a diabetes certificate program enhanced the diabetes management knowledge and counseling skills of PharmD students. The students' combined mean examination score was 11.3% higher than that of students in the control group (p < 0.05) (Figure 1). Although objective data were not collected, anecdotal reports from preceptors indicate that students who participated in the program had better diabetes management knowledge than students who did not participate. The superior examination scores of the certificate students are consistent with previous data from previous studies that showed that student knowledge and or skills improve after participation in a pregraduation certificate program.26,27
According to the course evaluations, the didactic elective was a positive learning experience. In the written comments, many students stated that the course was a positive learning experience. The primary complaint regarding the diabetes elective was the amount of outside reading required. This finding is consistent with the fact that the students' assessment of the required material received the lowest score of the 7 points evaluated. The average overall score was 3.9 on a 5- point scale, which is slightly higher than our institutional average of 3.77. These results are consistent with the reports found in the literature. The average evaluation score of the diabetes course in the Johnson et al report was 3.0 on a 4-point scale, and the Ogedard et al diabetes course evaluation scores ranged from 3.27 to 3.94 on a 5-point scale. 26,27
The only variable that related to examination performance was GPA. This finding is consistent with the results of the other findings that demonstrated that GPA is a predictor for future examination performance.30,31 However, the mean GPA of students who participated in the course was higher than that of the students in the control group. The students who decided to pursue certificates may have been better students in general. The other diabetes certificate programs did not report participant GPA; therefore, it is not possible to determine whether our findings were consistent with other certificate programs.
There are several possible explanations for why there is no statistical difference between the 2 groups' performance on demonstration skills. Perhaps, the interrater reliability was low, but the objective nature of the oral test questions makes this unlikely. Another explanation is that the College's standard curriculum is sufficient for teaching skills and the control group was able to perform proficiently. Or perhaps our certificate program did not emphasize the skills of insulin teaching and glucose meter training as much as we thought it did. The fact that the average score on the demonstration section was about 76% for both groups suggests that more emphasis on these skills during the students' education may be necessary.
Including more subjects would also improve the quality of this project. There were sufficient numbers of subjects included to detect at least a 10% difference in examination grades. The only examination grades that were not statistically different were the overall oral grade and the grade on the demonstration section of the oral examination. There was a 1% difference between the 2 groups in the overall oral grade, and the examination grades for the demonstration section were equivalent. It is unlikely that a 1% difference in grades on this examination would indicate a significant difference in students' true performance ability. It is unlikely that a type II error was made; therefore, including more subjects probably would not have changed the results.
Limitations
Although there was a control group and examiners were blinded, there were several limitations to this study. This study was not randomized, which could have meant there were different levels of interest in or aptitude for the subject of diabetes. It is not possible to know whether the subjects in the control group were motivated to perform their personal best. Control students were not rewarded for examination performance, but certificate students knew they had to score a minimum of 75% in order to receive their certificate.
Randomization of all students who were interested in pursing a diabetes program would have eliminated these problems; however, it was not practical. In order for this study to have been randomized, subjects would have had to have been placed in a certificate group or a control group during their third-professional year of pharmacy school. This would have required that some students complete a diabetes certificate program even if they had not wanted to, while possibly excluding other students who wanted to pursue a certificate. These restrictions may have also made recruitment difficult because students may not have wanted to agree to such limitations. Also, it may have been unethical for faculty members to request or recruit students to participate in a study with such restrictions. Faculty members have to consider these types of ethical dilemmas when conducting pedagogical research.32
In a study of this nature, there is always the possibility that others may question the validity and reliability of the examination. The examination used in this study was compiled with a registered dietitian and several pharmacists from different practice settings, one of whom is a certified diabetes educator. However, using a validated examination would have improved the legitimacy of the results. In addition, the use of oral examinations has been criticized.33,34 The major concerns are the examination format and low interrater reliability of the examiners. There is a large margin of error in oral examinations when an examiner makes up their own questions, board topics are tested, and/or a grading rubric is not used.33,34 In this type of oral examination format the examination questions may differ between students, each student's responses can be vastly different, and the examiners interpretation of the answers may vary. These problems can result in inaccurate assessment of student abilities.35,36 To avoid these problems, the questions in this study's oral examination were structured on specific clinical scenarios and a grading rubric was used. The strengths of oral examinations are that they allow the examiner to assess the student's ability to verbally communicate their knowledge, or demonstrate tasks or patient counseling skills. Despite their limitations, an oral examination based on clinical scenarios seemed the most practical way for determining a student's proficiency in counseling and patient care skills.
This study only examined the student's knowledge of diabetes and education skills. Therefore, the results do not predict the amount or quality of services the students will provide once they become pharmacists. However, it is reasonable to assume that pharmacists can not provide quality diabetes care if they do not have the knowledge and skills.
Additional research is needed to determine whether diabetes certificate programs are feasible in other colleges and schools of pharmacy. Other diabetes programs may also be structured differently and their effectiveness should also be documented. This project only examined the effect on students' test scores. It is important to determine whether students who earn a diabetes certificate use the skills they learned once they are practicing pharmacists. The services certificate holders provide probably will depend on the additional training the provider received (eg, a residency), the provider's work setting (ie, independent, chain, hospital, or ambulatory care), and whether they are reimbursed for these services. The question of how this program and these examination scores affect patient outcomes like glycemic control, quality of life, and rate of complications should also be researched.
CONCLUSION
Certificate programs in diabetes care completed before graduation are feasible and improve general knowledge in diabetes disease state management. Although the program did not enhance students' performance in insulin administration or use of a glucose meter, it did improve their diabetes knowledge and ability to counsel patients on diet, foot care, and uncontrolled blood glucose.
ACKNOWLEDGEMENTS
We would like to thank the following faculty for their valuable input into the publication of this manuscript: Grady Strom, PhD, Candace Barnett, PharmD, and Richard Jackson, PhD.
Appendix 1. Diabetes Care Elective Topic Outline
(1) Review of data – DCCT & UKPDS
(2) Social Issues in Patients with Diabetes
(3) Nutrition in Diabetes
(4) Exercise & Diabetes Potlock
(5) Special Populations – pregnancy, geriatrics, pediatrics
(6) Diabetes Self Care Products
(7) Insulin pump presentation
(8) Ongoing research, drugs and devices in development
(9) Foot Care – shoes and wound care
(10) Community pharmacy diabetes program management
(11) Review of patient education materials – print and videos
(12) Presentation of patient interviews
(13) Diabetes Role Play Presentation Session I
(14) Diabetes Role Play Presentation Session II
REFERENCES
- 1.Boyle J, Honeycutt A, Venkat-Narayan K, et al. Projection of diabetes burden through 2050. Diabetes Care. 2001;24:1936–40. doi: 10.2337/diacare.24.11.1936. [DOI] [PubMed] [Google Scholar]
- 2.Diabetes Complication and Control Research Trial Group. The effect of intensive treatment of diabets on development and progression of long-term complications in insulin dependent diabetes mellitus. N Engl J Med. 1993;329:977–86. doi: 10.1056/NEJM199309303291401. [DOI] [PubMed] [Google Scholar]
- 3.UK Prospective Diabetes Study (UKPDS) Group. Effect of intensive blood-glucose control with metformin on complications in overweight patients with type 2 diabetes (UKPDS 34) Lancet. 1998;352:854–65. [PubMed] [Google Scholar]
- 4.Koro C, Bowlin S, Bourgeois N. Glycemic control from 1988 to 2000 among U.S. adults diagnosed with type 2 diabetes: a preliminary report. Diabetes Care. 2004;27:17–20. doi: 10.2337/diacare.27.1.17. [DOI] [PubMed] [Google Scholar]
- 5.Cranor CW, Bunting BA, Christensen DB. The Asheville Project: long-term clinical and economic outcomes of a community pharmacy diabetes care program. J Am Pharm Assoc. 2003;43:173–84. doi: 10.1331/108658003321480713. [DOI] [PubMed] [Google Scholar]
- 6.Cranor CW, Christensen DB. The Asheville Project: factors associated with outcomes of a community pharmacy diabetes care program. J Am Pharm Assoc. 2003;43:160–72. doi: 10.1331/108658003321480704. [DOI] [PubMed] [Google Scholar]
- 7.Garrett DG, Martin LA. The Asheville Project: participants' perceptions of factors contributing to the success of a patient self-management diabetes program. J Am Pharm Assoc. 2003;43:185–90. doi: 10.1331/108658003321480722. [DOI] [PubMed] [Google Scholar]
- 8.Kiel PJ, McCord AD. Collaborative practice agreement for diabetes management. Am J Health-Syst Pharm. 2006;63:209–10. doi: 10.2146/ajhp050362. [DOI] [PubMed] [Google Scholar]
- 9.Lamer CC. Utilizing pharmacists in collaborative diabetes disease state management. Dis Manage. 2003;6:189–90. doi: 10.1089/109350703322425536. [DOI] [PubMed] [Google Scholar]
- 10.Leal S, Glover JJ, Herrier RN, Felix A. Improving quality of care in diabetes through a comprehensive pharmacist-based disease management program. Diabetes Care. 2004;27:2983–4. doi: 10.2337/diacare.27.12.2983. [DOI] [PubMed] [Google Scholar]
- 11.Leung WY, So WY, Tong PC, Chan NN, Chan JC. Effects of structured care by a pharmacist-diabetes specialist team in patients with type 2 diabetic nephropathy. Am J Med. 2005;118:1414. doi: 10.1016/j.amjmed.2005.07.050. [DOI] [PubMed] [Google Scholar]
- 12.Lindenmeyer A, Hearnshaw H, Vermeire E, Van Royen P, Wens J, Biot Y. Interventions to improve adherence to medication in people with type 2 diabetes mellitus: a review of the literature on the role of pharmacists. J Clin Pharm Ther. 2006;31:409–19. doi: 10.1111/j.1365-2710.2006.00759.x. [DOI] [PubMed] [Google Scholar]
- 13.McCord AD. Clinical impact of a pharmacist-managed diabetes mellitus drug therapy management service. Pharmacotherapy. 2006;26:248–53. doi: 10.1592/phco.26.2.248. [DOI] [PubMed] [Google Scholar]
- 14.Mikhail N, Cope D. Primary care-based disease management for patients with diabetes. Am J Med. 2005;118:1444–5. doi: 10.1016/j.amjmed.2005.04.038. [DOI] [PubMed] [Google Scholar]
- 15.Morello CM, Zadvorny EB, Cording MA, Suemoto RT, Skog J, Harari A. Development and clinical outcomes of pharmacist-managed diabetes care clinics. Am J Health-Syst Pharm. 2006;63:1325–31. doi: 10.2146/ajhp050430. [DOI] [PubMed] [Google Scholar]
- 16.Odegard PS, Goo A, Hummel J, Williams KL, Gray SL. Caring for poorly controlled diabetes mellitus: a randomized pharmacist intervention. Ann Pharmacother. 2005;39:433–40. doi: 10.1345/aph.1E438. [DOI] [PubMed] [Google Scholar]
- 17.Ragucci KR, Fermo JD, Wessell AM, Chumney EC. Effectiveness of pharmacist-administered diabetes mellitus education and management services. Pharmacotherapy. 2005;25:1809–16. doi: 10.1592/phco.2005.25.12.1809. [DOI] [PubMed] [Google Scholar]
- 18.Simoens S, Foulon E, Dethier M, Mathieu C, Laekeman G. Promoting targeted screening for Type 2 diabetes mellitus: the contribution of community pharmacists. Diabetes Med. 2005;22:812–3. doi: 10.1111/j.1464-5491.2005.01508.x. [DOI] [PubMed] [Google Scholar]
- 19.Cerveny J, Knapp R, DelSignore M, Carson D. Experience with objective structured clinical examination as a participation evaluation instrument in disease management. Am J Pharm Educ. 1999;63:377–81. [Google Scholar]
- 20.Kirk J, Joyner P, Lee A, Oles K. Experience and implementation of a diabetes pharmacotherapy certificate program. Am J Pharm Educ. 1998;62:307–10. [Google Scholar]
- 21.Monaghan M, Turner P, Skrabal M, Jones R. Evaluating the format and effectiveness of a disease state management training program for diabetes. Am J Pharm Educ. 2000;64:181–4. [Google Scholar]
- 22.Plake K, Chesnut R, Biebighauser S. Impact of a diabetes certificate program on pharmacists' diabetes care activities. Am J Pharm Educ. 2003;67:1–8. [Google Scholar]
- 23.Schneider E, Gardner S, Johnson J. Development of a practical examination utilizing standardized participants for disease state management credentialing. Am J Pharm Educ. 2000;64:173–6. [Google Scholar]
- 24.Patterson B. Distance education in a rural state: assessing change in pharmacy practice as a result of a pharmaceutical care certificate program. Am J Pharm Educ. 1999;63:56–63. [Google Scholar]
- 25.Hak E, Foster S, McColl M, Bradberry J. Evaluation of student performance in an immunization continuing education certificate program incorporated in a pharmacy curriculum. Am J Pharm Educ. 2000;64:184–7. [Google Scholar]
- 26.Johnson J, Chesnut R, Tice B. An advanced diabetes care course as a component of a diabetes concentration. Am J Pharm Educ. 2003;67:160–70. [Google Scholar]
- 27.Ogedard P, LaVigne L, Ellsworth A. A diabetes education program for pharmacy students. Am J Pharm Educ. 2002;66:391–5. [Google Scholar]
- 28.Franz M. A Core Curriculum for Diabetes Educators. Chicago, Ill: American Association of Diabetes Educators; 2003. [Google Scholar]
- 29.Davis MH, Karunathilake I. The place of the oral examination in today's assessment systems. Med Teacher. 2005;27:294–7. doi: 10.1080/01421590500126437. [DOI] [PubMed] [Google Scholar]
- 30.Houglum J, Aparasu R, Delfinis T. Predictors of academic success and failure in a pharmacy professional program. Am J Pharm Educ. 2005;69 Article 43. [Google Scholar]
- 31.Sansgiry S, Bhosle M, Sail K. Factors that affect academic performance among pharmacy students. Am J Pharm Educ. 2006;70 doi: 10.5688/aj7005104. Article 104. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Fergerson L, Yonge O, Myrick F. Students' involvement in faculty research: Ethical and methodological issues. Int J Qualitative Methods. 2004;3:1–14. [Google Scholar]
- 33.Kehm B. Oral examinations at German Universities. Assess Educ. 2001;8:25–31. [Google Scholar]
- 34.Weingarten MA, Polliack MR, Tabenkin H, Kahan E. Variations among examiners in family medicine residency board oral examinations. Med Educ. 2000;34:13–7. doi: 10.1046/j.1365-2923.2000.00408.x. [DOI] [PubMed] [Google Scholar]
- 35.Esmail A, May C. Commentary: Oral exams- get them right or don't bother. BMJ. 2000;320:375. [PubMed] [Google Scholar]
- 36.Roberts C, Sarangi S, Southgate L, Wakeford R, Wass V. Oral examinations- equal opportunities, ethnicity and fairness in the MRCGP. BMJ. 2000;320:370–5. doi: 10.1136/bmj.320.7231.370. [DOI] [PMC free article] [PubMed] [Google Scholar]