Impact of accreditation on Caribbean medical schools’ processes

Abstract

Background

Caribbean graduates contribute significantly to the US healthcare workforce. The accreditation requirements of local governments vary from one Caribbean island to another island. The Educational Commission for Foreign Medical Graduates (ECFMG) requirement that all future applicants be graduates from accredited medical schools drove Caribbean medical schools to seek accreditation. Accreditation has been found to significantly impact the educational processes of Canadian medical schools. Our study aims at investigating Caribbean medical school leaders’ perceptions of the impact of accreditation on their school’s processes.

Methods

This qualitative study and data analysis were done using a framework analysis. Academic leaders and faculty members from three different types of Caribbean medical schools (accredited, denied-accreditation schools, never applied for accreditation) were interviewed using semi-structured interviews.

Results

A total of 12 participants from six different Caribbean medical schools participated in the interview process. Themes of processes influenced by accreditation at Caribbean medical schools were similar to those found in the Canadian context and align with best practices of Continuous Quality Improvement (CQI).

Conclusions

Caribbean medical schools are changing their educational processes as a result of accreditation requirements. Some processes are not maintained in a continuous manner, raising questions about the development of a true CQI culture.

Introduction

There are 82 medical schools in the Caribbean; 47 are offshore, defined by the World Bank (2004) “as schools catering primarily to foreign (North American) students, wishing to practice medicine in the U.S. and Canada.” Of these 82 schools, 11 are accredited by the Caribbean Accreditation Authority for Education in Medicine and Other Health Professions (CAAM-HP);¹ two of those are on probation. CAAM-HP withdrew accreditation from another four schools. Six schools were denied accreditation, and four are pending review. Eight schools are accredited by the Accreditation Commission on Colleges of Medicine (ACCM).² Two schools are accredited by both agencies independently. More than 50% of Caribbean schools’ graduates are involved in primary care in the USA,³ and more than 75% of U.S.-born International Medical Graduates (IMGs) are from offshore Caribbean schools.⁴

Caribbean schools vary in their education programs.⁵ Students’ performance on United States Medical Licensing Exams (USMLE) and Educational Commission for Foreign Medical Graduates (ECFMG) certification rates differ among schools and among Caribbean countries.⁶ Some Caribbean countries have regulations or national accrediting bodies, while others do not require accreditation.⁷

The ECFMG mandated that by 2023, applicants to its examination be graduates from schools accredited by agencies recognized by the World Federation for Medical Education (WFME).⁸ The need of a large number of their graduates to apply for ECFMG certification drove Caribbean medical schools to seek accreditation.

Accreditation significantly impacts educational processes at Canadian medical schools.⁹ Caribbean education contexts differ from Canadian ones. Our study aims at investigating Caribbean medical school leaders’ perceptions of the impact of accreditation at their medical school.

Methods

This qualitative study used reflective experiences as an inquiry strategy to explore individual experiences with accreditation and elicit a broad perspective on its impact; we did not seek to contrast perceptions between participants or between schools. The Research and Ethics committee of Avalon University School of Medicine approved this research study (approval number P0002-18/08/02).

Using a purposive sampling,¹⁰ we invited 35 deans, associate deans, assistant deans, and faculty members from 25 Caribbean schools under the jurisdiction of CAAM-HP or ACCM and who participated in an accreditation workshop offered by CAAM-HP. These individuals were chosen for their interest in accreditation. Selected individuals were from schools that were either accredited by ACCM (n = 13) or the CAAM-HP (n = 20), denied accreditation (n = 5), or that had never applied for accreditation (n = 3). CAAM-HP and ACCM are both part of the 23 WFME-recognized accreditation agencies.¹¹ Participants consented to participate in semi-structured interviews to gather their reflections about accreditation.¹² Although predetermined questions helped participants express and interpret their experiences, they were also allowed to freely express their thoughts and experiences.^13,14 The interview schedule (Appendix A) included 19 yes/no questions and 16 open-ended questions.

We analysed data using a working analytical framework initially derived from coding the first two interviews; this emergent coding was complemented by themes from Blouin’s study.⁹ Framework analysis was used as this study has specific questions, a limited time frame, a pre-designed participant sample, and a priori issues derived from a previous study.^9,10 Framework analysis allows researchers the flexibility of either gathering all data before analysis or performing data analysis while still in the data collection process.¹⁰ The framework analysis involved piloting the interview protocol, transcribing and coding interview data, developing a thematic framework, charting data into the framework matrix, and interpreting data.¹⁵ Appendix B provides the position and school accreditation status of participants.

Results

Perceptions of the academic community regarding ECFMG’s announcement

All participants supported the ECFMG’s requirement, citing the growing number of Caribbean schools and the lack of uniform accreditation requirements across Caribbean countries. Participants commented:

It is not just a handful of schools [that] improve, but everyone is forced to raise their quality management (Participant 6).

Low-tier schools are the reason for accreditation. They are diploma-producing machines. They do not have a good curriculum and just producing diplomas (Participant 11).

Table 1.

Participants’ responses to the yes/no close-ended questions

Questions	Yes	No	No opinion/ Not Known
Governance
Did your institution make any changes in the by-laws of the board to meet the accreditation requirement? (Question no.7 CEQ 1)	8 (67%)	3 (25%)	1 (8%)
Documentation (Data collection and analysis)
Is there any self-study committee in place at your institution? (Question no.8 CEQ 2)	11 (92%)	1 (8%)
Did you make any changes in the data collection process to meet the requirements of accreditation standards? (Question no.12 CEQ 3)	7 (58%)	4 (33%)	1 (8%)
Creation and revision of policies and procedures
Did you create any new policies as a requirement of accreditation? (Question no.13 CEQ 4)	11 (92%)		1 (8%)
Did you make any changes in the policies and procedures to meet the standards of accreditation? (Question no.14 CEQ 5)	12 (100%)
Are there any changes in the admission policies and procedures recently for accreditation purposes? (Question no.15 CEQ 6)	9 (75%)	3 (25%)
Are there any changes in the students’ promotion policies and procedures lately? (Question no.16 CEQ 7)	10 (83%)	2 (17%)
Did you make any changes in students’ technical standards or emergency plans or strategic planning to meet the accreditation standards? (Question no. 17 CEQ 8)	9 (75%)	3 (25%)
Continuous quality improvement
Do you have an established CQI committee at your institution? (Question no. 18 CEQ 9)	6 (50%)	6 (50%)
Did you make any changes in the strategic planning or CQI committee as CAAM-HP included both of them in their revised standards in 2017? (Question no.19 CEQ 10)	6 (50%)	4 (33%)	2 (17%)
Is continuous quality improvement established in such a way to set up the culture of continuous improvement or to meet the requirements of the accreditation body? (Question no. 21 OEQ 11)	3 said only for accreditation (25%)		9 (for both) (75%)
Curricular reforms
Did you make any changes to the curricular model recently, either partially or complete curricular reform? (Question no. 23 CEQ 11)	12 (100%)
Did you make any changes to the curriculum as suggested by the accreditation site visit report or as a requirement of accreditation standards? (Question no. 25 CEQ 12)	9 (75%)	2 (17%)	1 (8%)
Did you include any curriculum pieces like humanities, alternative medicine, social sciences or medical jurisprudence as a requirement of accreditation? (Question no. 26 CEQ 13)	12 (100%)
Faculty Engagement
Are faculties involved in the governance of the medical school? (Question no. 27 CEQ 14)	10 (83%)	2 (17%)
Do the faculties feel that they are engaged in the educational program or are accountable for the program? (Question no. 28 CEQ 15)	10 (83%)	2 (17%)
Do you see any drastic faculty attrition after the accreditation site visit? (Question no. 30 CEQ 16)	2 (17%)	9 (75%)	1 (8%)
Does the negative accreditation report affect the morale of the faculty members? (Question no. 31 CEQ 17)	8 (67%)	4 (33%)
Does the positive accreditation report encourage and motivate the faculty members? (Question no. 32 CEQ 18)	11 (92%)		1 (8%)
Cost-effectiveness of accreditation site visits
Do you think the benefits outweigh the costs involved? (Question n0. 34 CEQ 19)	11 (92%)		1 (8%)

OEQ- Open-ended question CEQ- Closed-ended question

Governance

Caribbean schools have either a board of trustees or a board of directors. Board of trustees' bylaws were changed to meet accreditation requirements, even at non-accredited schools: “We started adopting bylaws in 2014 when we started applying for CAAM-HP. Overtime we adopted again in 2016” (Participant 4).

Documentation (data collection and analysis)

Most schools mentioned the existence of an institutional self-study committee. The one school without a self-study committee never applied for accreditation. One individual stated: “It is for self-evaluation. If changes are required, that will be implemented. It is not just for accreditation. It is for quality improvement” (Participant 3).

Creation and revision of policies and procedures

Schools created new policies or amended existing ones to meet accreditation requirements. One of the most significant changes reported relative to the admissions process was the inclusion of non-cognitive parameters along with cognitive parameters.

Continuous quality improvement (CQI)

Both accredited and non-accredited schools reported having established a designated CQI committee. Participants perceived that these changes were implemented both to meet accreditation requirements and as part of CQI initiatives. One respondent stated: “Even though it is for continuous quality improvement, the changes were mainly made for accreditation purposes. For our school, mainly for accreditation purposes” (Participant 9).

Curricular reforms

Curricular changes were implemented in recent times at all schools, mostly as a result of suggestions from accreditation site reports or to meet stated accreditation standards. Other times, changes were made as part of program improvement activities. One participant stated:

Curricular changes needed to be done for the best interest of students instead of making changes for novelty and for simply meeting accreditation standards. We need to develop the foundations. Students responding to the situation and critical thinking development are essential (Participant 4).

Not everyone agreed that curricular changes resulted in improvements:

I have seen schools changing the curriculum five-six times after CAAM-HP site visits. […] Now shifting from lecture to competency-based education. Students learn on their own. Give them learning habits and let them learn on their self-directed learning (Participant 4).

Faculty engagement in the educational program

Although difficult to relate directly to accreditation, participants perceived that faculty members were engaged in medical school governance and accountable to the educational program. Only two participants, from a non-accredited school, did not think so. Other Participants commented:

Most definitely. Yes, it affects the administration, students, and as well as the faculty. I mean, it is good that it impacts negatively so they can take positive steps to improve that process (Participant 12).

I think for the faculty it is not that impactful but for the deans and the leadership team, it is impactful. Like we all went through that process. This was great. Our school program was so much better after we went through CAAM. We were very proud of the progress we had made (Participant 6).

Cost-effectiveness

Cost for accreditation site visits varied from 100,000-150,000 U.S. dollars, covering only travel and accommodation expenses for site visit members and administrative fees charged by accreditation bodies; it excluded school administration personnel’s and faculty members' time during the preparatory phase to the site visit (self-study, preparation, and submission of documents, etc.), and the actual visit. This is in contrast to the USA and Canada, where accreditation bodies pay for travel and accommodation expenses of site visit teams. There is a strong feeling among Caribbean schools' leaders that repeated visits to schools and clinical sites can be burdensome. Participants commented:

Once we [get] accredited, it should outweigh [costs]. Some schools even though accredited, they still have a small number of students. If schools have some hundred students on campus, it should outweigh [costs]. Still, need to be reviewed and ongoing site visits and ongoing expenses. Based on recommendations, additional investments are required (Participant 4).

I can tell you, and everybody else does not agree with me. Most people do not believe it that the amount of money that it costs to do over and over and over again is that it leads to instability in the schools. Yeah. And at the same time, I believe 100 percent in the philosophy of accreditation and having ongoing evidence-based quality improvement. To me, this is essential (Participant 6).

The most beneficial effect of accreditation

All respondents felt that accreditation provided an opportunity for external validation of their program and a chance to improve. Participants commented:

I would say it improves, motivate, and desire to attain educational standards and ongoing improvement (Participant 12).

It is a combination of being validated that you are doing the right thing in finding out what you need to do to become what you are (Participant 5).

The number one reason for accreditation is to maintain the standards (Participant 1).

It is taking a look at yourself and dealing with what you see. The accreditation process is always guided by the standard (Participant 7).

Discussion

Our results show that accreditation impacts processes at Caribbean medical schools similarly than at Canadian schools.⁹ Most Caribbean schools in this study implemented a self-study committee and made changes to their data collection process, moving towards regular data collection and monitoring processes, which are central themes of CQI processes.¹⁶ Some schools also have a designated CQI committee. Irrespective of the presence or absence of a designated CQI committee, schools implemented changes either for accreditation purposes or for both accreditation and quality improvement. Almost all schools in this study changed existing policies or created new policies to meet accreditation standards and requirements. A previous study suggested that medical schools and programs respond and change their curricula according to the accreditation process and adverse accreditation reports.¹⁷ Caribbean schools are no exception. Significant reforms are observed with curriculum, admissions policies and procedures, promotion policies, graduation criteria, technical standards for the admissions process, strategic planning, and emergency plans.

Blouin et al’s study was based on the assertion that accreditation drives medical schools to design, implement, and strengthen processes that reflect CQI practices.⁹ The current study validates the concept that accreditation also drives the change process in the Caribbean context. It is disheartening but not unlike what is seen in other contexts that most Caribbean schools in this study appear to maintain a self-study committee only in preparation to and at the time of accreditation visits. Participants from one school also indicated that their school makes changes only for accreditation rather than quality improvement. These decisions hinder quality improvement as an ongoing process. Engagement and involvement of all stakeholders, including faculty members, in the CQI process, is crucial.¹⁶ Our results show that faculty members from both accredited and non-accredited schools are involved in medical school governance and in the educational program and feel accountable for the program. High financial resources are involved in accreditation site visits. Currently, CAAM-HP or ACCM must visit all clinical teaching hospitals affiliated with the medical school being accredited. As Caribbean schools have clinical sites spread across the USA, this can be a problem, especially for small schools that cannot afford the costs involved with visits to all clinical sites.

Conclusions

Despite the high costs associated with accreditation, academic leaders felt absolutely essential to attain accreditation for external validation and better outcomes. Similar to findings from previous studies, accreditation is driving Caribbean medical schools to adopt processes aligned with CQI practices; the culture of continuous improvement as an ongoing process needs further strengthening.

Appendix A. Interview schedule

OEQ- Open-ended question; CEQ- Closed-ended question

1. What are your feelings when you heard for the first time that ECFMG requires accreditation of medical schools by an accreditation body which WFME recognizes? (OEQ1)

2. What is your opinion in general regarding the educational standards in the Caribbean medical schools? (OEQ 2)

3. What are your experiences or perceptions, or feelings regarding the accreditation in the Caribbean region? (OEQ 3)

4. What are the governing arrangements that are in place at your medical university? (OEQ 4)

5. Who oversights the overall operations of medical school? Is this the board of trustees or any other particular board of governance? (OEQ 5)

6. What is the role of the board of trustees? Are they employed by the university or medical school or do they act as consultants, or do they have any contractual agreements with the institution? (OEQ 6)

7. Did your institution make any changes in the by-laws of the board to meet the accreditation requirement? (CEQ 1 probed to answer further depending on the response)

8. Is there any self-study committee in place at your institution? (CEQ 2)

9. How do you collect data from each department per se from the curriculum committee, students’ promotions and assessment committee, research committee, and admissions committee? (OEQ 7)

10. How do you use the data collected? Is the data used for accreditation requirements or just for self-evaluation or continuous quality improvement? (OEQ 8)

11. How often do you collect data from your graduates, or you contact your graduates? (OEQ 9)

12. Did you make any changes in the data collection process to meet the requirements of accreditation standards? (CEQ 3)

13. Did you create any new policies as a requirement of accreditation? (CEQ 4 probed to answer further depending on the response)

14. Did you make any changes in the policies and procedures to meet the standards of accreditation? (CEQ 5 probed to answer further depending on the response)

15. Are there any changes in the admission policies and procedures recently for accreditation purposes? (CEQ 6 probed to answer further depending on the response)

16. Are there any changes in the students’ promotion policies and procedures lately? (CEQ 7 probed to answer further depending on the response)

17. Did you make any changes in students' technical standards or emergency plans, or strategic planning to meet the accreditation standards? (CEQ 8 probed to answer further depending on the response)

18. Do you have an established CQI committee at your institution? ((CEQ 9 probed to answer further depending on the response)

19. Did you make any changes in the strategic planning or CQI committee as CAAM-HP included both of them in their revised standards in 2017? (CEQ 10 probed to answer further depending on the response)

20. How is the program evaluated? What are the parameters to evaluate the program? How are they collected, and how are they used? (OEQ 10)

21. Is continuous quality improvement established in such a way to set up the culture of continuous improvement or to meet the requirements of the accreditation body? (OEQ 11)

22. Which model of the curriculum is being used in your medical education program? (OEQ 12)

23. Did you make any changes to the curricular model recently, either partially or complete curricular reform? (CEQ 11)

24. If yes, what changes did you make? (OEQ 13)

25. Did you make any changes to the curriculum as suggested by the accreditation site visit report or as a requirement of accreditation standards? (CEQ 12 probed to answer further depending on the response)

26. Did you include any curriculum pieces like humanities, alternative medicine, social sciences, or medical jurisprudence as a requirement of accreditation? CEQ 13 probed to answer further depending on the response)

27. Are faculties involved in the governance of the medical school (CEQ 14)? If they are involved, how are they involved?

28. Do the faculties feel that they are engaged in the educational program or are accountable for the program? (CEQ 15)

29. Can you speak a little bit about faculty recruitment, evaluation, and promotion policies and procedures at your medical school? (OEQ 14)

30. How is the faculty attrition rate at your institution? Do you see any drastic faculty attrition after the accreditation site visit? (CEQ 16 probed to answer further depending on the response)

31. Does the negative accreditation report affect the morale of the faculty members? (CEQ 17 probed to answer further depending on the response)

32. Does the positive accreditation report encourage and motivate the faculty members? (CEQ 18 probed to answer further depending on the response)

33. How much cost involved in accreditation site visits? (OEQ 15)

34. Do you think the benefits outweigh the costs involved? (CEQ 19 probed to answer further depending on the response)

35. What is the most beneficial part of the accreditation site visit? (OEQ 16)

Appendix B. Positions and school accreditation status of participants

Participant Number	Accreditation status of the medical school	Position	Interview Medium
1	Accredited Code- School A	Associate Dean of Basic Sciences	In person
2	Accredited Code- School A	Faculty member/ Curriculum committee chair	In person
3	Accredited Code- School A	Faculty Member	In person
4	Accreditation denied Code- School B	Dean	In person
5	Accredited Code- School C	Dean and Provost	By phone
6	Accredited Code- School D	Clinical dean	By phone
7	Accredited Code- School D	Executive Dean/Dean	By phone
8	Accreditation denied Code- School E	Director of quality assurance	By phone
9	Accreditation denied Code- School E	Associate Dean	By phone
10	Never applied for accreditation Code- School F	Dean of Academics/Dean	In person
11	Accredited Code- School C	Associate Dean	By phone
12	Accredited Code- School A	Clinical Dean	By phone

Participant 1

‘The number one reason for accreditation is to maintain the standards’

Participant 3

‘It is for self-evaluation. If changes are required, that will be implemented. It is not just for accreditation. It is for quality improvement’

Participant 4

‘We started adopting bylaws in 2014 when we started applying for CAAM-HP. Overtime we adopted again in 2016’

‘Curricular changes needed to be done for the best interest of students instead of making changes for novelty and for simply meeting accreditation standards. We need to develop the foundations. Students responding to the situation and critical thinking development are essential’

‘I have seen schools changing the curriculum five-six times after CAAM-HP site visits. […] Now shifting from lecture to competency-based education. Students learn on their own. Give them learning habits and let them learn on their self-directed learning’

‘Once we [get] accredited, it should outweigh [costs]. Some schools even though accredited, they still have a small number of students. If schools have some hundred students on campus, it should outweigh [costs]. Still, need to be reviewed and ongoing site visits and ongoing expenses. Based on recommendations, additional investments are required’

Participant 5

‘It is not just a handful of schools [that] improve, but everyone is forced to raise their quality management’

‘It is a combination of being validated that you are doing the right thing in finding out what you need to do to become what you are’

Participant 6

‘I think for the faculty it is not that impactful but for the deans and the leadership team, it is impactful. Like we all went through that process. This was great. Our school program was so much better after we went through CAAM. We were very proud of the progress we had made’

‘I can tell you, and everybody else does not agree with me. Most people do not believe it that the amount of money that it costs to do over and over and over again is that it leads to instability in the schools. Yeah. And at the same time, I believe 100 percent in the philosophy of accreditation and having ongoing evidence-based quality improvement. To me, this is essential’

Participant 7

‘It is taking a look at yourself and dealing with what you see. The accreditation process is always guided by the standard’

Participant 9

‘Even though it is for continuous quality improvement, the changes were mainly made for accreditation purposes. For our school, mainly for accreditation purposes’

Participant 11

‘Low-tier schools are the reason for accreditation. They are diploma-producing machines. They do not have a good curriculum and just producing diplomas’

Participant 12

‘Most definitely. Yes, it affects the administration, students, and as well as the faculty. I mean, it is good that it impacts negatively so they can take positive steps to improve that process’

‘I would say it improves, motivate, and desire to attain educational standards and ongoing improvement’