Quality assurance of medical education in the Netherlands: programme or systems accreditation?

Abstract

Accreditation is an instrument that is used worldwide to monitor, maintain and improve the quality of medical education. International standards have been defined to be used in reviewing and evaluating the quality of education. The organization and the process of accreditation of medical education programmes in the Netherlands and in Flanders are described in some detail. Accreditation can be based on the results of a detailed assessment of an educational programme or on an evaluation of the educational system and the organization of the institution in question. The Flemish-Dutch accreditation organization (NAO) is moving from programme accreditation towards a combination of programme and systems accreditation. The pros and cons of these two approaches are discussed.

Introduction

Since the signing of the Bologna Declaration, accreditation of medical education has been used to promote the accountability of institutions for the quality of medical education and to assure the quality of medical education in Europe. This paper gives an overview of international developments and describes the process leading up to the accreditation of medical schools, with emphasis on medical education in the Netherlands.

After describing the international context of the accreditation of medical education programmes, I will present two important approaches to accreditation: programme and systems accreditation. The main focus is on accreditation in Flanders and in the Netherlands. In the final discussion the advantages of a combination of programme and systems analysis are emphasized.

International developments

After the publication of Abraham Flexner’s critical report on medical education in the United States and Canada in 1910 and following his later comparative study of European universities it became patently clear that there was an urgent need for improvement of the quality of medical education worldwide [1], [2]. Medical associations acted in accordance with their professional and social responsibilities and developed standards for educational quality and systems to monitor and assure the quality of medical education. This has resulted in an accreditation system that has been defined as ‘a process by which a designated authority reviews and evaluates an educational institution using a set of clearly defined criteria and procedures ’ [3].

Since 1942 the USA and Canada have had a tradition of non-governmental, independent accreditation procedures performed by the Liaison Committee on Medical Education (LCME) [4]. In the countries of central Europe, accreditation gained momentum after the signing of the declaration on higher education in Europe in Bologna in 1999. In response to the internationalization and globalization of medicine, the World Health Organization (WHO) and the World Federation for Medical Education (WFME) jointly developed a programme to promote global standards for medical education and the accreditation of medical education programmes [5]. Recent years have seen the gradual introduction and acceptance of accreditation of medical education in European and American countries. Under the umbrella of WHO, accreditation is gaining ground in other continents as well. Currently, over seventy countries worldwide have systems in place for the accreditation of medical education [3].

Standard and guidelines

LCME, WHO/WFME and the European Association for Quality Assurance in Higher Education (ENQA) all have formulated guidelines for standard setting and quality assurance. Fortunately, there is a remarkable degree of congruence and consistency among the existing guidelines [5], [6], [7]. What they have in common is: the recommendation to establish a national independent organization that is responsible for accreditation within a strong legal framework, the recommendation to formulate national quality standards, consistent with international guidelines, and the recommendation to design a three-step evaluation procedure: self evaluation, external evaluation, including a site visit by experts, and the final formal decision about accreditation.

Types of accreditation

The present accreditation systems vary significantly in purpose, institutional design, procedures, ownership and practices. Depending on national preferences, the principal accent tends to be either on accreditation of individual programmes or on systems accreditation.

Programme accreditation targets the content and outcomes of a particular programme. The programme is judged by national and international quality standards and the professional requirements of the academic field in question.

Systems or institutional accreditation targets the organization, the processes and procedures within an educational institution. Review and external evaluation examine whether the institution meets quality standards with regard to faculty, administration, curriculum, service, infrastructure and financial aspects.

Some national accreditation programmes like the one in Switzerland focus primarily on systems or institutional assessment. But many other countries have opted for programmatic evaluation as the basis for decisions on accreditation. Generally, independent organizations are entrusted with the accreditation of educational programmes. The spectrum ranges, however, from non-governmental accreditation bodies, such as LCME, set up by national medical associations, to accreditation organizations that are mandated by national governments and directly regulated by public legislation. Accreditation systems also vary in the balance they seek between accreditation as an instrument for public accountability and accreditation as a stimulus for quality improvement.

Within Europe there is a tendency towards internationalization of quality assurance with the realization of an effective European Higher Education Area in which the participating countries mutually recognize their accredited national programmes. This has resulted in a list of reliable quality assurance agencies: the European Register of Quality Assurance Agencies and the launch of Qrossroads, a website to help students find quality assured and accredited educational institutions within Europe [7].

NVAO, the Accreditation Organization of the Netherlands and Flanders

Before the Bologna Declaration, the Netherlands had already developed a system to assure the quality of medical education. Independent external assessment was required under the Higher Education and Research Act (WHW 1992). Initially, the assessment process was organized by the Association of Universities in the Netherlands (VSNU) and since 1991 the programmes of the eight Dutch faculties of medicine have been subject to external assessment every six years. Following the Bologna Declaration the governments of the Netherlands and Flanders in 2002 adapted the WHW to a system of bachelor-master programmes and accreditation. In order to facilitate the latter, in 2003 the Dutch and Flemish ministers of education jointly founded a transnational organization for the accreditation of higher education: the Dutch-Flemish Accreditation Organization (NVAO) (in Dutch: Nederlands-Vlaamse AccreditatieOrganisatie).

NVAO is the only organization that is legally authorized to accredit professional and academic higher education programmes both in Flanders and in the Netherlands. Although the accreditation processes used in Flanders and the Netherlands are largely identical, there are some discrepancies, reflecting differences in culture and tradition. For instance the accreditation period in Flanders is eight years and in the Netherlands it is six years [8].

I will restrict the scope of this paper to the limits of my expertise and focus on the system that is used in the Netherlands to accredit academic medical education programmes.

Accreditation of medical education programmes in the Netherlands

Accreditation of bachelor and master programmes in medicine takes place within the NVAO framework, which is also applied to other academic disciplines. In this framework the procedures, the quality standards and the assessment rules of the accreditation process are described in considerable detail [9]. As with most accreditation procedures the system consists of three consecutive steps: self evaluation, external assessment and formal accreditation.

Self evaluation

The first step is self evaluation. Universities and medical faculties critically appraise their own education programmes. In order to standardize the procedure NVAO has defined a set of six themes and 21 standards. The six themes are:

1. Aims and objectives of the programme

2. The content of the curriculum

3. The faculty and staff

4. The facilities

5. The internal quality assurance

6. The results.

Each theme has corresponding standards and each standard is based on corresponding criteria. Here I only present some examples of how the process works in practice. Self evaluation in relation to theme 1, aims and objectives, should examine whether or not the programme meets the quality requirements set by the national and international professional community in the field of medical education. For Dutch medical programmes this implies compliance with the European Directives [10] and with the Dutch Blueprint for undergraduate medical education [11]. Consequently, the learning outcomes have to correspond with the Dublin Descriptors for bachelor and master programmes and meet requirements in terms of credits (ECTS): for both programmes students have to gain 180 credits over a period of three years. Interaction between education and research is another important benchmark for academic education programmes.

Curriculum content, theme 2, requires consistency with the aims and objectives of the programme. An important issue is whether the curriculum actually enables students to achieve the intended competencies within the set timeframe. Another aspect of content is the usage of a variety of modern learning methods. The programme should be competency-based and there has to be an effective ratio of formal teaching and self study time for students. The benchmark for medical education is 30:70. Also the curriculum should offer flexibility to enable students to make choices in the programme to meet their personal interest and talents. Normally 20% of Dutch medical education programmes is dedicated to elective programmes, such as elective clerkships, research practicals or minor programmes in the humanities.

Self evaluation in relation to theme 3, faculty and staff, focuses on the number of teachers required to effectively run and deliver the programme. A benchmark for the ratio students:teaching staff is 16:1. In order to ensure good education, teaching should primarily be a task for active medical researchers and clinicians. A faculty development programme and regular and structural assessment of the quality of teachers and the teaching programme are essential to satisfy the conditions relevant to theme 3.

Self-reflection on theme 4, facilities, requires consideration of the question whether students have adequate access to an electronic learning environment, a library and a study landscape with digital study facilities (one personal computer available per ten students), laboratories and a skills lab.

Theme 5 examines the institution’s system of internal quality assurance. Is a formal evaluation cycle embedded in the institution’s routines and is there a programme committee consisting of staff and students that is in charge of evaluating, maintaining and improving the quality of the programme? Another aspect of theme 5 is quality assurance of the examinations. Institutions for higher education are bound by law to install an examination committee consisting of students and staff members, which is responsible for the quality and validity of the examinations.

Results and study outcomes, the issues of theme 6, require figures demonstrating that students progress effectively through the programme. Final success rates of 80% for bachelor and master programmes are considered the standard target in medical education.

External assessment

After the self evaluation, the second step on the way towards accreditation is the external assessment organized by an assessment agency. The NVAO is required by law to annually draw up a list of registered and qualified assessment agencies. The external assessments and site visits of the undergraduate medical programmes in the Netherlands are conducted by Quality Assurance Netherlands Universities (QANU), which nominates and appoints a panel of independent experts in the field. A typical panel has five to six members with expertise in medical education, research and practice and in the management of medical institutions. The panel is completed by two to three student members and a coordinating secretary on behalf of QANU. The panel analyzes and assesses the self-evaluation report. The key component of the external assessment is a site visit of two to three days, during which the panel checks and evaluates the accuracy of the self evaluation report by discussions with and inquiries from students, academic leaders, management, staff and alumni. The external assessment focuses on learning outcomes and the expert panel therefore reads and studies essays, assignments and theses produced by students. The panel also visits the facilities, such as the skills lab and the library.

Based on their analyses and findings the panel pronounces their expert judgement of the qualities of the programme and reports its assessment for each of the six themes and the 21 underlying standards of the NVAO framework as ‘satisfactory’ or ‘unsatisfactory’. Additionally, the panel can indicate exceptionally high quality of a specific feature of the programme and mark this as ‘good practice’. Each theme must be judged as satisfactory in order to warrant a positive final assessment. Part of the expert assessment is based on comparisons with other national and international medical programmes. The panel and the quality assurance agency publish the methodology they have applied, the evidence they have gathered and the arguments to substantiate their conclusions in an assessment report.

Formal accreditation

The final step is the formal accreditation by NVAO. The institution applying for accreditation submits the assessment report, which is then evaluated by NVAO based on the criteria of the accreditation framework. Within three months after the assessment process is completed the final decision must be published. If it is positive, accreditation is granted for a period of six years. The accreditation and the assessment report are published on the NVAO website.

Consequences of accreditation

Accreditation has important consequences. Accredited programmes are registered in the Central Register of Higher Education (CROHO), which entitles the institution in question to award degrees recognized by the Dutch and other European authorities. Only accredited programmes can receive public funding and students enrolled in these programmes are eligible for government study grants. The eight undergraduate medical programmes delivered by Dutch universities are all accredited, which means they have the exclusive right to educate and train future medical doctors in the Netherlands.

Programme or systems?

The ultimate goal of accreditation of medical education programmes is good patient care. But to the best of my knowledge there is no sound evidence to show that doctors educated in accredited programmes do better than doctors graduating from programmes without accreditation. Students from LCME accredited schools in the United States perform better on board certification exams, but the relation between exam scores and quality of care remains to be established [12], [13].

Of course many more factors besides the quality of medical education contribute to good clinical practice. This leaves us with surrogate markers, as is customary in other fields of medicine.

Possible surrogate markers for the quality of medical education could be derived from assessment of educational systems, institutional audit, or assessment of the content and outcomes of learning programmes.

In systems evaluation, as in industrial quality control, the results of an assessment of procedures, processes and management could provide such a surrogate marker of quality. There is a good chance that sound procedures and processes can increase the chances of delivering good outcomes. Moreover, procedures and processes can be assessed objectively.

Programme assessment focuses on curriculum content and on learning outcomes. Like peer review of papers, it is more complicated and more subjective than systems assessment. Learning outcomes can only be assessed by a panel of international experts in the medical domain. The panel has to spend considerable time studying papers and theses of students and engaging in in-depth discussions with students and alumni to evaluate the achieved levels of learning outcomes and competencies.

The choice between systems and programme evaluation is strongly influenced by the objectives of accreditation. If accreditation is used to promote public accountability for the quality of medical education, the emphasis will be on systems assessment. If accreditation is used as an instrument to improve the quality of medical education, programme evaluation will be more appropriate. So the balance between control and accountability on the one hand and quality enhancement on the other hand determines the preference for systems or programme evaluation.

The legal framework of accreditation in the Netherlands demands programme accreditation. The present accreditation framework, however, is a combination of systems and programme assessment. This means that in order to evaluate a specific programme the educational institution offering it is assessed as well. This appears to be a bureaucratic, time-consuming and costly combination.

Work in progress

NVAO is currently developing a new accreditation system [12], which will be based on institutional audit. The administration, the procedures, the quality of the staff, the services, the facilities and especially the internal quality assurance procedures will be assessed at the aggregation level of the institution. If the result of the institutional audit is positive, programme evaluation can be less extensive and focus on content and learning outcomes, assessed by an international expert panel, appointed by an accreditation agency.

NVAO wants to strengthen quality improvement and diminish the bureaucratic and administrative burden of the new accreditation system. In this way the accreditation system will become more balanced. Public accountability is served by institutional audit, while content and outcome based programme accreditation will further the quality of medical education. Some sort of combination of systems and programme accreditation also seems to be the outcome of the evaluation of accreditation systems in Europe. Countries like England and Switzerland, where the emphasis is on institutional assessment, are moving towards programme assessment, while other countries, like the Netherlands, which tend to favour programme accreditation appear to be moving in the opposite direction of the balance [14].

One important aspect of the accreditation of medical education in the Netherlands deserves preservation in a new system. Since 1991 external quality assessment of the medical programmes of the eight universities has been conducted by the same expert panel in the same year. The outcomes of the assessment and later accreditation rounds are extensively discussed by the deans of the faculties after each round of accreditation. This approach has opened the discussion to an exchange of best practices, defining quality benchmarks and a common strategy for the steady improvement of the quality of medical education [15]. Perhaps this has been the most profitable effect of accreditation of medical education in the Netherlands so far.

The author

H.F.P. Hillen, MD PhD, is professor of Internal Medicine and dean emeritus Faculty Health Medicine and Life Sciences Maastricht, the Netherlands.