Strengths, Weaknesses, and Suggestions for Improvement in Postgraduate Assessment in Community Medicine in India: A Delphi Study

Abstract

Objectives:

It was to understand the strengths and weaknesses of the current postgraduate assessment system in community medicine in India, to identify recommendations for change, and to build a consensus around them.

Materials and Methods:

A conventional Delphi technique was preferred for consensus building among experts. We completed three Delphi rounds over a period of 4 weeks, and 16 experts participated in the study. Content analysis was done for open-ended responses, and consensus analysis was done for Likert-type scale questionnaire. In round three, we obtained their top five preferences for change in assessment.

Results:

The experts agreed to have an assessment system based on ongoing formative and one end-of-year summative assessment. Apart from this, they agreed on the various occasions for carrying out the formative assessment. Furthermore, they clearly agreed on measures such as blueprinting, improving test formats, and adequate briefing of test-taking students.

Conclusion and Recommendations:

Most of the consensus items were found to be in alignment with the modern assessment theory. Regulating body and policymakers should revise the current postgraduate assessment system in community medicine to enhance its validity and reliability.

INTRODUCTION

The Lancet Commission on the Education of Health Professionals for the 21^st century, highlighted that the present curriculum in health profession education is fragmented and outdated and recommended reform in health profession education and assessment to supply skilled professionals to health-care systems.[1] In India, the Medical Council of India has envisioned reforms in the postgraduate curriculum of all the subjects. In recent times, there has been explicit discussion of improving training and assessment in postgraduate courses in the overall subjects,[2] pharmacology,[3] and Ophthalmology.[4] Nevertheless, the National Medical Commission envisions reform in all postgraduate programs including community medicine.[5]

Over the years, postgraduates in community medicine have been assessed using a traditional assessment framework. The decision about final certification highly relies on the student's performance in a single final university examination. This assessment framework is less defensible as it uses fewer methods which are focused on knowledge assessment and done on one occasion in the course. Although teachers and experts feel the need for a change and improvement in the current assessment practices,[6,7] there is limited systematic effort on development of consensus on what to improve and how to undertake the change? with the group of experts. Hence, the aim of the present study was to understand the strengths and weaknesses of the overall current assessment system in community medicine in India and to build a consensus on recommendations for improvement in the overall assessment.

MATERIALS AND METHODS

Research design and assumptions

The present study was based on the constructivist paradigm of knowledge building in research. Being a complex issue, assessment reform requires discussion and accommodation of different views.[8] A conventional Delphi technique was preferred for the present study as it allows iterative data collection for exploration and consensus building from a panel of experts who are geographically scattered.[9] The Delphi method assumes that group opinions are more valid than individual judgments.[10]

Sample size, sampling, and access to sample

There is no agreement on the number of experts required for the Delphi technique, however, it has been found that a panel of 10–15 experts is usually enough to explore the topic and build a consensus.[11] Out of 18 experts invited, 16 experts in the field of community medicine from India participated in the present study. The expert was defined to be an individual with relevant knowledge and experience on a particular topic or the one who has a position in the hierarchy, a recognized person in the field, or someone recommended by other participants.[12]

We used criterion type of purposive sample[13] of experts who were accessible, willing to express, experienced in postgraduate assessment, and had training in the field of medical education. The purpose was to capture their wisdom acquired during professional practice. The experts had different levels of hierarchy, affiliations to type of university, and geographical regions. The details of their characteristics are given in the first paragraph of results section.

Study tools, data collection, and analysis procedures for each Delphi round

The study was proved by SMVMCH-Institutional Ethics Committees (EC code No: 14/2018).

Delphi round one: Data collection and analysis

In the first round of Delphi, experts (n = 18) were asked to respond to a questionnaire consisting of broad, open-ended questions based on the research questions. The purpose of this round was to explore the strengths, weaknesses, and suggestions for improvement in assessment from the experts' point of view and finally develop the questionnaire for round two.

We used electronic mail (e-mail) to interact with the Delphi experts. This virtual mode of communication was found to be cheap, rapid and was found to yield high response rate.[14] The soft copy of the predesigned and pretested questionnaire was E-mailed to their preferred E-mail address; they were given 2 weeks to respond.

It is to be noted that researchers' academic background, and their pre-conceived ideas about assessment might have influenced the research process and interpretations.[15] It involves summarization and classification of data as categories and themes in simple quantitate terms. We used a practical approach to verify the categories generated, where the first investigator (AD) carried out the categorization with his other colleague in the department. Both were trained in the use of qualitative research methods and data analysis. Later, the second author (JN) reviewed the complied findings for face validity.[16]

Delphi rounds two and three: Data collection and analysis

In the second round, Delphi experts (n = 16) were E-mailed a soft copy of a Likert-type scale based on findings from round one. The experts were asked to rate the 14 items on a five-point Likert-type scale having options from “strongly agree to strongly disagree.” Experts were asked to consider feasibility, existing resources, and context while rating the items.

In the third round of Delphi, the experts (n = 16) were asked to select their top five among the items which reached consensus of more than 90% in the second round. One of the experts refused to rank order as he felt that all items were relevant and mutually related. The experts were also given an opportunity to revise their opinions on the items which did not achieve consensus in the second round of Delphi. All three rounds were completed over 4 weeks.

We used consensus measure to analyze experts' rating on items on suggestions for improvement in the assessment practices. We calculated weighted mean (WM) and the consensus values which range from 0% to 100%, where 0 indicates no consensus at all and 100% indicates full consensus.[17] A value above 80% was considered to reflect good consensus. In round three, experts were asked to select their top five items among items which reached a consensus of more than 90% in round two. The purpose was to find out the most prominent and representative suggestions for the improvement in assessment by calculating Smith's salience score, where score close to one indicates the prominence of items across the lists.[18]

RESULTS

We had a high response rate (89% for round one and 100% for rounds two and three). Out of 16 experts, 9 (56.2%) were male and 7 (43.8%) were female. Half of the experts were affiliated with government medical colleges, 6 (37.5%) were working in private medical colleges, and 2 (6.2%) experts were affiliated with a semi-government medical college in India. Out of 16, two experts had their medical college affiliated with a deemed university, and 14 (87.5%) experts had their college affiliated with the government universities. Six experts were professor and head, five were professors, two were associate professor, and one each was additional professor, dean (medical education), and a consultant to the university on curriculum development. All of them had interest and training in the field of medical education.

Results from Delphi round one

Overall, experts were concerned about the weaknesses in the current assessment system. Six experts mentioned that it is highly focused on testing the cognitive domain with little emphasis on competence or skills development. Two experts had concerns about the unfair and subjective nature of the current system as the pass/fail decision relies on performance in only one final test. To improve the overall assessment system, the suggestions varied from a radical change to incremental modifications to the current system. Four experts suggested breaking the course into smaller learning units with summative assessment at the end of each unit and formative assessments between the summative tests. They felt that the assessment system should be based on an assessment of competencies and a blueprint should bridge the gap between them and the assessments. The experts emphasized the need for formative assessment, choice of tools, selection, and training of examiners and development of an evaluation system for the overall and incremental improvement in the system [Table 1].

Table 1.

Experts’ opinion on strengths, weaknesses, and suggestions for improvement in overall assessment system (n=16)

Strengths	Weaknesses	Suggestions for improvement
Strengthen cognitive domain	Focus on cognitive domain	Overall organization of assessment
It helps the students to create the knowledge base - 1 Suitable to our context  Overall, it is a fair system in the given context - 3 Authentic  In clinico-social case, the setting is authentic - 1 Traditional and time tested  It assesses wider area of content using different methods - 2  Traditional and comfortable to faculty - 1  Time tested system - 1	It is not based on competency  Assessment of affective domain is not done - 1  It does not test all the skills required for a public health professional - 6 No formative assessment  No formative assessment - 7 Methods  Methods are not used in a right way - 1 Poorly trained examiners  Mostly examiners are not trained/poorly trained in assessment - 1 More subjective  Lack of objectivity and reliability - 3 Not fair  Decision on pass/fail relies on performance in one test - 2 An Implementation problem  The system is good, but there is problem in implementation - 1	Course can be divided into blocks specifying the competencies to be achieved at each level - 3  Divide the course into small learning units, have on-going formative assessment with a summative assessment at the end of each unit - 1 Focus of assessment  Outcome based assessment - 2  Define the outcomes to be assessed - 3 Link outcomes and assessment  Link outcomes with assessment through blueprinting - 4 Formative assessment  Develop formative assessment system - 6  Develop Internal assessment with some percent of marks added to summative assessment - 4 Summative assessment  Summative assessment can be structured - 3 Assessment tools  Decide the assessment tools for assessment - 1  Develop a question bank - 1  Use tools for WPBA - 1 External examiners  Select trained and experienced examiners - 5  University certification of examiners - 1 Evaluation of assessment system  Exit interviews with students - 1

Numbers indicate the frequency of the statement. WPBA: Workplace-based assessment

Results from Delphi round two

Consensus on suggestions for improvement in overall assessment

Out of 14 items, there was a good consensus (score >80%) on eight items. There was a consensus for aligning the assessment with the desired competencies (100%) and developing the formative assessment system (WM – 1.1; score – 91.5%). The experts agreed on the need to have a blueprint to align test content with the desired competencies (WM – 1.1; score – 91.5%) and they agreed that the blueprint should be shared with all the examiners (WM – 1.1; score – 91.5%). Experts thought that the examiners should be trained in the assessment tools which they use (WM – 1.3; score – 83.4%). Notably, they agreed to have an exit feedback system in place to obtain students' feedback on their experiences with current assessments (WM – 1.5; score – 80.7%) [Table 2].

Table 2.

Average rank of the ten items with the highest consensus (>90%) (n=15)

Items	Average rank	Smith’s S value
Assessment should focus on desired competencies	1.43	0.85
The current system should have formative assessment	2.80	0.43
Occasions such as posting to RHTC/UHTC/PHC/national health programs/seminars/journal club/thesis/teaching assignments should be used for formative assessment	3.67	0.37
The existing PG logbook use should be improved (providing space for students’ reflection/regular monitoring by supervising faculty)	3.70	0.31
Structured guidelines on occasions, frequency, competency to be assessed and tools for formative assessment should be specified	3.14	0.27
Blueprinting to ensure alignment between expected and assessed	3.00	0.24
Communicated blueprint to paper setters and examiners	3.43	0.24
Theory questions to follow higher order thinking, problem-solving	3.20	0.18
Examiners should not intimidate students	3.67	0.09
Needs structured format/guidelines for pedagogy test	5.00	0.01

PHC: Primary health center, UHTC: Urban health training center, RHTC: Rural health training center

Results from Delphi round three

The five most salient (prominent) items among the among the ten highest consensus (>90%) points were related to the need for having a competency-based assessment (Smith's S – 0.85), the need for formative assessment (0.43), the need for occasions for formative assessment (0.37), and the revision of the current logbook to include students' reflections (0.31) and structured guidelines for formative assessment (0.27) [Table 3].

Table 3.

Model framework for postgraduate assessment based on expert consensus

Assessment	Consensus items
Ongoing formative assessment	Assessment based on desired and (defined) competencies
	Blueprinting to align competencies with the tests
	Improvement in use of current assessment measures
End of course	Guidelines and checklists for objective assessment
Summative assessment	Training of faculty in the use of assessment measures
	Exit interviews with the students

DISCUSSION

The panel of experts had a mixed reaction to the current assessment system. Some felt that the current assessment system is fair, time-tested, and appropriate for the resources and context, while other experts pointed out that it is prejudicial, subjective, and focused too much on the assessment of lower-order cognitive domains of Bloom's taxonomy such as remembering and understanding. Notably, postgraduate assessment in other subjects of medicine in India[4] and overall, in the South Asian region[19] is also based on lower-order knowledge assessment.

The experts agreed to have an assessment system of community medicine based on ongoing formative and one end-of-year summative assessment. They emphasized the need to align the desired competencies and assessment measures. The experts agreed to the occasions such as community posting at rural health training center/urban health training center/primary health centers/posting at national health programs for formative assessment. In the West, such occasions are used for workplace-based assessment.[20] The success of such assessments depends on the student–teacher interactions, adequate number of assessments using a valid and reliable tool, and faculty training.[20]

In community medicine, the practice of maintaining PG logbooks is almost universal and it primarily includes accounts of activities done at different sites by the postgraduates. The experts agreed to improve the scope of exiting PG logbook by providing space for student's reflections on the activities with regular faculty supervision. The key practical tip includes aligning the learning objectives of the logbook with the curriculum; these learning objectives should be the part of the assessment.[21]

The expert panel suggested a better use of the current measures by bringing more objectivity to scoring through the introduction of guidelines and checklists. Overall, the experts clearly agreed that measures such as blueprinting, improving test formats, adequate briefing of test-taking students to reduce their anxiety, the training of examiners, and better test conditions would improve the validity and reliability of assessment scores.[22] It is noteworthy that most of the consensus items in the second round of Delphi meet the requirements for the development of an effective assessment system.[23] It is encouraging to note that the experts achieved consensus on having an exit feedback system in place to obtain students' views on their recent assessments, which is expected to help in an evaluation of overall assessment system.[23]

The summative assessment which is conducted by the university as an external examination includes both written tests and practical skills. In India, medical teachers have limited training in the “correct” use of traditional assessment measures and there is limited value given to feedback.[24] Since it is the “users” who can make a difference, formal training of teachers and their behavior change would be the key to successful transformation.[25] Therefore, experts suggested a better use of traditional assessment measures through faculty training and support. There are programs where a mix of formative and summative assessment, blueprinting, and training of faculty in the use of assessment measures could achieve its key goals.[26]

Experts could not reach a clear consensus on having standards for external examiners. “An external examiner is defined as a visiting assessor of high academic standing and possessing absolute integrity and objectivity – appeals to the notions of universality of educational standards and justice for the individual student” (pp. 1093).[27] In the United Kingdom, external examiners[28] and in North America, external examinations[27] have a role in ensuring quality in higher education. More discussion and research are required on this topic.

There was no consensus on the minimum pass level (i.e., standard setting). According to Postgraduate Medical Education Regulation in India, a student is required to score 50% in written test and 50% in practical skills assessments in their end-of-course summative examinations.[29] This is conventional arbitrary cutoff value. However, there is a growing realization that the pass–fail cutoffs should be based on scientific standard setting procedures to make them more credible and defensible.[30] There are several standard setting methods, and none is agreed upon as the best method,[31] hence the choice depends on a method that is acceptable in the given context. Table 3 summarizes the model framework for assessment based on findings.

To the best of our knowledge, the present Delphi study is the first systematic effort on this topic. The high response rate and rigorous scientific data collection and analysis brought credibility to the study findings. The limitations of the present study should be kept in mind. The researchers' own academic background in the field of community medicine and their qualification in the field of medical education might have influenced the research design and its results. Study participants' prior training in the field of medical education might have influenced their responses in support of modern assessment practices. However, the findings of the present study can be used for a large-scale survey.

CONCLUSION

In summary, most of the consensus items were in alignment with modern assessment theory. Regulating body and policymakers should revise the current postgraduate assessment system in community medicine to standardize and enhance its validity and reliability of assessment to produce competent professionals for health-care system.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.