Oral examinations are widely used in undergraduate and postgraduate medical examinations, including those for membership of royal colleges. In recent years the validity and reliability of oral examinations have been questioned and they have been dropped from many assessment programmes.1–4 The Royal College of General Practitioners has a tradition of reflecting on these issues and has improved its membership examination through the careful selection and training of examiners, the use of an examination blueprint, the development of clear questions and criteria for marking, and continuing discussion of techniques and problems.5 None the less, the debate about equal opportunities and the possible discriminatory outcomes of the examination has continued as failed candidates and their colleagues have suggested that their lack of success might be due to racial discrimination.
Ethnographic and sociolinguistic discourse analysis
Discourse analysis is the detailed study of language in use. Unlike many traditional ways of assessing communication that use coding schemes, it examines the finegrained detail of interaction—sentence structure, intonation, taking turns, and non-verbal communication—as well as the content18
Ethnographic method involves observing and interviewing, taking the informant's perspective as far as possible in order to understand the cultural and communicative patterns which inform his or her behaviour and perceptions
Sociolinguistics examines variety in language use and, in this context, studies how differences in talk and self presentation can lead to negative assessments and possibly to racial discrimination
A study in 1995 showed that although the success rate among British Asians was not in question, Asian doctors trained abroad did less well than other groups.6 They were relatively more likely to fail the written examination, which meant that they were not called for the oral examination. But even among those called for the oral examination, ethnic minority candidates trained overseas did proportionately less well. The examination board's commitment to fairness led to further scrutiny of the examination procedures in the light of these findings, and there was still concern that some groups were receiving relatively low grades. This led the board to consider whether the oral examination might, in hidden and subtle ways, be a contributing factor, and if a study of the use of language in the oral examinations would illuminate possible indirect discrimination.
In terms of psychometric testing, “discriminatory ability” refers to the ability to distinguish between pass and fail, good and less good candidates. However, in the context of this paper and of equal opportunities, we use the term “discrimination” to mean the outcomes of systems, procedures, and judgments that may have an unfair impact on some social groups. The examination board of the Royal College of General Practitioners wished to evaluate proactively its oral procedures and practices to establish whether there were any grounds for believing that the examination could be indirectly discriminatory, in this latter sense of the term.
Summary points
Candidates from ethnic minorities and those trained abroad may experience particular hidden difficulties with oral examinations
Examination boards—especially those admitting candidates trained abroad—should educate their examiners about these difficulties and their implications
Examples of oral questions, a range of answers, and examiners' comments on these should be published so that all candidates sitting oral examinations have equal opportunities
Oral examinations typically involve one or two examiners (two in the membership examination of the Royal College of General Practitioners) who interview candidates about a defined area of clinical practice and judge their ability on the basis of their responses. The psychometric approach to assessment seeks to elicit responses that can be scored and to guard against bias based on irrelevant individual psychological factors. Though these concerns are important, they tend to be addressed as if the interaction making up an oral examination is a transparent channel through which facts, values, and opinions pass. From sociolinguistic and interactional perspectives, this is not the case. Detailed analyses of the use of language in oral examinations—studying the interaction as a topic in its own right and not just as a resource for assessing candidates' knowledge—raises new issues about oral assessment.7 It is candidates' talk that is assessed, and yet this talk is “constructed” by examiners' questions and interaction. Sociolinguistic discourse analysis (box) can provide insights into this interactional complexity.
In this study, detailed linguistic and interactional analysis was related to the ethnographic knowledge gained from observations and interviews. These were also related to sociolinguistic knowledge of variety in language use in terms of the setting (the interview versus the doctor-patient consultation) and social and ethnic identity.
Method
The main data collection took place over two days and comprised direct observation of 24 oral examinations (each lasting 30 minutes) for membership of the Royal College of General Practitioners. We selected examinations conducted by different pairs of examiners and in which the candidates came from ethnic minorities. Interviews were then conducted with 14 ethnic minority candidates immediately after their oral examination. We did not interview the examiners for practical reasons—they were examining. We also studied videotapes of 11 oral examinations involving candidates from ethnic minorities; these were used for training examiners. To provide a context for the analysis of this material, CR and SS observed and participated in the selection and training days for new examiners. In each instance, consent from individuals and groups was obtained.
SUE SHARPLES
Data analysis was based on ethnographic8 and sociolinguistic-discourse analytic approaches,9–11 which resulted in a discursive synthesis of the findings. These approaches have been used extensively to study clinical settings, but they are also particularly relevant for investigating the ways in which language constructs life chance events such as interviews and tests.12,13 The videotaped examinations were transcribed and a close linguistic and interactional analysis was made of “awkward moments”—times when there was tension or poor communication between examiners and candidates, as identified by a group of oral examiners.
Our analysis of all the sources of data identified three areas of particular difficulty for examiners and candidates. We refer to these as hybrid discourse, interactional complexity, and “slippery areas.” The first area of difficulty—hybrid discourse—is the most important.
Hybrid discourse
The language of the oral examinations is not a transparent medium through which information passes but a set of discourses that actively construct a particular way of looking at the world.9 The different ways in which clinical practice is talked about in different institutional and professional domains and by different groups, produces a range of discourses that call up particular types of vocabulary, metaphor, and grammatical constructions and certain lines of argument and representation.
Analysed in this way, the oral examinations of the Royal College of General Practitioners consist of different types of discourse that combine professional competence and knowledge based on experience. A question such as “Let's take patient number 16, the 17 year old requesting a pregnancy test” will tend to elicit a particular kind of discourse, an account of the general practitioner's routine practice and the decisions he or she has to make. The question “What do you understand by listening skills?” is likely to elicit an abstract, analytical response that is relatively removed from the general practitioner's actual professional competence.
Types of discourse
Personal experience discourse is talk concerned with the individual's experiences and feelings. It usually takes the form of a narrative (anecdotes, reminiscences, etc) and deals with the “here and now experience of the concrete particulars of a case in hand” and “the accumulated experience of a similar case over time”19
Professional discourse is the talk of doctors in practice, exemplified in doctor-patient interviews, in case rounds in hospitals, and in a range of doctor-doctor discussions and meetings. It is the discourse of shared ways of knowing and seeing that characterise the community of medical practitioners
Institutional discourse is not the actual talk that general practitioners use in their consultations (that is, professional discourse); rather it is the ways in which general practitioners account for this talk. In other words, the everyday competencies and practices of the general practitioner have to be presented in institutional terms through language that reifies and abstracts these practices. The dominance of institutional discourse over other forms of discourse is maintained through typical institutional encounters such as selection examinations, departmental meetings, and quality assessments that hold institutions together both politically and organisationally
Sociolinguists make a distinction between three types of discourse—personal experience discourse, professional discourse, and institutional discourse (box). The membership oral examination included all three types of discourse. The personal experience and professional discourses often merged, but institutional discourse was identifiably different, tended to dominate, and was the problematic element. For example, the ability to communicate well, manage one's own levels of stress, and be flexible in dealing with patients has to be presented in the examination through means of rationalisation and reification. Meeting the requirements of the institution requires that the “what” and “how” of general practice are transformed into the “why” of medicine, health, and illness. Although an examiner's question seems, on the surface, to elicit a response based on personal experience or professional discourse, the most acceptable answer is often at the institutional discourse level. In other words, examiners' questions are necessarily ambivalent in terms of eliciting different discourses.
The following question combines elements of both professional and personal experience. Question: If someone took out a personal complaint against you, how would you react? Answer: That's where my personal stress management plan comes in.... Although the theme of the question was professional and personal, the candidate responded in an institutional way. Instead of describing how she would feel (personal experience discourse) or what she did in her practice in relation to a specific case (professional discourse), she produced an abstract and analytical statement (institutional discourse) that satisfied the examiners as a response they could mark. Head nodding and mutual smiles provided confirmation. As in this example, general practitioners who have a good command of institutional discourse and are able to manage the “hybridity” between different discourses of the examination are most likely to do well (box).
The institutional response to personal or professional questions
These questions from the oral examinations have professional or personal experiences as the potential theme. However, a response couched in institutional discourse terms would count as one that could be marked, and it would therefore be acceptable to the examiners.
What do you understand by the term “values”?
What does the concept of “patient-centredness” mean to you?
What strategies would you use for coping with uncertainty?
Interactional complexity
The hybrid discourses of the oral examination place demands on examiners and candidates, and these demands are reinforced by the interactional complexities of the examination. There are two particular problems to be discussed here—“joint construction” and “shifting frames.” Although the examiners set the topics and decide when to close them, the progress and quality of the interaction is constructed by the examiners and candidate together. Like any interaction, the oral examination is jointly constructed.14 This means that every question contributes to the formulation of the response and produces, for both examiner and candidate, a relatively more or less comfortable moment (see box for example).
Combining to construct an uncomfortable moment
Question: What do patients feel like when they are ill?
Answer: [pause]
Question: What does illness make you feel? Fear? Pain? How do you find out what a patient wants from the service? That's what I'm getting at ...
Answer: [pause]
In the example below, the candidate's pause leads the examiner to rephrase the question about patients' responses to illness and to turn it from an open to a closed question. The candidate's “Yes” or “No” response to this closed question might then be judged as inadequate, but it is the examiner who has brought about this response. In addition, the examiner, looking for a way out of the communication impasse, produces a rapid fire series of questions, which makes the candidate even less able to answer.
Questioning, therefore, is not only a matter of eliciting responses that can be marked, and of maintaining clarity and speed. Every question—through the choice of words used, their order and structure, their insertion at a particular moment in the interaction, and so on—serves to manage the interaction in a particular way. Thus the questions help to construct the candidate's response. Similarly, a candidate's response contributes towards the absence or presence of further questions on a topic. Candidates' answers are then assessed, in a psychometric model, as an individual response rather than the product of a joint interaction.
The relation between hybrid discourses and interactional complexity can lead to examiners and candidates talking at cross purposes or examiners talking down to candidates. A subtler example has to do with “shifting frames” during the oral examination.15 Sometimes examiners shift into a role play frame (that is, they talk as a patient would in the surgery) or some version of this to try to elicit more professional discourse from the candidate.16 However, the cues for this frame are subtle features in intonation and voice quality, which may be particularly difficult for candidates from non-English speaking backgrounds to pick up on. The candidate may thus continue with his or her institutional discourse while the examiner has moved into a role play consultation. Thus, the shift in frames and the consequent mismatch in discourse types places demands on the candidate which he or she cannot meet.
Conversely, interactional difficulties may lead to an unexpected shift in frame and discourse type. Reconsider the example in the box above. The first question should elicit an answer using institutional discourse in which the candidate lists, in an analytical way, the possible experiences and feelings of patients. The candidate's silence is interpreted as an interactional difficulty. In his next turn, the examiner shifts frame—firstly, to a personal experience question (“What does illness make you feel?”) and then to a professional discourse question (“How do you find out what a patient wants from the service?”). Notice the shift from “What do patients feel?” to “What does illness make you feel?” Whereas the first formulation is ambivalent and could access all the three discourse types, the second formulation is specific and narrows it down to personal experience. The further question about “How do you find out” extends the personal experience domain to the domain of professional competence.
Examiners and candidates have to manage these hybrid discourses within their conversation. Dealing with hybrid discourses can create interactional difficulties and uncomfortable moments. But, as we have just seen, the interactional demands of managing a face to face interview can, in turn, produce hybrid discourses. It could be argued that the smooth management of hybrid discourses should be part of a general practitioner's competence—in which case, this needs to be made explicit in information about and preparation for the examination. But it is the combination of managing both hybrid discourses and the interactional complexities of the oral examination that can place unfair demands on candidates, particularly those from ethnic minority and non-British backgrounds. Uncomfortable moments which arise either because the interaction is a “bumpy ride” or because the candidate has not produced the expected discourse or blend of discourse can readily lead to negative judgments about candidates.
Slippery areas
The inclusion in the oral examination of topics covering the areas of communication, problem solving, and values and attitudes was an important step towards defining the special nature of general practice as a discipline. But such themes are notoriously slippery areas when it comes to assessment in a face to face format. There are three main difficulties here.
Values and attitudes
Talking about values and attitudes (which will invariably draw on personal experience) in institutionally acceptable ways can be difficult for many candidates—especially candidates trained overseas in a language other than English, or in a local variety of English, and in a culturally different value system.
Areas of uncertainty
The nature of the oral examination means that candidates may be pushed into areas of relative uncertainty (for example, the sociology of medical knowledge), and managing this type of uncertainty requires a high degree of linguistic finesse. For example, it requires considerable communicative skill to be able to attribute uncertainty to the current state of medical knowledge rather than to one's own ignorance.
Cultural differences
With some topics, cultural differences and experiences of racism may be the focus. For example, attitudes to contraception, termination, sexuality, and so on are shaped by social and cultural experiences over time. Candidates may have to manage conflicting perspectives—that of the majority, as evidenced in the medical journals, and their own view. The point here is not that there are potentially conflicting views about these topics but that the difficulty in the examination is talking about them in institutionally appropriate ways.
Equal opportunities and ethnicity
Identifying and managing hybrid discourses is difficult for all candidates, but interactional difficulty and slippery areas are particularly problematic for candidates from different ethnic and linguistic backgrounds or when the examiner and candidate come from different cultures. It is the combination of the three areas of difficulty that makes the oral examination particularly problematic.
We have already raised some of the difficulties for candidates trained overseas whose style of English is different from that used in the United Kingdom and who have difficulty in talking about values and attitudes in an institutional discourse mode. These difficulties are not matters of vocabulary or grammar but of style of communication,17 such as how direct or indirect to be, how personal or impersonal, what constitutes an appropriate length of answer, and how literally to interpret a question. Communicative differences can be compounded further by different assumptions about the purpose of the oral examination—whether it is aimed at assessing professional competence rather than institutional performance.
It is the interplay of the factors we have discussed—hybrid discourses, interactional complexity, slippery areas, and issues of ethnicity and language—that produces the potential for discrimination in the oral examinations. This example shows how a candidate from a linguistic minority could misread the cues.
Question: A young chap of 26 who has difficulty with sleeping comes to see you. How does that make you feel?
Answer: It feels a threatening consultation. It's a difficult consultation. It's quite possible to encounter a difficulty in the beginning. “I'm sorry you can't sleep. I remember when my child was little.”
Question: Can you briefly list alternative ways…
The initial question was framed in terms of personal experience, but the examiner's follow up question, “Can you briefly list alternative ways…”was couched in institutional discourse. The reification of the patient into a list of alternatives would produce a response that could be marked. The fact that the examiner reformulated his original question suggests that the candidate's initial response did not meet these criteria. In an interview after the examination, the candidate explained that she thought the examination was designed to assess personality and the suitability of the doctor's psychological make up for general practice. So she tried to put herself across as a thoughtful, caring person, taking the perspective of the patient. She relied on personal experience discourse, putting herself into a role play frame to act out the situation—living within the consultation rather than treating it from the outside analytically. The examiner shifted the frame back to the oral examination, thus undermining the frame shift initiated by the candidate. Since she produced an “inappropriate” answer, the examiner intervened with his follow up question, which required a rapid shift from her role play, couched in personal experience and professional discourse mode, into institutional discourse mode.
In the follow up interview this candidate said that she had been trained overseas and was aware that her communicative style in English was often perceived as too direct and abrupt. Because of this she had tried not to seem too direct. This may have helped account for the fact that the examiners judged her as “rambling” during the examination. Other contributory factors may have been that in her first language there is a higher tolerance of digression in the way in which information is structured. Linguistic and cultural differences also contributed to her literal reading of the examiner's question, “How does that make you feel?” She responded to it as a question about her personal feelings and so answered it descriptively and anecdotally.
The hybrid discourses in the examination led to communication that was at cross purposes, and the candidate's style of communication was at odds with that of the examiners. Both factors may have contributed to the candidate's relatively low mark. There was certainly no case of direct discrimination in this or any of the cases in this study. The issue is to what extent examiners could be certain that they had made the right judgment of a candidate such as this. To sum up, the question is not how competent a doctor she is—she certainly seemed concerned in her interview about being patient centred—but that the discourse of the examination put her at a disadvantage.
Our analysis has focused on certain groups of ethnic minority candidates, but the issues surrounding hybrid discourses, and in particular managing institutional discourse, are ones that may affect many other groups. For example, social class, or membership of the group of older doctors who have been practising for many years and are not used to taking examinations, may also adversely affect how candidates cope with the institutional discourse of the oral examinations. Though minority candidates from different ethnic and linguistic backgrounds may be disproportionately disadvantaged by these factors, this group's presence highlights difficulties that non-traditional candidates from a variety of backgrounds may face.
Conclusion
The hybridity and complexity of the oral examination puts additional hidden demands on both examiners and candidates. These stem from the fact that the oral examination seems to assess candidates' professional discourse but does so through institutional discourse or a hybrid of all three discourses. This can lead to misunderstandings, mismatches, and cross purposes reinforced by the difficulty of managing any oral examination or selection interview where both time pressure and the social pressures of face to face interaction must be taken into account.
These demands make the oral examination difficult for all participants, but particularly so for those from ethnic minorities who have been trained overseas. This is most clearly evident when they have to tackle questions on what we have called the slippery areas. The potential for discrimination for this group of candidates, in particular, but for all non-traditional candidates more generally, is an important issue to address. There is no room for complacency when it comes to issues of reliability and fairness in oral examinations where candidates' professional identity, competence, and self esteem are open to scrutiny. This is particularly the case when issues of social justice—relating to equal opportunities for minority groups—may be at stake. In the now modular examinations for entry to the Royal College of General Practitioners, a candidate must pass each component, including the orals, to pass overall.
Practical implications
These findings may also apply to other comparable examinations. They may be particularly relevant to examining bodies that allow large numbers of overseas candidates to take their United Kingdom oriented examinations (using British examiners). Moreover, many of the issues raised could also apply to other assessment methods, and we plan to examine this possibility within our examination by further statistical analysis. But intercultural oral examinations, especially, need to be monitored to identify any possible patterns of discrimination that relate to content.
There are several implications for very practical action in respect of the membership examination of the Royal College of General Practitioners. Firstly, examiners need sensitising about the issues and explicit training about their role in producing hybrid discourses and interactional complexity in the oral examination. Secondly, the examination board needs to develop and publish examples of oral questions, together with examples of different candidates' answers and examiners' comments on these, with the issues raised in this paper in mind. Thirdly, examiners need guidance on making the intention(s) of questions more explicit and should not shift frame without making it clear what they are doing.
Acknowledgments
We thank David Haslam, chairman of the Royal College of General Practitioners' examination board and the members of the board for their encouragement and support; Michael Thirlwall, other members of the Oral Development Group, and the many examiners who gave us their time and insights; and the participants for their assistance, time, and forbearance at what was a difficult moment for many of them. Although the Royal College of General Practitioners and its examining board supported the work reported here, nothing should be interpreted as representing their views.
Footnotes
Funding: Examination Board, Royal College of General Practitioners.
Competing interests: None declared.
References
- 1.Wakeford R. General policies. Vol. 1. London: General Medical Council; 1992. Report of the survey of undergraduate medical education 1990; pp. 53–54. [Google Scholar]
- 2.Wakeford RE, Belton A, Maatsch J, Norman GR, Rainsberry P. Proceedings of the annual conference on research in medical education. Washington, DC: Association of American Medical Colleges; 1986. Symposium: development of the oral examination as part of specialist certification examinations—an international perspective; pp. 339–346. [PubMed] [Google Scholar]
- 3.Evans LR, Ingersoll RW, Smith EJ. The reliability, validity and taxonomic structure of the oral examination. J Med Educ. 1966;41:651–657. doi: 10.1097/00001888-196607000-00002. [DOI] [PubMed] [Google Scholar]
- 4.Thomas CS, Mellsop G, Callender J, Crawshaw J, Ellis PM, Hall A. The oral examination: a study of academic and non-academic factors. Med Educ. 1993;27:433–439. doi: 10.1111/j.1365-2923.1993.tb00297.x. [DOI] [PubMed] [Google Scholar]
- 5.Wakeford R, Southgate L, Wass V. Improving oral examinations: selecting, training and monitoring examiners for the MRCGP. BMJ. 1995;311:331–335. doi: 10.1136/bmj.311.7010.931. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Wakeford R, Farooqi A, Rashid A, Southgate L. Does the MRCGP discriminate against Asian doctors? BMJ. 1992;305:92–94. doi: 10.1136/bmj.305.6845.92. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Kvale S. Examinations re-examined: certification of students or certification of knowledge? In: Chaiklin S, Lave J, editors. Understanding practice: perspectives on activity and context. Cambridge: Cambridge University Press; 1996. pp. 473–503. [Google Scholar]
- 8.Hammersley M, Atkinson P. Ethnography: principles in practice. London: Tavistock; 1991. [Google Scholar]
- 9.Gumperz J. Discourse strategies. Cambridge: Cambridge University Press; 1982. [Google Scholar]
- 10.Fairclough N. Language and power. London: Longman; 1989. [Google Scholar]
- 11.Roberts C, Sarangi S. “But are they one of us?” Managing and evaluating identities in work-related contexts. Multilingua. 1995;14:363–390. [Google Scholar]
- 12.Roberts C, Davies E, Jupp T. Language and discrimination. London: Longman; 1992. [Google Scholar]
- 13.Sarangi S. Accounting for mismatches in intercultural selection interviews. Multilingua. 1994;13:1–2. ,163-94. [Google Scholar]
- 14.Atkinson M, Heritage J, editors. Structures of social action: studies in conversational analysis. Cambridge: Cambridge University Press; 1984. [Google Scholar]
- 15.Goffman E. Forms of talk. Philadelphia: University of Pennsylvania Press; 1981. [Google Scholar]
- 16.Roberts C, Sarangi S. Hybridity in gatekeeping discourse: issues of practical relevance for the researcher. In: Sarangi S, Roberts C, editors. Talk, work and institutional order: discourse in medical, mediation and management settings. Berlin: Mouton de Gruyter; 1999. pp. 473–503. [Google Scholar]
- 17.Scollon R, Scollon S. Intercultural communication. Oxford: Blackwell; 1995. [Google Scholar]
- 18.Elwyn G, Gwyn R. Stories we hear and stories we tell: analysing talk in clinical practice. BMJ. 1999;318:186–188. doi: 10.1136/bmj.318.7177.186. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Atkinson P. Medical talk and medical work. London: Sage; 1995. [Google Scholar]