Documentary research and evaluation in medical education

Medical education research has blossomed over the past 30 years. There are a growing number of general medical education journals and even some that cater for sub disciplines within medical education. [1] The content of these journals ranges from original research to systematic reviews to perspectives and opinion pieces. The original research itself takes a variety of forms from quantitative to qualitative-both of which categories contain a variety of methodologies. However there is one form of research that rarely features in the medical education literature and that is documentary research. Documentary research is research involving the analysis of documents (typically historical documents). It is undertaken for a number of reasons but primarily to gain insight into activities of the past and into the processes of change that have led from the past to the present. Hegel said that “we learn from history that we do not learn from history,” but here I would beg to differ. In medical education, we can learn from the past and just one example of this is the recent paper by Gill and Griffin [2] in Medical Education. In this paper, the authors analysed the text of Good Medical Practice produced by the General Medical Council between 1963 and 2010. Their analysis showed that there had been a “shift from a doctor-centred regulatory discourse to a patient-centred health improvement agenda over the period of time examined.” The core purpose of documentary research is ultimately educational evaluation. The reason why we conduct documentary research on policy papers, journal articles or even written curricula is to evaluate them and share the lessons learned from evaluation on to others.

So what is a document? A document may be defined as “an original or official paper relied on as the basis, proof, or support of something” or as “a writing conveying information” according to Merriam-Webster Dictionary. A document may be produced by an individual or a team of individuals or by an organisation. Documents might be private and created for an individual’s own use or perhaps for family friends only (e.g., diaries or personal letters). Alternative documents might be public documents published in journals or public records. In the medical education research domain, documentary research will mainly involve documents published in journals or public records. A document may be primary or secondary. A primary document is an original document and typically a direct report of a research study or an opinion or a perspective. A secondary document is a document produced by means of an analysis or study of one or more primary documents. Either or both may be used in documentary research but the researcher should be clear as to which type of document is being reported on. A variety of forms of documents are available to the medical education documentary researcher the most common ones are as below.

Textbooks of medical education are one type of document. Textbooks on the subject have been around since the middle of the last century. They can offer deep insights into the formats, values and theories that have driven medical education in the past and into how these have evolved into the formats, values and theories that we hold today. Educational reports are another source of important evidence from the past. Sometimes they can show us what has changed and sometimes what hasn’t changed; sometimes they can show us what we continu-ally plan to change but never actually do. According to Christakis [3], there has been remarkable similarity between the proposals over the past 100 years to reform medical education in the USA: “Reforms such as increasing generalist training, increasing ambulatory care exposure, providing social science courses, teaching lifelong and self-learning skills, rewarding teaching, clarifying the school mission, and centralizing curriculum control have appeared almost continuously since 1910.” Growing transparency in all walks of life means that board papers of educational authorities are increasingly made publically available and they can be another rich source of documentary evidence. Board papers of health authorities or teaching hospitals are also increasingly available. Curricula of past and present are also frequently available for documentary research and such documents frequently lend themselves to educational evaluation. The written curriculum evolves over time and the evolution of curricula can give deep insights into the planned provision of medical education within undergraduate and postgraduate institutions. Journals articles are another rich source of documentary evidence in medical education. Some journals are exclusively devoted to medical education; others publish general education research and some medical education research; others still publish general medical research and some medical education research. Virtually all are now online and many have published their archives online. They are probably the most accessible way of conducting documentary research in medical education. One simple example of documentary research is that conducted by Walsh and Hally by reviewing the frequencies of competing interests reported in the medical education research journals [4]. Some medical education journals are open access and thus their content is made free available to the readers. However other journals put their content behind paid access controls - this can be a barrier to the documentary researcher. Fiction can also be a powerful source in documentary research. Richard Gordon’s Doctor in the House [5] is a work of light fiction and yet gives an insightful picture of life as a medical student in the middle of the twentieth century. Here is Gordon on selection methods for medical school: “I wondered for some time afterwards how he had been able to discover from these questions that I had the attributes of a successful doctor, but I later found out that even this brief interview was superfluous, as the Dean always took the advice of his old secretary and told applicants this man disliked the look of that there were no vacancies.” Diaries or personal correspondence can also be used in documentary research, and are often used in general historical documentary research. However there is not a rich source of available material in the medical education domain.

Sometimes a number of different types of documents can be researched; this can give deeper insights into medical education phenomena. For example in the case of UK medical education, a documentary researcher could decide to do documentary research on the establishment of Modernising Medical Careers-the programme for postgraduate medical education in the UK [6]. The researcher might analyse the official Department of Health documents that relate to Modernising Medical Careers and then the documents that were published about Modernising Medical Careers in the medical education literature (for example editorials, perspectives, opinion pieces and correspondence). This might give insight into what Modernising Medical Careers actually said and what the medical education community subsequently said it said.

Documentary research is carried out in a number of steps. The first step is the document search and the above introduction should be a useful initial guide. The next step is reading the documents found. After that comes analysis of the documents. The first part of the analysis is establishing that the documents are genuine and reliable. Establishing that a document is genuine is usually reasonably straightforward unlike in historical research where fake documents might be produced. Establishing that the document is reliable is inevitably much more subjective. In medical education documentary research, the documents might be reliable in themselves and yet biased in their perspective. Many of the documents available (be they texts, research papers or white papers) will be written by academics, researchers or policy makers. It can be hard to find the voice of the learner or the patient in medical education documents. This is perhaps a weakness of this form of research in this domain. The next step is the analysis itself. The analysis cannot happen in a vacuum, it should be based on a theory or methodological process. Three broad approaches are usually employed in documentary research analysis. The first is positivism [7]. The positivist approach is a quantitative one; it based on the premise that all reliable sources of knowledge are based on logical and rational evidence or science. The positivist approach suggests that in research we should adhere closely to what we can objectively view and quantify. The second is the interpretive approach [8]. In the context of documentary research this suggests that documents are socially constructed, and that to understand and interpret documents we need to understand the subjective social context in which they have been created. The third is the critical approach [9]. In documentary research this approach means assessing and evaluating documents from the perspective of different ideologies and social sciences and the intention is political-the liberation of individuals and groups in a free society.

The final stage in documentary research is writing it up. Here the general rules of good writing and reporting approach apply as much to documentary research as they do to other forms of medical writing and reporting. A few rules however are particularly important in the context of documentary research. Authors should be explicit about the provenance of the documents on which their report is based and should be explicit about what documents they examined and what they did not examine and why the included and excluded certain documents of types of documents. Just as important authors should be clear about the theory of research that underpinned their analysis. Authors should also ensure that their report is usable and useful from the perspectives of the readers and stakeholders. If the true purpose of documentary research is in the final analysis evaluation, then authors should ensure that the report is written in a way that will enable the lessons learned from such an evaluation to be disseminated. Finally documentary research is only possible because of the archivists, journal editors or authors who make their documents available. It is good practice as well as common courtesy to thank and acknowledge them.

Documentary research has an important role to play within medical education research. Documentary research can enable us to learn from activities of the past and to put the lessons learned into practice for the benefit of medical learners.