The epidemiology of medication errors: the methodological difficulties

Abstract

1. Medication errors should be amenable to epidemiological analysis, giving insights into the causes of error and the effects of interventions to prevent them or reduce harm from them.

2. There are formidable difficulties in establishing the rates of medication errors.

3. There is no agreement on a clear operational definition of the condition.

4. The methods used to enumerate cases so far have been unreliable or incomplete or both.

5. There is disagreement about whether cases of error that do not cause harm should be included in calculations of error rates.

6. When harm occurs in association with drug therapy, it is often unclear whether the harm might have been prevented, and its occurrence should therefore be considered to result from error.

7. The denominator for calculating the rate of error is both ill-defined and inconsistently measured.

8. Better definitions, more complete evaluation, and more thorough impact assessment may improve matters.

The tragedy that well-intentioned medical interventions lead to harm is greater when the harm is the result of some error on the part of one or more healthcare professionals. Although error has long been recognized as an occasional consequence of therapeutic interventions, the last decade has seen a dramatic re-evaluation of the frequency of medication errors and of their most serious consequences. It is clear that the problem is important, and that its solution requires accurate information on the rate and consequences of errors. What is less clear is the way in which such epidemiological information can be gathered, and the reliability of the data obtained so far. There must be reservations about reliability when estimates of error rates vary by orders of magnitude [1].

In this review I consider some of the problems in counting and classifying medication errors. The basic requirements for epidemiological study include: a clear study design; and an operational definition for the numerator, the condition to be evaluated; and for the denominator, the population in which it is to be evaluated. These requirements pose problems when the object of study is error.

Defining a medication error

Aronson and I have listed and analysed many definitions of medication error, and have proposed the following: ‘A medication error is a failure in the treatment process that leads to, or has the potential to lead to, harm to the patient’[2]. This proved to be the most robust of several definitions subsequently evaluated by Yu and others [3]. The Delphi method, originally designed to predict events rather than decide facts, allows consensus to be reached (often by attrition). The method has been used to formulate a definition of prescribing errors (a subset of medication errors), in the hope that this would be acceptable to others and provide a basis for comparison of different studies [4]. This hope has not yet been realized: a survey of error reporting in 132 intensive care units showed that many different definitions were used, and that often more than one definition was used in the same institution [5].

In deciding whether a medication error has occurred, it is necessary to judge the extent to which a deviation from some ideal treatment process can be classed as a failure. It is also necessary to judge, if there is harm, the association between the putative error and the harm. When there is no observable harm, a judgement has to be made of the possibility that the failure would have led to harm in other circumstances – for example, if it had not been intercepted, or if it had occurred in another context. There is division over whether errors that cause no harm should anyway be excluded [6]. Those who argue in favour of counting ‘near misses’ sometimes invoke the principle that the frequency of major harms is proportional to the frequency of no-harm incidents, the constant of proportionality being the Heinrich ratio. This seems not to be the case for medication errors [7]. The absence of a relationship is perhaps less a reason for counting only errors that result in harm than a reason for detailed investigation of the likelihood of possible harmful outcomes from errors that themselves cause no harm.

Study design

Others have specified the desiderata for studies of medication errors [8]. One of these is that studies be prospective. Prospective studies have substantial advantages in epidemiological research, since they can in principle ensure that the required information is collected, reduce bias, and allow complete ascertainment. While it should be straightforward to distinguish between prospective and retrospective designs in epidemiology, this is not always possible in error research. For example, it may only be possible to determine whether an adverse event was due to error by retrospective analysis of case notes and prescribing records, even if the event is sought prospectively. This uncertainty may have unwittingly biased systematic reviews that include only prospective studies [9].

Evaluating the numerator

Epidemiological studies might be expected to be explicit about the assumptions underlying the counting of medication errors. There are many methods for counting errors, none of which is of itself entirely satisfactory.

Reviewing prescriptions and hospital records

Studies that count errors in written prescriptions fit neatly into the system of hospital clinical pharmacy, in which a pharmacist tours each ward inspecting standardized drug charts, widely adopted in the UK and elsewhere [10]. It can also be used in community pharmacies. Examples include the early studies by Jones [11] and Tesh et al.[12], and the influential study by Dean et al.[13]. The method remains popular [14]. Such studies might identify trivial deviations from ideal that would never be harmful, and will necessarily miss errors later in the medication process. Reviewing six studies of prescription writing, Bobb et al. found a 30-fold difference in the reported incidence of errors [15].

The examination of medical records (‘chart review’) has been widely used [16, 17]. A more complete understanding of medication errors and the harm they cause can come from reviewing a wide range of information: hospital discharge summaries, procedure notes, physician progress notes, laboratory reports, physician orders, and nursing/multidisciplinary progress notes were suggested in one US study [18]. In the UK, Hogan et al. identified the sources of data on patient safety in hospitals as: clinical incident database, health and safety incident database, complaints database, claims database, inquest database, the patient administration system, and case notes [19]. The existence of several sources helps to compensate for the incompleteness of each source. Although formal capture–recapture techniques have been used in analysing adverse drug reactions [20], they do not seem to have been applied to the problem of medication errors.

The advent of computerized physician order entry (prescriptions typed into a computer) has brought the possibility of automatic monitoring of potential medication errors. For example, if midazolam is prescribed followed by flumazenil, one would be justified in believing that the flumazenil was required to reverse the effect of midazolam, perhaps because of an iatrogenic overdose [21]. Although the theory is encouraging, and the results are objective, or at least even-handed, there are problems in designing rules of sufficient specificity and sensitivity.

In one study, spontaneous reporting, using a simple computer interface, was compared with automatic searching, using algorithms to search for signals, including discontinuation of medications, reductions in dosages, ordering of known antidotes, ordering of specific laboratory tests (such as drug concentrations), Clostridium difficile toxin assays, and specific laboratory abnormalities [22]. To overcome the difficulty of specificity, each ‘signal’ was assessed in detail by a clinical pharmacist. In 18 months, nine events were reported spontaneously, whereas 731 were detected by computer algorithm, supplemented by the pharmacist's review; but the false-positive rate is not stated. In a similar study, 88% of alerts concerning phenytoin were false positives [23].

Observing the administration of medicines directly

The medication process can be observed directly [24]. Barker et al. used trained observers, mainly nurses, to compare prescribed and administered dose [25, 26], having found this the most reliable way to determine the incidence of error [27]. Frequent errors can be detected, even where the observer's purpose is obvious and those being observed have given informed consent [28]. The effect of the presence of the observer can limit such studies, although it may not be a severe difficulty, since the true purpose of the exercise can be disguised. Taxis and Barber used disguised observation in an attempt to clarify the incidence of errors in intravenous administration, four previous studies having yielded rates of 13–84%, and found a rate of approximately 50% [29].

Measuring the amount of medicine administered

The final result of dosage calculation, prescription, and preparation for administration of an intravenous infusion is the solution that is actually infused, and its concentration is a measure of the overall error rate. Substantial deviations from the prescribed dose have been found, and could be clinically important when dosages are critical, as with catecholamine infusions in intensive care [30]. Studies of complex infusion regimens, such as the regimen for intravenous acetylcysteine, show surprisingly high error rates – only one-third of infusion bags have concentrations within 10% of the anticipated value [31].

Evaluating spontaneous reports

Ideally, all practising clinicians would recognize and report their own errors, so that common causes can be elucidated and systems improved. Certainly, organizations such as the US Food and Drug Administration, the US Institute for Safe Medication Practice, and the National Patient Safety Agency in the UK, which collect spontaneous reports of errors, have been able to issue warnings about serious hazards. Examples include warnings of dosage errors in the use of heparin flushes in neonates [32] and the risk of overdosage of midazolam during conscious sedation [33]. Spontaneous reporting schemes have been used to determine error rates [34, 35]. However, there is little hope of reliable epidemiological information when reporters differ greatly in what they choose to report, or whether they report at all. Even in the reporting of adverse drug reactions, when the reporter is unlikely to face disciplinary difficulties, spontaneous schemes are notorious for underreporting. One important hospital-based study showed that fewer than 10% of all serious adverse drug reactions were reported to the UK Yellow Card Scheme [36].

Eliciting reports of medication errors

Nurse investigators ‘solicited information from nurses, pharmacists, and clerical personnel’ in the influential Boston study [37]; they did not ask doctors. In a questionnaire survey, first-year doctors were asked to recall errors that they had made in administering intravenous medicines; there was an 80% response rate [38]. A study in an Italian intensive care unit showed that facilitated reports by staff revealed only half the events found by direct observation [39].

Reviewing hospital discharge data and mortality reports

A great deal of hospital information is now available as computer data that can be searched systematically, although suggestions for classifying data by algorithms based on Bayes theorem [40] seem not to have been widely adopted. This may change as evidence accrues that natural language programs detect significantly more errors than spontaneous reporting [41]. Coroners' investigations of unnatural deaths can yield more complete information on the cases identified, but are subject to (under)reporting biases [42]. One extreme form of mortality report is the criminal charge sheet, and it sometimes happens that doctors and other healthcare professionals are charged with criminal negligence manslaughter [43]. Doctors are being charged more often, but this is much more likely to reflect the views of prosecutors and others on the threshold for charging an individual, and the overall increase in medical intervention, than it is to imply an increase in the frequency of culpable medical error.

Counting errors under experimental conditions

When asked to calculate drug doses, nurses made 10-fold errors in 8% of calculations; experienced nurses made as many errors as inexperienced nurses, but were surer of their abilities [44]. The rate of 10-fold errors was slightly higher in a separate study of paediatric resident doctors [45]. When healthcare professionals were asked to prepare intravenous infusions for use in neonates, 35% of the infusions contained concentration errors [46].

Analysis of the collected data

Error and harm

Sometimes the collected data are accepted as indicating error, either because they have been spontaneously volunteered as representing an error or because they have been collected by an observer (usually a pharmacist or nurse) who has designated the event an error. This inevitably leads to considerable observer bias and substantial differences in reporting rates by different observers [13]. Many studies, however, have incorporated some form of analysis to decide whether an adverse event has been related to drug therapy – a causality assessment; whether the event was ‘preventable’– a characteristic often used as a surrogate for assigning an event to the category of medication error rather than an inevitable adverse drug reaction; and to what extent the event caused, or might have caused, harm – an impact assessment. Bates et al., for example, asked two independent assessors to decide whether an ‘adverse drug event’ (an unsatisfactory term intended to encompass ‘suspected adverse drug reaction or harmful medication error or both’) had occurred and, if so, whether it was an error (‘preventable’) and what its severity was [47]. In another study, two pharmacists independently assessed documentation for possible errors, and a physician then examined all the possible errors identified by either of them [48]. The Naranjo algorithm, designed to improve the assessment of adverse drug reactions, has been used to assess causality [22].

Preventability and error

Preventability has often been assessed by the Hallas criteria [49] or some variation of these; it has sometimes been rated on a Likert scale [50]. The question of hospital admission caused by preventable drug-related harm has been reviewed [51]. However, there is little theoretical underpinning of the notion of preventability in medication errors, and research to demonstrate that errors deemed ‘preventable’ can be prevented seems to be lacking.

Severity of error

Severity has sometimes been divided into two [13], three [52, 53] or four [54] categories, and the between-observer reproducibility of the categorization has sometimes been validated [55]. The US National Co-ordinating Council distinguishes nine categories, and raters agreed closely when using a modified form of this categorization to describe severity [56]. Likert scales [57] and other numerical scales anchored by illustrative statements have also been used to rate severity [58, 59].

Subdivisions of error

Medication errors can, for example, be classified by place where the error occurred [60], the medicine involved [61], the stage of the medication process [62], the form of the error (‘wrong drug, wrong dose, wrong patient …’) [63], or the underlying psychological genesis of the error [62]. Allan and Barker listed 14 different ways of categorizing medication errors [8]. An Australian taxonomy of ‘adverse drug events’‘contains 827 natural categories arranged on branches to a depth of 12 levels’[64].

Defining the denominator

The denominator is a measure of the exposure to the hazard; the risk is the probability that the exposure results in harm. Definition of the denominator is therefore important for both interpretation and comparison. As with other aspects of the problem, there is disagreement, and this is at least partly the result of differences in perspective. When discussion focuses on individual patients, the occurrence of even one harmful event and the total sum of harm are relevant. This does not entirely solve the problem of the denominator, since the hazard to patients is a function of the exposure to harm – the number of medicines, the routes of administration, and the duration of hospital stay, for example, might be important.

In error prevention, the focus of the prescriber, in contrast to the patient, and perhaps the medical insurance organizations, is on the proportion of opportunities for error that in fact result in harmful, or potentially harmful, error. This, too, is a slippery concept. Barker and others have proposed using the ‘total opportunities for error’[8], but this has sometimes been interpreted to mean the number of intended prescriptions plus the number of administered but unintended prescriptions [65]; and sometimes the number of errors occurring at each part of the process; and sometimes both errors per dose and errors per process stage [29]. Errors that occur early in the medication process are more easily detected and intercepted than errors that occur later [66]. In other circumstances, checking processes detect approximately 90% of errors [67]. This implies that not all errors detected at every stage will reach the patient and introduces another difficulty in modelling true error rates. Yet there have been very few studies in which the impact of errors, i.e. the product of the frequency with which an error occurs and the harm that results, has been explicitly calculated. One study in which this was done, and in which the ‘criticality’ was estimated by including the probability of detection in the risk analysis, gave very strong guidance towards safer systems of administering cancer chemotherapy in hospital [68].

Discussion

A major reason for carrying out research in medication errors is to provide methods of reducing the frequency of their occurrence, and mitigating the harm that they do if they occur. Demonstrating that interventions are worthwhile requires comparison. This is impossible, unless the definitions of error are explicit, the methods of counting errors are reproducible, the denominators are known, and the assignment of severity is standardized. Others have made similar points [69,70]. Yet there are pitfalls in all of these areas, as described above. There is still some hope. Disguised observation, and capture–recapture techniques, could improve the ascertainment of error rates. A careful decomposition of the tasks of prescribing and administering medicines may yet help to define the denominator, and incidentally ensure that those parts of the process most susceptible to error are redesigned to be safer. An appreciation of the need to make more careful judgements of the probability and severity of potential harms will also encourage a rational approach to the debate on whether to collect no-harm errors.

None of this should detract from the essential message: medicines commonly cause harm, some of which arises from errors that could be avoided by better systems of prescribing and giving medicines. We should, though, be careful before we decide if a system is ‘better.’

Conclusion

There are great difficulties in comparing data from different studies of medication errors. The information that is used to decide whether errors have occurred depends strongly on the way in which the information is gathered, and many different methods have been used. Under experimental conditions, and when direct disguised observation is used, many errors are detected. Such errors might or might not be propagated, and may or may not cause harm. From the patient's perspective, harm is probably the major determinant of whether an error is important, but for the healthcare organization, errors signify hazard whether or not they cause harm; and safety-conscious organizations will want to remove hazards. One way of deciding on priorities for action is to estimate the potential impact, a method that takes into account both the risk of occurrence and the potential for harm; or the criticality, which in addition considers the probability of detection. This implies that studies should collect as high a proportion of errors as possible; estimate the harm that might occur; and decide how likely it is that the error can be detected. Only if this is done will it be possible to compare two systems of prescribing and administering treatment and decide which is safer.

Competing interests

None to declare.