Extent, nature and consequences of adverse events: results of a retrospective casenote review in a large NHS hospital

Abstract

Objectives

To estimate the extent, nature and consequences of adverse events in a large National Health Service (NHS) hospital, and to evaluate the reliability of a two‐stage casenote review method in identifying adverse events.

Design

A two‐stage structured retrospective patient casenote review.

Setting

A large NHS hospital in England.

Population

A random sample of 1006 hospital admissions between January and May 2004: surgery (n = 311), general medicine (n = 251), elderly (n = 184), orthopaedics (n = 131), urology (n = 61) and three other specialties (n = 68).

Main outcome measures

Proportion of admissions with adverse events, the proportion of preventable adverse events, and the types and consequences of adverse events.

Results

8.7% (n = 87) of the 1006 admissions had at least one adverse event (95% CI 7.0% to 10.4%), of which 31% (n = 27) were preventable. 15% of adverse events led to impairment or disability which lasted more than 6 months and another 10% contributed to patient death. Adverse events led to a mean increased length of stay of 8 days (95% CI 6.5 to 9). The sensitivity of the screening criteria in identifying adverse events was 92% (95% CI 87% to 96%) and the specificity was 62% (95% CI 53% to 71%). Inter‐rater reliability for determination of adverse events was good (κ = 0.64), but for the assessment of preventability it was only moderate (κ = 0.44).

Conclusion

This study confirms that adverse events are common, serious and potentially preventable source of harm to patients in NHS hospitals. The accuracy and reliability of a structured two‐stage casenote review in identifying adverse events in the UK was confirmed.

Studies across the world have shown that between 3% and 17% of hospital admissions result in an adverse event (defined as any unintended event caused at least partly by healthcare and which resulted in harm), and that between 28 and 75 percent of them are preventable.¹,²,³,⁴,⁵,⁶,⁷,⁸ Only one study, conducted in 1998, has estimated adverse events and their preventability in the UK.¹ The research reported here updates the findings of the UK study on the scale, nature and consequences of adverse events and also addresses several other important issues not previously researched in the UK, such as the accuracy of the screening tool and the inter‐rater reliability of casenote review instruments.

Methods

Setting and sampling

We carried out the present study a large National Health Service (NHS) hospital trust in England between January and June 2005. A sampling frame of all admissions lasting more than 24 h between January and May 2004 in eight specialties was obtained from the hospital information system. A sample of 1000 admissions was calculated to be sufficient to estimate the prevalence of adverse events with a 95% CI of ±2%.⁹ We selected a stratified random sample of 1050 admissions from the eight specialties: surgery, urology, orthopaedics, general medicine, medicine for the elderly, oncology, ENT and ophthalmology (table 1).

Table 1 Number and percentage of admissions with positive screening criteria, agreement for each criterion, and association between positive criteria and presence of adverse event.

	Criterion	n (%)	Agreement, κ	p Value*	Odds ratio (95% CI)*
(1)	Unplanned admission related to previous healthcare management	153 (15.2)	0.64	0.4	1.4 (0.7 to 3.0)
(2)	Hospital‐incurred patient injury, (eg fall, burn, pressure ulcer)	58 (5.8)	0.69	0.001	3.9 (1.7 to 8.8)
(3)	Other patient complications (eg DVT, MI, CVA, PE**)	58 (5.8)	0.56	<0.001	6.9 (3.0 to 15.9)
(4)	Hospital‐acquired infection/sepsis	68 (6.8)	0.59	<0.001	21.7 (10.7 to 44.7)
(5)	Cardiac/respiratory arrest, low Apgar score	7 (0.7)	0.66	0.01	18.5 (2 to 179)
(6)	Adverse drug reaction	18 (1.8)	0.49	0.003	6.5 (1.9 to 22.2)
(7)	Unplanned transfer from general care to intensive care	13 (1.3)	0.66	0.04	10.3 (1.05 to 101)
(8)	Unplanned transfer to another acute care hospital	2 (0.2)	NA	0.9	0 (NA)
(9)	Unplanned return to the operating theatre	4 (0.4)	NA	0.6	2.0 (0.1 to 19)
(10)	Unplanned removal, injury or repair of organ during surgery	5 (0.5)	0.66	<0.001	88 (6.8 to 114)
(11)	Development of neurological deficit not present on admission	4 (0.4)	NA	0.8	1.4 (0.07 to 28.0)
(12)	Inappropriate discharge	25 (2.5)	0.55	0.004	50 (3.4 to 736)
(13)	Unexpected death	33 (3.3)	0.66	0.7	1.3 (0.4 to 4.1)
(14)	Injury related to abortion or delivery	0 (0.0)	NA	NA	NA
(15)	Dissatisfaction with care documented in medical record	21 (2.1)	0.80	0.1	2.9 (0.7 to 12.0)
(16)	Documentation or correspondence indicating litigation	0 (0.0)	NA	NA	NA
(17)	Any other undesirable outcomes not covered above	116 (11.6)	0.54	<0.001	5.3 (2.7 to 10.3)
(18)	Unplanned readmission related to the care provided in the index admission	99 (9.8)	0.78	0.009	2.7 (1.3 to 5.8)
	Admissions with at least one positive criterion	448 (44.5)	0.68

CVA, cerebral vascular accident; DVT, deep vein thrombosis; MI, myocardial infraction; PE, pulmonary embolism.

A confidence score of ⩾4 was used for causal link.

NA, based on 107 admissions to check the inter‐rater reliability, when no criterion was recorded positive, the kappa could not be calculated.

*p Value and odds ratio is for multivariate logistic regression.

Process of medical record review

In stage 1 of the casenote review, five trained nurses screened the patient records, using a tool (review form (RF) 1) consisting of a list of 18 explicit criteria (table 1). Identification of one or more positive criteria was used as an indicator of a potential adverse event and these medical records underwent further scrutiny (stage 2). To assess inter‐rater reliability, a 10% sample of these admissions was independently reviewed by another reviewer using the same tool (fig 1, b). In addition, 20% of admissions where no positive criteria were identified were fully reviewed by medical staff to detect false negatives and calculate the sensitivity of the screening tool in identifying adverse events (fig 1, e).

Figure 1 Flowchart of the casenote review process. AE, adverse event; RF, review form (tool).

In stage 2 of the casenote review, three trained hospital doctors reviewed the records found to have at least one positive criterion in stage 1 (fig 1, c). The doctors used a structured review form (RF2) to judge whether adverse event had occurred, and to assess the type, preventability and consequences of the adverse event. An adverse event was considered to have occurred if the reviewer was confident that:

• there was an unintended event;

• the event resulted in patient harm (prolongation of hospital stay, disability at discharge and/or extra cost of treatment);

• it was caused at least partly by healthcare rather than by disease process alone.

We used a six‐point scale to assess the likelihood of a causation link between the event and injury. A similar six‐point scale was used to assess the likelihood of preventability. The preventability of an adverse event was assessed on the basis of the standard of care expected from an average practitioner in that area.¹ To check the inter‐rater reliability, 90 medical records were independently reviewed by another doctor (fig 1, d).

The RF1 and RF2 review forms used in this study are similar to those used in previous studies.¹,²,³ These review forms are available from the authors on request.

We obtained ethical and research governance approval. All data extracted were anonymised and kept confidential.

Analysis

We calculated the proportion of admissions with adverse events and preventable adverse events, and the sensitivity and specificity of the screening criteria in identifying adverse events, were calculated along with the 95% confidence interval. The number and proportion of adverse events which led to an increased length of hospital stay or a subsequent hospital admission and the number of extra bed days resulting from each adverse event were also calculated. The Cohen κ coefficient was used to assess the inter‐rater reliability.¹⁰ Multivariate logistic regression was used to assess the relationship between the presence of positive screening criteria and occurrence of an adverse event.

The sensitivity of the estimated rate and preventability of adverse events to variation in the raters' confidence in the likelihood of causality and preventability was explored by comparing the results obtained when using a likelihood score of ⩾2 (any evidence for preventability or management causation) and of ⩾4 (likelihood of ⩾50%).

Results

Prevalence of adverse events

Full data were extracted for 1006 admissions (fig 1, a). Using the lower threshold of confidence in causality (any evidence for management causation),³,⁴ we found that 110/1006 (10.9%; 95% CI 9% to 12.8%) admissions had at least one adverse event (total = 136 adverse events). Using the higher threshold of confidence in causality, as in the previous UK study (likelihood score of AE of ⩾4),¹ 87 (8.6%; 95% CI 6.9% to 10.3%) admissions had at least one adverse event (total = 107 adverse events) (table 2). The agreement between doctors on the presence of adverse events was 86% (κ = 0.64). The clinical categories of adverse events are presented in table 3.

Table 2 Number and percentage of preventable adverse events (AEs) using different thresholds.

Preventability confidence score		Adverse events n (%)		Admissions with adverse events n (%)
		Causation score ⩾2	Causation score ⩾4	Causation score ⩾2	Causation score ⩾2
(1)	Virtually no evidence for preventability	67 (49.3)	48 (44.6)	50 (45.4)	36 (41.4)
(2)	Slight to modest evidence for preventability	26 (19.1)	19 (17.8)	21 (19.0)	14 (16.1)
(3)	Preventability not quite likely; less than 50–50 but close call	13 (9.6)	11 (10.3)	12 (10.9)	10 (11.5)
(4)	Preventability more likely than not; more than 50–50 but close call	13 (9.6)	12 (11.2)	11 (10.0)	11 (12.6)
(5)	Strong evidence for preventability	11 (8.0)	11 (10.3)	10 (9.1)	10 (11.5)
(6)	Virtually certain evidence for preventability	6 (4.4)	6 (5.6)	6 (5.5)	6 (6.9)
Total		136 (100)	107 (100)	110 (100)	87 (100)

In admissions which had more than one adverse event the preventability score is for the adverse event which had a higher preventability score.

Table 3 Type of adverse events and preventable adverse events and extra bed days attributable to each type of adverse event.

Type of adverse event	Adverse events n (%)	Preventable adverse events n (%*)	Mean extra bed days per adverse event	SE of mean	Total extra bed days in the sample
Hospital acquired infection	19 (14.0)	2 (11)	7.1	1.5	135
Post‐operative infection	23 (16.9)	3 (13)	7.4	1.6	170
Other operative complications	27 (19.9)	7 (26)	3.5	0.8	94
Diagnostic adverse event	7 (5.1)	4 (57)	6.4	2.0	45
Pressure ulcer	12 (8.8)	2 (17)	7.3	1.6	87
Drug complication	19 (14.0)	1 (5)	5.1	1.2	96
Fall	3 (2.2)	0 (0)	12.3	6.7	37
Procedural adverse event	11 (8.1)	6 (55)	6.6	2.6	73
Clinical management adverse event	10 (7.4)	4 (40)	6.5	2.1	65
Other types of adverse event	5 (3.7)	0 (0)	13.2	7.4	66
Total adverse events	136 (100)	29 (21)	6.5	0.6	868

*Percentages are of the total number of adverse events reported in the previous column; the confidence score for assessing the preventability is ⩾4.

Box 1: Examples of preventable adverse events

• Oesophageal dilation wrongly using an 18 mm balloon (rather than 12 mm), leading to muscle rupture and bleeding

• Avoidable delay in diagnosis of malignant condition

• High‐risk patient with no prophylaxis. Developed deep vein thrombosis and pulmonary embolism.

• Diathermy burn to skin during elective cholecystectomy.

• Bleeding from penis after urinary catheter removed without first balloon deflating.

• Surgical team took over patient care with avoidable delay, did not arrange review after admission, did not call the consultant, inadequate treatment; patient died.

• Common bile duct was perforated during endoscopic retrograde cholangiopancreatography, required urgent open operation with pancreaticoduodenectomy. Patient developed postoperative complications and died.

• Injury to the penile urethra during operation requiring repair and leading to readmission and another operation.

• Intravenous pyelogram taken of a renal patient with impairment the condition, worsening of situation (technical error).

• Inadequate postoperative monitoring, leading to hypovolaemia, collapse and renal failure.

• Spleen was torn during nephrectomy resulting in loss of 6 l blood, removal of spleen, blood transfusion, and antibiotic prophylaxis for life.

• Patient required high dose of opioid throughout the admission. Still taking high dose of opioid on discharge. Drug continued after discharge with high dose. Repeated by general practitioner. Became addicted.

Consequences of adverse events

Increased length of stay

An adverse event increased the length of hospital stay or led to a subsequent hospital stay in 85% of 136 adverse events (88% of 107 adverse events, when likelihood score ⩾4). Adverse events were responsible for a total of 896 extra bed days, ranging from 0 to 45 (0–31) days. On average, adverse events prolonged the length of hospital stay 6.5 (6.0) days per adverse event (SE = 0.6 (0.58)); and 8.0 (7.4) days per patient who experienced an adverse event (SE = 0.78 (0.72)) (table 3).

Patient harm

Impairment or disability which was resolved within a month was caused by 56% of the 136 adverse events (57% of 107 adverse events, when likelihood score ⩾4); 17% (18%) adverse events led to a more serious impairment or disability which was resolved within 6 months; 4% (4%) led to impairment or disability which was resolved between 6 and 12 months; 11% (11%) led to permanent disability; and 10% (9%) contributed to patient death. In 2% of adverse events the reviewers could not assess the consequences of adverse events.

Preventability of adverse events

The estimated proportion of preventable adverse events varied depending on the degree of confidence expressed by the clinical reviewers. When using a likelihood of more than 50% for causation link and preventability (likelihood score ⩾4) 29/107 (27%) adverse events were considered preventable (31% of the 87 admissions). However, using a lower confidence threshold (any evidence for causation link and preventability—likelihood score ⩾2) 69/136 (51%) adverse events were considered preventable (55% of the 100 admissions). Table 2 shows the number and percentage of preventable adverse events when different thresholds were used for identifying adverse events and assessing the preventability. Some examples of preventable adverse events are shown in box 1. The agreement between doctors on presence of a preventable adverse event was 83% (κ = 0.44).

Accuracy and inter‐rater reliability of screening criteria (RF1)

In 44.5% (448) of the 1006 admissions at least one of the 18 selection criteria was recorded as positive (fig 1, b) (95% CI 41.4% to 47.6%, table 1). The sensitivity of the screening tool in identifying adverse events (based on the 10% sample of notes with no positive criteria) was 92% (95% CI 87% to 96%) and the specificity (based on all the records with an RF2) was 62% (95% CI 53% to 71%). Criteria 2, 3, 4, 5, 6, 7, 10, 12, 17 and 18 in table 1 were all independently associated with a significantly increased risk of the occurrence of an adverse event (table 1).

The RF1 screening form was independently completed by two nurse reviewers for 107 records. The agreement for presence of positive criteria was 84% (κ = 0.68). The κ for each individual criterion is shown in table 1.

What is already known on this topic

• Adverse events are common in inpatients and a considerable proportion are preventable and lead to serious patient harm.

• Little work has been done on the extent and nature of adverse events in the UK.

• The accuracy of the screening tools and the reliability of the review forms in identifying adverse events have not been evaluated in the UK.

What this study adds

• Our study confirms the findings of previous studies particularly the previous UK study on the rate, preventability and consequences of adverse events.

• The accuracy of the 18 screening tool and the inter‐rater reliability of the review forms were good in the UK context.

• Inter‐rater reliability of assessment of preventability of adverse event is moderate and needs more research.

• Despite several limitations, casenote review is suitable for identifying the scale and nature of adverse events and monitoring safety improvement programmes.

Discussion

We found that between 8.6% and 11% of hospital admissions were associated with adverse events depending on the degree of reviewer's confidence. This is comparable with rates found in studies using similar casenote review methods in the UK (10.8%)¹ and internationally (7.5% to 12.5%).⁴,⁵,⁶,⁷ One US study reported a lower estimate (3.7%).² This study used a causation threshold of ⩾4 (likelihood of adverse event >50) to identify adverse events. It did not include relevant adverse events which were detected after the index admission.² With a similar high threshold, and by excluding adverse events detected after discharge, our study showed a rate of 7.2%. An Australian study found a higher rate of 16.6%.³ It used a low threshold for causation and the rate included those adverse events which occurred before the index admission but which manifested or were detected in the index admission.³

The reliability of instruments for identifying adverse events depends on the quality of rater training and ongoing monitoring.¹¹ The reviewers in this study were specifically trained and we found a reasonably good agreement between doctors on presence of adverse events (κ = 0.64). In previous research this has ranged from moderate (0.4–0.6)²,³,⁴,⁵,⁶,¹²,¹³ to good (>0.6).¹⁴,¹⁵,¹⁶

The proportion of adverse events estimated to be preventable depends on the degree of certainty expressed by the clinical reviewers. Our estimates are consistent with several other studies which used similar methods (28% to 75%).¹,²,³,⁴,⁵,⁶ Davis et al,⁴ Forster et al⁵ and Baker et al,⁶ using a likelihood score of ⩾4 found that around 37% of adverse events were preventable compared with 27% in our study. Like other studies we found that operative adverse events were more common but less preventable, and diagnostic adverse events were less common but more preventable.¹

McDonald et al suggested that the proportion of preventable adverse events, particularly preventable deaths, has generally been overestimated because of inadequate consideration of other factors such as the severity and complexity of patient disease.¹⁷ Brennan has suggested that in a reasonable proportion of deaths associated with medical error, death would have occurred even in the absence of error.¹⁸ Hayward and Hofer found that “many preventable deaths occurred at the end of life or in critically ill patients in whom death was the most likely outcome either during the hospitalisation or in the coming months, regardless of the care received”.¹⁹

There was less agreement in assessing preventability of adverse events, similar to several previous studies (κ<0.5).²,³,¹³,¹⁹ However, others have reported a higher agreement—Bates et al (κ = 0.7)²⁰ and Thomas et al (κ = 0.8)⁸—although surprisingly Thomas et al⁸ reported a low agreement (κ = 0.4) for identifying adverse events. The difference in reported agreement might be because of differences in the methods and criteria used for assessing preventability or the way the criteria were applied. In Bates et al²⁰ and Thomas et al⁸ the investigators used summaries of adverse events to assess preventability, whereas in our study the record reviewers reviewed the whole medical record to assess preventability. Bates et al²⁰ may have reported a higher inter‐rater agreement also because: (1) they focused on one type of adverse event—drug event; (2) they used a combination of three methods (reporting, speaking with nurses and pharmacists and record review) to collect data; and (3) they used a four‐point rather than a six‐point scale to assess preventability, so reducing the spread of scores. Further research is needed to assess the effect of using summaries or the full medical record to assess preventability and these other factors on the degree of agreement in assessing preventability of adverse events.

Accuracy and reliability of the screening instrument (RF1)

Using the results of the full casenote review as a gold standard,²,³,²¹ the sensitivity of the 18‐item screening criteria was 92%, which is in the range of estimates from similar studies from the USA (84%)¹² and Australia (97.6%).³ The agreement between nurses on presence of screening criteria was good (κ = 0.68) and similar to the Australian study (0.67).³

Our study was carried out in one UK hospital, which may limit the generalisability of its findings. Nevertheless the rate and type of adverse events we found were comparable to a similar study in the UK¹ and ones elsewhere.⁴,⁵,⁶,⁷

Research implications

Medical record review seems to be an efficient and reliable method to provide data, inform and monitor these safety improvement programmes and do so more accurately than current systems for routine incident reporting.²² However, since quality improvement focuses more on those adverse events which are preventable, more research is needed to better conceptualise and reliably assess preventability. Research needs to focus more on how the data collected on adverse events can be fedback effectively and used to inform the design and evaluation of strategies to reduce the scale and consequences of adverse events. Interventions to promote safety, nested alongside longitudinal series of casenote reviews using this same method, may provide useful information for improving strategies.

Conclusion

In the light of the findings from this study and the previous UK study it is now clear that around 8–10% of patients in NHS hospitals may experience some kind of adverse events, of which between 30% and 55% are to some extent preventable. Structured retrospective casenote review was found to be a reliable method to identify adverse events in a large NHS hospital, but the reliability of assessing the preventability of adverse events is typically poor and needs further research. Despite several limitations of casenote review, until a cheaper and more reliable method is designed it should be used for estimating the rate, preventability and consequences of adverse events, and for monitoring safety improvement strategies.