CareTrack Kids—part 3. Adverse events in children's healthcare in Australia: study protocol for a retrospective medical record review

Abstract

Introduction

A high-quality health system should deliver care that is free from harm. Few large-scale studies of adverse events have been undertaken in children's healthcare internationally, and none in Australia. The aim of this study is to measure the frequency and types of adverse events encountered in Australian paediatric care in a range of healthcare settings.

Methods and analysis

A form of retrospective medical record review, the Institute of Healthcare Improvement's Global Trigger Tool, will be modified to collect data. Records of children aged <16 years managed during 2012 and 2013 will be reviewed. We aim to review 6000–8000 records from a sample of healthcare practices (hospitals, general practices and specialists).

Ethics and dissemination

Human Research Ethics Committee approvals have been received from the Sydney Children's Hospital Network, Children's Health Queensland Hospital and Health Service, and the Women's and Children's Hospital Network in South Australia. An application is under review with the Royal Australian College of General Practitioners. The authors will submit the results of the study to relevant journals and undertake national and international oral presentations to researchers, clinicians and policymakers.

Strengths and limitations of this study.

• Data will be collected from a range of healthcare practice types (hospital, general practice and specialists).

• Collecting data from a range of healthcare practices and classifying the types of adverse events encountered will allow priorities to be set as to where improvement efforts are needed to reduce harm to children.

• Potential selection bias for experts involved in the development process of adverse event triggers.

• The use of a single surveyor to review the medical record.

Introduction

A high-quality health system should deliver care that is free from harm.¹ However, population studies have shown that 10% of adult hospital admissions are associated with adverse events (AEs).^2–4 For children, a leading hospital in the USA found 37 AEs per 100 admissions,⁵ while a multicentre Canadian paediatric study using a validated tool showed that 9.2% of patients experience an AE, with higher rates recorded in academic centres when compared with community settings.⁶ Few large-scale AE studies have been undertaken internationally in children's healthcare, and none in Australia.

Retrospective medical record review (MRR) can provide data on the frequency and type of AEs.⁷ One method, the Global Trigger Tool (GTT), initially developed in 2003 by the Institute of Healthcare Improvement (IHI) for use in hospitals,⁸ has been modified for paediatric hospitals,^{5 6 9–11} neonatal intensive care units,¹² paediatric intensive care units,¹³ paediatric otolaryngology¹⁴ and primary care.^15–17 For general paediatric inpatients in North America and Europe, the GTT has detected 6–37 AEs per 100 admissions.^{5 6 9 10} There are no published studies using a GTT in Australian paediatric care.

This research project (‘CareTrack Kids’—CTK) involves three related aims and studies: part 1—developing a set of clinical indicators for common paediatric conditions; part 2—measuring the appropriateness of paediatric care in Australia using these clinical indicators; and part 3—this study—measuring the frequency and types of AEs encountered in Australian paediatric care during 2012 and 2013. Study protocols describing the methodology of the other two CTK studies are presented in separate protocol papers.^{18 19}

This study is novel in that data will be collected from a range of healthcare practice types (hospital, general practice and specialists), which has never been undertaken in adult or paediatric settings. Collecting data from a range of healthcare practices and classifying the type of AEs encountered will allow priorities to be set as to where improvement efforts are needed to reduce harm to children.²⁰

Methods and analysis

There are nine components to the CTK AE study protocol (figure 1). Elements of component 1 (develop inclusion criteria and sampling strategy), component 5 (recruit healthcare practices) and component 7 (train and quality check data collectors and measuring inter-rater reliability (IRR)) are described in more detail in the part 2 CTK appropriateness of healthcare delivered study protocol.¹⁸

Figure 1.

Components of the CareTrack Kids (CTK) adverse event study.

Component 1: develop definitions, inclusion criteria, sampling strategy and tools

Definitions

This study will use the definitions (box 1) for patient safety from the WHO's International Classification for Patient Safety (ICPS).²¹ They are recent, have been subjected to formal development work and are inclusive with few caveats. They also broadly align with those used for the National Health Service (NHS) Institute for Improvement and Innovation's Paediatric Trigger Tool,²² which facilitates making comparisons between countries.

Box 1. Definitions from the International Classification of Patient Safety²¹.

Patient safety is “the reduction of risk of unnecessary harm associated with healthcare to an acceptable minimum. An acceptable minimum refers to the collective notions of given current knowledge, resources available and the context in which care was delivered weighed against the risk of non-treatment or other treatment”

A patient safety incident is “an event or circumstance that could have resulted, or did result, in unnecessary harm to a patient”. The use of the word ‘unnecessary’ in this definition recognises that errors, violation, patient abuse and deliberately unsafe acts occur in healthcare. These are considered incidents. Certain forms of harm, however, such as an incision for a laparotomy are necessary. This is not considered an incident. Incidents arise from either unintended or intended acts. Errors are, by definition, unintentional, whereas violations are usually intentional, though rarely malicious, and may become routine and automatic in certain contexts

An adverse event is “an incident that results in harm to a patient”.²¹ Harm “implies impairment of structure or function of the body and/or any deleterious effect arising there from, including disease, injury, suffering, disability and death, and may be physical, social or psychological”²¹

Healthcare-associated harm is “harm arising from or associated with plans or actions taken during the provision of healthcare, rather than an underlying disease or injury”

Inclusion criteria

Medical records will be reviewed of children aged <16 years managed for at least 1 of 16 CTK conditions,¹⁸ in one of three states of Australia: New South Wales (NSW), Queensland (Qld) or South Australia (SA) during 2012 and 2013.

Sampling strategy

We aim to review 6000–8000 records using a multistage, randomised stratified sampling plan.¹⁸ The number of MRRs that we propose is large compared with previous paediatric GTT studies which range from 50 to 3992 and average 1200.^{5 6 9–14 23} Our relatively large sample size will enable analysis and characterisation of a large number of types of AEs.²⁰

Tools

We will use a modified version of the GTT⁸ to collect data. GTTs use a series of ‘triggers’ to screen the record for a potential AE. The presence of a trigger signals the need for an in-depth review. AEs are coded according to type, and AE rates are calculated. The original IHI version of the GTT has been shown to have high specificity, moderate sensitivity and favourable inter-rater and intra-rater reliability using hospital-based records for adult patients.²⁴

Component 2: collate and ratify triggers

Triggers applicable to Australian paediatric healthcare settings will be developed. We will search the literature using MEDLINE and CINAHL for the term ‘global trigger tool’ to collate existing tools and triggers. Examples of triggers used previously in GTT studies^{6 10 17 25} include: clinical events—transfusion or use of blood products, cardiac and respiratory arrest; use of medications—vitamin K administration, antiemetic use; laboratory investigations—positive blood culture, glucose <2.8 mmol/L, rising blood urea nitrogen or serum creatinine >2 times baseline; and attendance or admissions—transfer to a higher level of care, readmission within 30 days.

A set of candidate triggers will be collated, listing their positive predictive values where available. Using a Delphi process, clinical experts (members of the CTK team and experts nominated by the team) will vote on the most applicable triggers over two rounds. We will develop three sets of triggers—one for hospital use (encompassing emergency department visits and inpatient admissions), one for general practice and one for specialist clinics and offices, including outpatients. Clinical experts will vote on their appropriate trigger sets.

Component 3: resolve data collection requirements and structure

Data collection

A new module will be added to the web-based tool developed for the CareTrack Australia study^{26 27} to include the collection of AEs. The purpose of the tool is to enter data during MRR and enable subsequent data analysis. The tool will be on dedicated laptop computers and support secure data access, data encryption, offline data collection and subsequent database synchronisation (to mitigate against the problems of fire walls and poor internet connectivity in various healthcare settings). The tool will also support validation and confirmation procedures to measure IRR and workflows between researchers with a range of roles in reviewing records (see component 8).

Data fields

If a trigger is positive, the following data fields will be recorded: positive triggers (see component 2); a narrative based on relevant information in the record; AE origin; incident type;^{20 21} contributing factors; causation (box 2); prevention (box 3); level of outcome or severity (box 4); type of patient harm and fields to measure the rate of AEs. These are described below.

Box 2. Healthcare management causation scale^{30 32}.

1=Virtually no evidence

2=Slight to modest evidence

3=Close call (less likely than not)

4=Close call (more likely than not)

5=Moderate or strong evidence

6=Virtually certain evidence

Box 3. Healthcare management prevention scale^{30 32}.

1=Virtually no evidence

2=Slight to modest evidence

3=Close call, <50:50

4=Close call, >50:50

5=Moderate/strong evidence

6=Virtually certain evidence

Box 4. The National Coordinating Council for Medication Error Reporting and Prevention (NCC-MERP) scale for adverse event outcome³⁸.

Category E: Temporary harm to the patient requiring intervention

Category F: Temporary harm to the patient requiring initial or prolonged hospitalisation

Category G: Permanent patient harm

Category H: Intervention necessary to sustain life

Category I: Patient death

The narrative

A brief medical history of the patient and the context and circumstances in which the AE occurred will be recorded.

Origin of AEs

The type of healthcare practice where the AE primarily originated will be recorded. Previous studies have found that AEs may occur at different practices or practice types from where they are detected.^{17 25}

Incident type

The incident type ICPS²¹ classification will be recorded. An incident type is a category made up of incidents of a common nature, grouped because of shared agreed features and is a ‘parent’ category under which many concepts may be grouped (box 5).²¹ The benefit of collecting incident type is that this is the main method of classifying what goes wrong into clinically useful categories.²⁰ A brief free text descriptor of what happened in the form of a ‘Principal Natural Category’(PNC)²⁰ will be included, for example, delayed diagnosis of fractured ulna, rash secondary to an adverse reaction to measles, mumps and rubella vaccine.

Box 5. International Classification of Patient Safety Incident Types²¹.

Clinical administration

Clinical processes and procedures

Documentation

Healthcare-associated infection

Medication/intravenous fluids

Blood/blood products

Nutrition

Oxygen, gas or vapour

Medical device or equipment

Behaviour

Patient accidents

Infrastructure

Building and fixtures

Resources and organisational management

Contributing factors

Contributing factors are the circumstances, actions or influences which are thought to have played a part in the origin or development of an incident or to increase the risk of an incident.²¹ Examples include human factors such as behaviour, performance or communication; system factors such as work environment; and external factors beyond the direct control of the organisation, such as the natural environment or legislative policy.²¹ More than one contributing factor or hazardous circumstance is typically involved in a patient safety incident. We will develop and use the contributing factors classification based on existing systems.^{28 29}

Causation

Causation is the degree to which harm is judged to have been caused by healthcare management, rather than a disease process or injury. Although the GTT⁸ does not use a causation scale, we will apply a scale developed originally for the Californian Medical Indemnity Feasibility Study (MIFS)³⁰ in the 1970s and the Harvard Medical Practice Study (HMPS)^{31 32} and subsequent studies. Other GTT studies have incorporated similar scales into their protocols.^{6 33} Box 6 shows the questions that will guide surveyors in determining the level of causation while box 2 shows the causation scale. The study will only include as AEs those scored between 4 and 6 for causation.

Box 6. Questions to guide surveyors to determine the level of causation^{30 32}.

Is there a note in the medical record which indicates or suggests that healthcare management caused the injury?

Is there a note in the medical record which predicts the possibility of an injury from the patient's disease?

Does the timing of events suggest that the injury was related to the treatment?

Are there other reasonable explanations for the cause of the injury?

Was there an opportunity prior to the occurrence of the injury for intervention which might have prevented it?

Is there recognition that the intervention in question causes this kind of injury?

Did the AE respond to new management to neutralise or modify the effects of former management?

Prevention

Prevention is the degree to which an AE can be avoided.³⁴ As with causation, the GTT⁸ does not use a prevention scale. We will use the MIFS/HMPS scale,^30–32 following other GTT studies.^{6 17 33 35 36} Box 7 shows the questions which will guide surveyors in determining levels of prevention in the HMPS while box 3 shows the HMPS prevention scale.

Box 7. Questions to guide surveyors to determine the level of prevention^{30 32}.

Is there consensus about diagnosis and therapy regarding this case?

How complex was the case?

Was the management in question appropriate?

What was the comorbidity of the case in which the AE occurred?

What was the degree of deviation of management from the accepted norm?

What was the degree of emergency in management of the case prior to the occurrence of the AE?

What potential benefit was associated with the management?

What was the chance of benefit associated with the management?

What was the risk of an adverse event related to the management?

On reflection, would a reasonable doctor or health professional do this again?

There is considerable potential for differing interpretations for both the causation and prevention scales when reviewing records as with any implicit MRR.³⁷ We will implement robust quality assurance processes to limit discrepancies between data collectors by training, measuring inter-rater reliabilities and having regular review meetings (see component 7).

Level of outcome or severity

Patient outcomes relate to the impact on a patient which is wholly or partially attributable to an incident.²⁰ These can be classified according to the type of harm, the degree of harm and any social and/or economic impact.²⁰ The National Coordinating Council for Medication Error Reporting and Prevention (NCC-MERP) scale will be used to score the outcome (box 4).³⁸ This is the most frequently used outcome classification in studies using the GTT. Only categories E–I are related to harm and will be considered an AE (box 4).

Type of patient harm

The type of patient harm will be based on an existing patient safety classification which was developed from incidents and previous AE studies.²⁸ The detailed classification is shown in online supplementary appendix A. The broad categories under clinical harm are pathophysiological/disease-related, injury, psychological/emotional distress, death and cardiorespiratory arrest. More than one category can be used for each AE.

Fields to measure AE rates

Depending on the healthcare practice type, different fields (‘denominator data’) will be collected to enable the AE rates to be calculated. For inpatients, the number of occupied bed days (OBDs) for that admission (component 7) will be collected. For non-inpatients (ie, records reviewed in general practice and specialists’ rooms), the denominator will be the number of consultations reviewed (component 7).

Component 4: undertake pilot study

Given the complexity of collecting data over a geographic area and the need to conduct 6000–8000 MRRs from at least three healthcare practice types, using three sets of triggers, a pilot study will be undertaken. The pilot study will assist in determining the types of issues that might be encountered in evaluating the selection of triggers, the data fields to be collected, the workflow and interpretation of definitions and criteria for both surveyors and reviewers, and the logistical and practical aspects of recruiting healthcare practices, accessing records and extracting, recording, storing and analysing data.

Component 5: recruit healthcare practices

Major tertiary children’s hospitals, general hospitals (metropolitan and regional), general practitioners, and specialist paediatricians and psychiatrists will be invited to participate in the research to allow MRRs to be undertaken.¹⁸

Component 6: recruit surveyors and reviewers

Two types of researchers are required to complete the final AE data set—‘surveyors’ and ‘medical AE reviewers’. Key selection criteria for both roles will be experience in clinical audit and MRR together with computer literacy. Nurses will be employed to simultaneously act as surveyors for this study and CTK part 2 (appropriateness).¹⁸ We estimate that eight full-time equivalent staff will be required. The selection process will involve an aptitude test using triggers and detection of AEs in artificially constructed medical records.^{26 27}

Medical practitioners will be recruited as ‘medical AE reviewers’ to undertake a confirmation review of the information collected and recorded by the surveyor. Six doctors with paediatric experience and who fulfil the key selection criteria will be recruited.

The use of a single surveyor is a modification of the original GTT method⁸ which used two surveyors and one reviewer per MRR. While some GTT studies have used the original method,^{5 11 33 35 39} a variety of other models have been used including one surveyor and two reviewers,³⁶ computerised screening,⁴⁰ and one surveyor and one reviewer.²³

Component 7: train and quality check data collectors and measure IRR

Details on educational interventions, instructions and teleconference supports to align rules and reduce variability between surveyors and medical AE reviewers are outlined in the CTK appropriateness protocol.¹⁸ Dual MRRs will be undertaken for a sample comparing findings to those of a CTK research team member.¹⁸ IRR will be measured at a number of points including between dual surveyor reviews for a sample, between surveyor and medical AE reviews with respect to whether an AE has occurred, incident type, and causation, prevention and outcome scores.

Component 8: undertake medical record reviews

MRRs will be conducted for a randomly selected hospital index admission. All general practice and specialist consultations undertaken during 2012 and 2013 will be reviewed. All available information relating to the index admission will be reviewed including discharge summaries and letters.⁸

If no triggers are detected, no further action will be taken by the surveyor and the review will be considered complete. If one or more triggers are detected, the surveyor will undertake an in-depth MRR to search for AEs. If a surveyor detects a potential AE, they will record all data fields outlined in component 3.

When a surveyor has recorded a potential AE, the web-based tool will electronically notify a medical AE reviewer (see component 3). The reviewer will then be able to securely enter the web-based tool, review the information supplied by the surveyor, and provide a determination as to the presence of an AE. For confirmed AEs, a medical AE reviewer will record incident type, and causation, prevention and outcome scores.

Component 9: analyse data

Data capture will be structured to allow identification of incident types (see component 3) and calculation of AE rates. CIs will be calculated and stratification will be undertaken by healthcare practice type.

The primary outcome measures for inpatients will be the number of AEs per 100 admissions, the percentage of admissions with an AE and the rate per 1000 OBDs. We are collecting all three measures to allow comparisons with other GTT studies as there is no consensus on which metric to use. The first two are likely to be more intuitive for the lay public and policymakers, while the latter is a more accurate indicator of risk exposure and may be more acceptable to clinicians. For general practices and specialists, the number of AEs per 100 consultations will be calculated. Secondary outcome measures will include scores for preventability, causation, contributing factors and outcomes (severity).

Ethics and dissemination

Ethics

Relevant Human Research Ethics Committee (HREC) approvals have been secured and Site Specific Approvals will be sought and received prior to participant and healthcare practice recruitment and MRRs in all jurisdictions, authorities and health services. Single ethical review has been obtained from a lead HREC in each state in order to provide ethical approval for the hospitals within that state. The lead HRECs include: Sydney Children's Hospitals Network (15 NSW hospitals), Queensland Royal Children's Hospital (12 Qld hospitals) and Women's and Children's Hospital Network (8 SA hospitals). Site-specific approvals will be sought from each hospital. The Royal College of General Practitioners National Research and Ethics Evaluation Committee application is under review.

As part of the HREC application, we are proposing that patient and individual healthcare practice consent be waived as it complies with the NHMRC “Guidelines approved under Section 95A of the Privacy Act 1988”⁴¹ and the NHMRC Chapter 2.3.10 “Qualifying or waiving conditions for consent”.⁴² In summary, the study involves: minimal risk to healthcare practices and participants and cannot be achieved without access to records; with dispersed geographic regions across three states, the large number of healthcare practices and records (6000–8000), it is logistically difficult to obtain consent; information is retrospective and there is no likely reason patients would not consent; data are entered directly onto a database which will not contain personal information; and only aggregated data will be disseminated.

Statutory immunity

Statutory immunity protects from disclosure any identifying information obtained through an approved quality assurance activity.⁴³ CTK has applied to the Federal (Commonwealth) Minister for Health for statutory immunity under Section VC of the Commonwealth Health Insurance Act 1973.

Dissemination

We will submit the results of the study to relevant national and international journals with the intention of publishing the results widely. Also, we will make national and international oral presentations to stakeholder groups including those involving patients, researchers, clinicians, managers and policymakers.

Discussion

A selection bias may be introduced as practices which agree to participate in the study may have a higher rate of participation in research, proactive audit and existing feedback processes. These factors may affect AE rates or the quality of AE documentation in the medical record. The use of a single reviewer to undertake the MRR may also be a limitation. Dual surveyor reviews and measurement of IRR will be undertaken to minimise this limitation.

In conclusion, this study will measure the frequency and types of AEs encountered in Australian paediatric care in a range of healthcare settings. There have been few large-scale studies of AEs undertaken in children's healthcare internationally, and none in Australia. This study will allow priorities to be set as to where improvement efforts are needed to reduce harm to children.