Measuring evidence-based clinical guideline compliance in the paediatric intensive care unit

Abstract

Background

Evidence-based clinical care guidelines improve medical treatment by reducing error, improving outcomes and possibly lowering healthcare costs. While some data exist on individual guideline compliance, no data exist on overall compliance to multiple nuanced guidelines in a paediatric intensive care setting.

Methods

Guideline compliance was observed and measured with a prospective cohort at a tertiary academic paediatric medical-surgical intensive care unit. Adherence to 19 evidence-based clinical care guidelines was evaluated in 814 patients, and reasons for non-compliance were noted along with other associated outcomes.

Measurements and main results

Overall facility compliance was unexpectedly high at 77.8% over 4512 compliance events, involving 826 admissions. Compliance varied widely between guidelines. Guidelines with the highest compliance were stress ulcer prophylaxis (97.1%) and transfusion administration such as fresh frozen plasma (97.4%) and platelets (94.8%); guidelines with the lowest compliance were ventilator-associated pneumonia prevention (28.7%) and vitamin K administration (34.8%). There was no significant change in compliance over time with observation. Guidelines with binary decision branch points or single-page decision flow diagrams had a higher average compliance of 90.6%. Poor compliance was more often observed with poor perception of guideline trustworthiness and time limitations.

Conclusions

Measuring guideline compliance, though onerous, allowed for evaluation of current clinical practices and identified actionable areas for institutional improvement.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Clinical practice guidelines improve and standardise treatment, optimise patient care and may reduce mortality and healthcare costs. Yet, clinical guidelines are underused in clinical settings.

WHAT THIS STUDY ADDS

• In prospective observation of 19 nuanced and complicated clinical care guidelines as they applied to 814 paediatric intensive care unit patients, compliance varied widely between guidelines and did not change over time with known observation. High compliance was seen in guidelines with easy readability and multidisciplinary stakeholder engagement, while poor adherence was related to perception of guideline trustworthiness and time to complete guideline.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• Clinical guideline compliance can be measured even with multiple nuanced guidelines. Although onerous, this provides valuable feedback for clinical quality improvement.

Introduction

The disease and complexity of paediatric critical care is increasing, and evaluation and management is constantly evolving. There is a growing need and clear benefit in up-to-date standardised evidence-based clinical practice guidelines (CPGs) to support clinicians.^1–3 CPGs improve and standardise treatment, optimise patient care and may reduce mortality and healthcare costs.^4–6 Heterogeneity in care contributes to adverse patient outcomes.⁷ Yet, guidelines remain underused in clinical settings.⁸ Challenges in standardised guideline implementation are not only in the development of an effective and evidence-based protocols but also in ensuring reliable uptake. Possible barriers include a low number of internationally recognised CPGs applicable to paediatric critical care,⁶ clinician perception of lack of guideline trustworthiness,⁹ reliance on passive dissemination¹⁰ and paucity of compliance measurement data. Several interventions may improve guideline trustworthiness and clinician compliance, such as guideline appraisal tools, local development and implementation strategies, standardised checklists^{5 9 11} and active dissemination through local educators.¹⁰ However, no initiatives have proven completely effective.^{12 13}

Our paediatric intensive care unit (PICU) has developed and implemented multiple evidence-based CPGs.^{3 14–16} Once introduced, there were no formal processes for ongoing staff education to ensure continued uptake or compliance. Therefore, we designed this study to examine the compliance with a complex and comprehensive set of CPGs. The primary objective of this study was to investigate the degree of compliance to multiple complicated guidelines in the PICU. We hypothesised an increase in compliance as the study progressed in response to observation.^{17 18} Secondarily, our goal was to describe associations on patient outcomes to identify areas for improvement.

Materials and methods

Design and setting

The design of this study was a prospective cohort, in the medical-surgical PICU at a tertiary care paediatric hospital. The manuscript adheres to ‘The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies’ format¹⁹ included as online supplemental material. Multiple evidence-based, internally reviewed CPGs are available on a regional, secure website, easily accessed by any medical care clinician within the hospital.

Adherence to elements of 19 different clinical practice guidelines (table 1) were prospectively measured by two research coordinators by observation on patient rounds with limited direct interaction, no prompting given and through chart review, for 1 year. There was no predetermined sample size, as the goal was to sample throughout a year to capture the breadth of PICU admissions and management. All patients consecutively admitted to the PICU were included. The only exclusion criteria were PICU length of stay less than 24 hours. Recruitment occurred on admission, and if a patient was discharged within 24 hours of admission they were removed from the study. Clinical staff were aware of observation. Compliance to each guideline was measured in real time on specified days of admission (see table 1). Coordinators collected additional data related to patient outcomes during this same period.

Table 1.

PICU compliance guidelines and criteria

Guideline	Days	Compliance measures
Airway Intubation	1	Standardised checklist attempt or completion
ARDS*	1,3	Tidal volume ml/kg average day 1 and 3: 4–6 mL/kg
Ventilator-assisted pneumonia†	1	Head of bed 30°–45°, ventilator tubing below head, mouth care (including teeth brushing, chlorohexidine), in-line suction applied and no stress ulcer prophylaxis
Bronchiolitis	1,3	Epinephrine, salbutamol, 3% saline, corticosteroids not ordered. No heliox used. Was antibiotic usage appropriate = yes if consolidation reported on CXR by radiologist.
Asthma	1	Receiving inappropriate antibiotics?
Intravenous fluids	1, 3, 5	≤75% of total fluid intake ordered at 24 and 72 hours
Sedation analgesia	1,3	Correct first and second choice agents days 1 and 3 Use of delirium score and assessment for CAPD score >9
Sepsis	1	Initiation of inotrope after 60 mL/kg fluid if unstable blood pressure and/or perfusion. First dose antibiotic within 1 hour in PICU, if not in ED
Status epilepticus	1	Continuous EEG in situ as per protocol
Stress ulcer prophylaxis	1, 3	Appropriate administration of medication from defined risk factors Correct choice of medication: yes/no
Feeding	1	If no contraindication, nasogastric feeding tube inserted and feeds started ≤ 4 hours postadmission at 10 mL/kg/day
CVL/ART	1, 3, 7	CVL/ART insertion bundle checklist done day 1 CVL maintenance—days 3+7: Dressings, cap changes and alcohol prep, line changes. Antibiotic coated if prolonged access or immunosuppressed
Febrile	1,3	Tylenol or Ibuprofen not given in IDSS; external cooling used for temperature >38°C
Packed RCC transfusion	1	Transfusion only when; Hgb>70 only if meets indication of low cardiac output with end organ dysfunction or chronic hypoxia <85% for >1 hour
Platelet (plt) transfusion	1	Transfusion only when; plt<10 except in consumptive disease, antiplatelet antibodies plt<50 in active bleeding, invasive procedure, intrinsic plt disorder plt<100 if CNS injury, neurosurgical procedure, intrathecal catheter Normal count, with bleeding from a qualitative cell dysfunction
Fresh frozen plasma transfusion	1	Transfusion only when high risk surgical intervention and INR>1.5 PTT >38; low-risk surgical intervention and INR>2.0 PTT >2×upper limit; massive packed RCC protocol; a known bleeding disorder and a delay in receipt of factor concentrate; consumptive coagulopathy with bleeding or INR>2 PTT >2×upper limit; major bleeding in an anticoagulated patient
Vitamin K	1	Only given in mildly elevated INR with liver disease, vitamin K antagonists or severe nutritional deficiencies. No role in active bleeding.
Nitrous oxide (iNO)	1	iNO initiation appropriate based on Oxygenation Index>20 OR FiO2>80% for SpO2>80% or evidence of right heart failure iNO continuation based on proper response; improved echocardiogram or cardiac output or ↑SpO2>10% or ↑PaO2>20% mm Hg iNO weaning commenced if FiO2<0.65 and SpO2>85%

‘Days’ refer to day of admission compliance was measured. Optional criteria are italicised.

*ARDS defined by the American-European Consensus Conference.³⁴

†Defined by modified CDC criteria.

ARDS, acute respiratory distress syndrome; ART, arterial line; CAPD, The Cornell Assessment for Pediatric Delirium; CVL, central venous line; CXR, Chest X-ray; ED, Emergency Department; EEG, electroencephalogram; Hgb, haemoglobin; IDSS, inotrope dependent septic shock; INR, international normalised ratio; PICU, paediatric intensive care unit; PTT, Partial thromboplastin time; RCC, red cell count.

Guidelines and measure of compliance

All existing guidelines at our institution were reviewed. The guidelines applicable to the 19 most common clinical presentations seen at our institution were included in this study (table 1). Including all 70+ available guidelines in this initial study was not feasible. Using the guidelines applicable to the most commonly seen clinical presentations were selected to minimise potential bias, as guidelines for rarely seen clinical syndromes may artificially increase compliance scoring due to a higher chance of need for reference by care clinicians. Guidelines were gradually introduced over the course of 3 years before study initiation.

Compliance criteria were identified and recorded electronically by research personnel using a confidential REDCap database.²⁰ A compliance score was generated for each patient by choosing which guidelines are relevant to that patient that day and assigning a binary value of ‘0’ for non-compliant and ‘1’ for compliant. A ‘Non-Compliance Reason Score’ (NCRS) was generated when a guideline element was not adhered to. NCRS was scored as; non-compliance reason (NCR1)—good reason as determined by an independent review, NCR2—no time for guideline, NCR3—guideline not believed by care clinician/individual practice variance and NCR4—no reason was given. These scores were determined by multidisciplinary discussion either at the time of scoring, or retrospectively through clinical charting and unit review with attending physicians. If further clarity was required for reason for non-compliance, there was a direct conversation with the attending physician. Relevant NCR was denoted by way of Boolean values. Each guideline had mandatory and optional criteria determined by evidence-based consensus (table 1). All mandatory criteria had to be met, or else deemed non-compliant. If optional scoring criteria were met, they were included in the compliance scoring. Relevant outcome measures were collected for each guideline (see table 2).

Table 2.

Guideline compliance and outcome measures

Guideline	ID	Event (n)	Comply (%)	Outcome measures per event
Airway intubation checklist	AIC	94	86.2	Desaturation<85% or drop>15% No (63.6%), yes (24.4%), no data (12%) ↓BP? No (84.9%), yes (1.9%), no data (13.2%) CPR? no (88.9%), yes (0.0%) no data (11.1%)
Acute respiratory distress sydnrome	ARDS	15	73.3	Mortality n=3; 20%
Ventilator-associated pneumonia prevention	VAP	415	28.7	8 events isolated a pathogenic organism.
Bronchiolitis	BRON	80	74.9	Inappropriate therapies given (day 1/day 3/both) Nebulised epinephrine: (3.7%/0%/0%) Nebulised salbutamol: (50%/33.3%/25.9%) Nebulised 3% saline: (57.4%/16.7%/14.8%) Corticosteroids: (74.1%/37%/25.9%) Antibiotics: (40.7%/64.8%/72.2%)
Asthma	ASTH	64	79.7	Inappropriate antibiotics in 11.9%.
Intravenous fluid	IVF	828	90.8	CRRT for fluid overload: 0.0036% (n=3)
Sedation analgesia	SEDN	477	65.7	Intubated: 48.9%, NIV: 24.1%, none: 27% WAT>4: no: 15%, yes: 5.9%, no data: 79.1%
Sepsis	SEPSIS	23	80.4	Mortality n=1; 4.3%
Epilepsy	EPIL	24	87.5	Seizure control in 1.1–70.3 hours, mean 19.9±22.7 hours
Stress ulcer prophylaxis	STRES	1093	97.1	GI bleeding in 1.7%
Feeding	FEED	544	91.4	None recorded.
Central venous/arterial line	CVL	401	44.6	Pneumothorax: 0% haematoma: 0% bleeding: 2.6% puncture: 0%. CLABSI in 1000 line days: 0.813
Febrile-tylenol	FEBR-TYL	159	58.5	Mortality n=4; 2.5%. temperature out of range in 43.4% patients
Febrile-shock	FEBR-SHOCK	23	50.0	Mortality n=1; 4.3%.
Transfusion-PRBC	TRAN-PRBC	99	89.9	Transfusion outside of threshold: 16.4%
Transfusion-platelet	TRAN-PLATE	97	94.8	Transfusion outside of threshold: 9.3%
Transfusion-FFP	TRAN-FFP	39	97.4	Transfusion outside of threshold: 4.2%
Transfusion-vit K	TRAN- K	23	34.8	Transfusion outside of threshold: 71.4%
Nitrous oxide	iNO	14	90.5	None recorded

Total guideline (abbreviated ID as per table 2) admits, events, mortality and related patient outcomes per guideline. Non-compliance is the adverse outcome in our transfusion protocols, given the true outcome variable is inappropriate transfusions given.

Sepsis had a total of 41 patients/events, but 18 events had insufficient documentation to include in analysis.

↓BP, low blood pressure; CLABSI, central line-associated bloodstream infection; CPR, Cardiopulmonary resuscitation; CRRT, Continuous renal replacement therapy; CVL, central venous line; FFP, fresh frozen plasma; GI, Gastrointestinal; NIV, non-invasive ventilation; PRBC, Packed red blood cells; WAT, Withdrawal Assessment Tool.

Data analysis

Guideline data were manually exported from the confidential server REDCap²⁰ as a comma separated value (.csv) data file and imported into tables in an Oracle 11g Express (V.4.2.0.17.089)²¹ database where data normalisation and compliance tabulations were completed. Some key definitions in our analysis are as follows; ‘admits’ is the number of admissions; ‘patients’ is the individual patients who may have multiple admissions; ‘events’ are events during the admission where guideline compliance was applicable and measured. There may be multiple events during one admission.

Compliance was described by each single event, on a single day; then combined for each guideline to show the proportion of compliance and NCR values, as shown here:

Criteria xompliance=1–∑NCR value.

All events were combined for each guideline to show compliance throughout the year. Further details are provided in online supplemental materials. To test for change in compliance throughout the year under observation and a possible Hawthorne effect, a linear regression was completed with compliance as the dependent variable, and month, events and admissions as predictors, on IBM SPSS V.6 software.²²

Results

CPG compliance

The study recorded 4512 compliance events involving 826 admissions of 814 unique patients between 22 June 2016 and 21 June 2017. General demographics from the study sample are shown in table 3. Compliance values for each CPG are presented in table 4. Compliance values ranged between 28.7% for the ventilator-associated pneumonia (VAP) guideline (415 events; see table 4) and 97.4% for the fresh frozen plasma transfusion guideline (39 events; see table 4) with an overall average annual compliance across all 19 guidelines of 77.8% (figure 1). The guideline with the highest non-comply score for ‘no time for guideline’ was VAP with an NCR2 of 70.1% (table 4). The guideline with the highest non-comply score for ‘guideline not believed by care clinician/individual practice variance’ was Vitamin K administration, with an NCR3 of 52.2%.

Table 3.

General data and demographics of study population

	PICU population (range; mean±SD)
No of patients (n)	814
Age (years)	0–24; 5.3±5.58
Sex	Male: 501, female: 363
# Admissions per patient	1–2; 1.02±0.13
Standardised mortality ratio (PIM3)	0.835
Standardised mortality ratio (PRISM III)	0.898
% Mortalities in total patients	1.8%
Standardised length of stay (PRISM III)	0.813

PICU, paediatric intensive care unit; PIM, Pediatric Index of Mortality; PRISM III, Pediatric Risk of Mortality III.

Table 4.

Compliance for each guideline

Guideline	ID	Admit: patient	Event	Comply	NCR2	NCR3	NCR4
Airway intubation checklist	AIC	82:82	94	86.2	1.1	1.1	11.7
Acute respiratory distress sydnrome	ARDS	9:9	15	73.3	20.0	0.0	6.7
Ventilator-associated pneumonia prevention	VAP	285:283	415	28.7	70.1	0.0	1.2
Bronchiolitis	BRON	54:54	80	74.9	9.0	15.4	0.8
Asthma	ASTH	64:64	64	79.7	4.7	12.5	3.1
Intravenous fluid	IVF	477:470	828	90.8	7.2	0.2	1.0
Sedation analgesia	SEDN	305:303	477	65.7	2.7	3.7	28.0
Sepsis	SEPSIS	23:23	23	80.4	15.2	4.3	0.0
Epilepsy	EPIL	24:24	24	87.5	4.2	4.2	4.2
Stress ulcer prophylaxis	STRES	810:798	1093	97.1	1.5	1.3	0.1
Feeding	FEED	544:538	544	91.4	7.9	0.0	0.7
Central venous/arterial line	CVL	152:152	401	44.6	15.3	0.0	40.0
Febrile-tylenol	FEBR-TYL	137:132	159	58.5	32.1	3.8	5.7
Febrile-shock	FEBR-SHOCK	20:19	23	50.0	50.0	0.0	0.0
Transfusion-PRBC	TRAN-PRBC	61:58	99	89.9	4.0	6.1	0.0
Transfusion-platelet	TRAN-PLATE	15:12	97	94.8	1.0	0.0	4.1
Transfusion-FFP	TRAN-FFP	24:22	39	97.4	0.0	2.6	0.0
Transfusion-vit K	TRAN-K	21:21	23	34.8	13.0	52.2	0.0
Nitrous oxide	iNO	14:12	14	90.5	0.0	7.1	2.4
		Total	4512	78.1	12.7	1.8	7.6

‘Comply’ events are defined as events where there was full compliance to the guidelines, or an established good reason (NCR1) as defined by an objective evaluator. All compliance criteria shown as %.

Sepsis had a total of 41 patients/events, but 18 events were excluded with incomplete data as defined in methods.

CVL, central venous line; FFP, fresh frozen plasma; NCR3, clinician did not believe guideline; PRBC, Packed red blood cells.

Figure 1.

Evidence-based clinical guideline compliance by year and events. (A) Compliance (diamond marker, grey; %), events (black; n scaled×10¹), and admissions (grey; n scaled×10¹) for all guidelines across each month in the data collection period from June 2016 to June 2017. No significant changes in compliance by month throughout the study. Dotted line indicates total annual average comply score of 77.8%. (B) The number of recorded events per guideline (black) compared with compliance (grey). Events may occur at multiple times during a patients’ admission, and in combination with one another, see table 4 for details. ‘Comply’ defined as full compliance or an established good reason (NCR1) as defined by an objective evaluator. Guideline ID abbreviations on x-axis full definition shown in table 4. ARDS, acute respiratory distress syndrome; CVL, central venous line; FFP, fresh frozen plasma; iNO, nitrous oxide; NCR, non-compliance reasons; VAP, ventilator-associated pneumonia. AIC, airway intubation checklist; BRON, Bronchiolitis; ASTH, Asthma; IVF, Intravenous fluid; SEDN, sedation and analgesia; EPIL, epilespy; STRES, stress ulcer prophylaxis; FEED, Feeding; FEBR-TYL, Febrile-Tylenol; FEBR-SHOCK, Febrile-Shock; TRAN, Transfusion; PRBC, packed red blood cells; PLATE, platelets; K, vitamin K.

The frequency of compliance events varied by guideline, ranging from 1093 events for stress ulcer prophylaxis (table 4; figure 1B) to 15 events for acute respiratory distress syndrome (table 4; figure 1B). Overall, compliance did not vary significantly with volume of admissions throughout the year (F=0.587, p=0.463; figure 1A). The range of compliance from each month average was 73.6%–84.3%, and the range of events per month were 26–462 (scaled to 10¹ on figure 1A), with the event peak in December 2016.

In characteristics of the CPGs themselves, 11 had binary decision branch points or clear checklist, and the average compliance of these was 90.6%, higher than the overall average facility compliance of 77.8%. CPGs requiring multidisciplinary teams (PICU registered nurses, respiratory therapists, pharmacists) for implementation (Airway Intubation Checklist or AIC, VAP, Sedation and Analgesia or SEDN, central venous line (CVL); table 4) had an average compliance of 49.9% compared with 90.1% for guidelines that did not have multidisciplinary involvement.

Outcome measures for each guideline

Adverse outcome measures for each guideline are shown in table 2. Some specific outcomes include; related to the bronchiolitis guideline, bronchiolitis was inappropriately treated with nebulisers (salbutamol in 50%, 3% saline in 57%) and inappropriate antibiotics (40.7%; table 4), with a lower compliance score of only 76%. Guidelines with higher compliance that also had a lower rate of guideline-specific complications included the airway intubation checklist, stress ulcer prophylaxis and intravenous fluid administration (see table 4).

Discussion

Here, we show for the first time, not only was it possible to measure multiple complicated evidence-based clinical practice guidelines; also, these measures of compliance can be compared with reasons for non-compliance and patient-specific outcomes to improve quality of care. Overall, guideline compliance was higher than anticipated. CPG compliance did not change with the presence of an external evaluator over time. Compliance varied widely between CPGs, with practical implications for our centre. These data support measurement of compliance to address barriers to implementation and adherence and provide valuable insight into local practices for possible quality improvement.

Overall facility compliance to all guidelines was comparable to the few studies reporting on guideline adherence.²³ Characteristics common to guidelines with high compliance included easy readability, one page decision flow diagrams or checklists, consistent with previous research,^{11 24 25} with the caveat that time requirements and the complexity of steps irrespective of easy readability, may negatively impact compliance. The sepsis guideline only had 80.4% adherence, despite being one of the few internationally recognised guidelines³ and single-page flowsheet summary. However, it has a higher degree of time complexity, with multiple steps and time-consuming clinical reassessments in-between interventions, which may be why 15.2% of non-compliance events were due to ‘no time’. Ultimately, ease of guideline readability and practicality of implementing straightforward steps appears to contribute to CPG use. Other possible barriers to compliance described in our data included physician perception of guideline trustworthiness, consistent with previous data.⁹ For example, NCR3 (‘did not believe guideline’) scores were 52.2% for Vitamin K administration (table 4) despite following best evidence-based practice. A possible strategy to address this barrier to compliance in this guideline, and others with high NCR3 scores, may be implementing targeted education sessions for clinicians. Further, it may be prudent to reassess current literature to ensure these guidelines are up to date with best practice. Finally, recent data in health knowledge translation has identified involving key stakeholders in a productive way as essential to improving adherence. Emphasis on interdisciplinary collaboration and early onboarding may improve guideline use.²⁶ This may have been a possible barrier to compliance in CVL management, VAP and sedation/analgesia, which all had a high proportion of NCR4 scores (no reason given; table 4). All required multidisciplinary involvement with nursing staff implementation, which was not introduced early in these guidelines. In comparison, the airway intubation checklist had a high rate of compliance and did involve early multidisciplinary stakeholder engagement, which may reflect this principle and encourage early multidisciplinary stakeholder engagement in guideline development in the future.

We did not observe a change in compliance with an external evaluator (figure 1A). Although our data were collected prospectively, we did not share results with the clinical staff in real time. Specifically, if a guideline was poorly adhered to, it was not known until after the event, and direct prompting was not given. In fact, research staff had limited direct interaction with the bedside team during compliance data collection. Interestingly, there are studies showing increased compliance not with an evaluator, but with knowledge of peer practices and peer-to-peer comparison.²⁷ Intradepartment comparison may be a more potent motivator. An interventional trial would provide valuable information on whether changes in these factors were correlated to a change in compliance.

Data collected here on patient outcomes supported existing literature that guideline adherence and standardised checklists can improve patient outcomes. For example, data collected on intubation showed a low adverse event rate when compliance scores were high. This is consistent with previous research on emergency intubation showing that a preprocedural checklist improves outcomes.^{25 28–30} Specifically, in one study, first pass success in neonatal and paediatric patients was shown to improve by to 20% with preprocedural checklist compliance and simulation.³¹ In this frame, it is possible our comparatively low complications may be associated with high preprocedural checklist compliance. Data collected on patient outcomes also can help with local practice and quality improvement. With our descriptive comparison to patient outcomes, we identified areas for local improvement to benefit patients; for example, we identified bronchiolitis remains inappropriately treated, indicating a potential area for targeted evaluation and education initiatives to change local practice with potential patient benefit.

Future directions, in addition to addressing the possible barriers noted above, should focus on guideline accessibility and forced functionality to improve guideline adherence. Lack of a written protocol is shown as a barrier to compliance.³² Our data suggested that the presence of easily accessible electronic written guidelines may improve adherence in our institution. In the future, an electronic repository that automates guideline compliance within electronic medical records would make widespread implementation more feasible. For example, in children with trisomy 21, creating prompts and health maintenance records that track key components of a guideline significantly improved adherence.³³

There are several strengths and limitations to this study. Strengths include the breadth of data collection relative to 19 complicated clinical practice guidelines, over 4512 events. The data collection over the full course of a year with all admissions meant we captured all ‘seasons’ of a PICU clinical practice year, in typical higher and lower volume stretches of time. The manuscript is reported according to the STROBE reporting guidelines. Some limitations include that the overall high mean compliance should be interpreted with the caveat that the guideline with the highest compliance (stress ulcer prophylaxis) also had the highest number of events. In our non-compliance criteria, one of the options was ‘no good reason was provided’; of course, this is inherently subjective, and at the discretion of the evaluator, despite efforts to standardise evaluation. Similarly, data on specific reasons why physicians may not believe in a guideline was not collected. There is potential bias in which criteria were mandatory, and which were optional, in the guideline compliance criteria. To help inform targeted interventions to improve trustworthiness, it would be important to gather qualitative data to better delineate what makes a guideline untrustworthy. Guidelines were gradually rolled out prior to study initiation, and data on which guidelines were more recent, and therefore, may in theory be accessed more, was not available. A limitation to applicability in other centres is the time intensive and costly nature of implementing a guideline educator to record compliance, and with the emerging widespread uptake of electronic medical records, it may be more useful to implement as part of an electronic system. The descriptive nature of the study is valuable for local quality improvement practices and identifying areas for future study, however, is difficult to make correlations from and would benefit from a future interventional study.

Conclusions

In conclusion, we found that it is possible to measure guideline compliance in multiple complicated evidence-based clinical care guidelines. Measuring guidelines compliance helps inform specific quality improvement initiatives and targeted modifications to improve patient outcomes and possible costs. CPG compliance did not change with the presence of an external evaluator, possibly due to the indirect nature of supervision. Compliance varies widely between CPGs, with practical implications for our centre. Comparing guideline adherence with patient outcomes identifies variations in paediatric critical care practice and areas for potential targeted improvement strategies. From our CPG comparisons, possible strategies to improve compliance in the future include educational sessions, easy readability (single page decision flow diagrams or checklist), simplicity and binary decision branch points and early multidisciplinary stakeholder engagement. These data highlight the utility and importance of not only the development of evidence-based CPGs, but measurement of compliance to address barriers to implementation and adherence and improve patient outcomes.

Data availability statement

Data are available on reasonable request. All data relevant to the study are included in the article or uploaded as online supplemental information. All data should be available in the manuscript tables. If any additional data is requested, we will happily provide.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

Consent was waived for this study. This research was approved by Calgary Health Research Ethics Board (CHREB) REB15-0910, approval date 17 July 2015, titled 'Pediatric Intensive Care Guideline Compliance and Outcome Study'. All procedures were followed in accordance with the ethical standards of CHREB and the Declaration of Helsinki.