Evaluation of Implementation, Adaptation and Use of the Recently Proposed Urea Cycle Disorders Guidelines

Abstract

Background: Implementation of guidelines and assessment of their adaptation is not an extensively investigated process in the field of rare diseases. However, whether targeted recipients are reached and willing and able to follow the recommendations has significant impact on the efficacy of guidelines. In 2012, a guideline for the management of urea cycle disorders (UCDs) has been published. We evaluate the efficacy of implementation, adaptation, and use of the UCD guidelines by applying different strategies.

Methods: (i) Download statistics from online sources were recorded. (ii) Facilities relevant for the implementation of the guidelines were assessed in pediatric units in Germany and Austria. (iii) The guidelines were evaluated by targeted recipients using the AGREE instrument. (iv) A regional networking-based implementation process was evaluated.

Results: (i) Download statistics revealed high access with an increase in downloads over time. (ii) In 18% of hospitals ammonia testing was not available 24/7, and emergency drugs were often not available. (iii) Recipient criticism expressed in the AGREE instrument focused on incomplete inclusion of patients’ perspectives. (iv) The implementation process improved the availability of ammonia measurements and access to emergency medication, patient care processes, and cooperation between nonspecialists and specialists.

Conclusion: Interest in the UCD guidelines is high and sustained, but more precise targeting of the guidelines is advisable. Surprisingly, many hospitals do not possess all facilities necessary to apply the guidelines. Regional network and awareness campaigns result in the improvement of both facilities and knowledge.

Introduction

Medical decisions should ideally be based on firm data obtained from well-designed studies. However, many fields in medicine lack such information – especially when rare diseases are involved – rendering the patient’s management subject to individual doctors’ best knowledge and experience as well as to the advice of limited numbers of experts (Arnold et al. 2008, 2009; Spiekerkoetter et al. 2009). In an attempt to overcome such “eminence-based” approaches, guideline projects have been initiated to collect, carefully discuss, and weigh all available information on the basis of standardized methodological strategies and finally propose the highly needed recommendations and guidance for medical decision making (Kishnani et al. 2010; Kölker et al. 2011a). There has been some criticism addressing the standardized methodological strategies applied because those have been developed to evaluate large pharmacological trials but not research in the field of rare diseases (Vockley et al. 2013). While this criticism is appropriate for the SIGN methodology (Scottish Intercollegiate Guideline Network, http://www.sign.ac.uk), the recently proposed GRADE (short for Grading of Recommendations Assessment, Development and Evaluation) approach aims at ameliorating this shortcoming (http://www.gradeworkinggroup.org/).

The list of guidelines is currently steadily growing, and according to the intention of such processes, this should result in some benefit for the patients. First experiences in the field of rare inborn errors of metabolism (IEM), e.g., with the guidelines addressing glutaric aciduria type 1, support this assumption (Heringer et al. 2010). We have recently proposed a guideline for the management of urea cycle disorders (UCDs) (Häberle et al. 2012), a group of rare defects of ammonia detoxification. The process of this guideline development involved the SIGN methodology as of 2009 and a standardized nominal group discussion approach. Target groups for the guidelines were metabolic specialists and nonspecialists in the metabolic field. Aims of the guidelines encompassed harmonization of diagnostic, therapeutic, and disease monitoring strategies in expert centers as well as raising awareness for the disease to enhance the probability for early diagnosis and effective treatment in the nonexpert field.

Targeted implementation of guidelines and the assessment of their adaptation are a complex and not extensively investigated process in the field of rare diseases. This study investigates the general interest in the published guidelines and addresses the following questions: Do the guidelines reach the targeted readers? Do nonexperts consider understandable, reliable, and feasible in everyday practice what experts have written? Do users of the guidelines have the facilities and resources available to follow the recommendations? These are questions important to answer in order to tailor future updates of existing guidelines or in the planning of novel ones (Knai et al. 2012; Legido-Quigley et al. 2012).

Accordingly, we evaluated the efficacy of implementation, adaptation, and use of the UCD guidelines, and hereby the “real-world” impact, within a proportion of the target population by applying different strategies. The aim of this publication is not only to document the current situation with respect to the UCD guidelines, but in addition to motivate colleagues to consider the evaluation and implementation of any planned guidelines already during their development.

Methods

Download Statistics from Online Sources

Implementation strategies included publication of the full guideline text as well as the method report from July 2012 on the website of the “Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften” (AWMF; http://www.awmf.org/leitlinien/detail/ll/027-006.html), the well-established German guideline server. The AWMF follows a strict protocol to determine the classification of guidelines. Downloads from this source are free of charge. Generally, the AWMF guidelines are also accessed from German-speaking physicians from Switzerland and Austria.

Additionally but with a different format and scope, the UCD guidelines were published in an open access online journal specially addressing rare diseases, the Orphanet Journal of Rare Diseases (OJRD; http://www.ojrd.com/content/7/1/32 (Häberle et al. 2012)). To evaluate the guidelines’ quantitative use, the frequency of downloads from both online sources since publication at the AWMF server on July 9, 2012 until May 31, 2014 (data missing for months January and February 2014 due to technical problems; total 632 days) and since publication at the OJRD server on May 29, 2012, until May 21, 2014 (total 722 days) was counted.

Assessment of Facilities Necessary for the Practical Implementation of the Guidelines in Pediatric Units

To estimate whether resources in pediatric units were suitable to enable physicians to act on the basis of the guidelines’ recommendations, a brief Internet-based survey (Survey Monkey; https://de.surveymonkey.com/) was sent out to all pediatric units in Germany (n = 408) and Austria (n = 238). The survey addressed the following issues:

• Hospital category: primary, secondary, or tertiary care center

• Availability of ammonia measurements: 24/7, only during main working hours, or in an external lab only

• Time span between collection of sample and result: <1 h, 1–3 h, 3–6 h, >6 h

• Mode of ammonia determination: bedside test, lab test

• Sample and preanalytical requirements (e.g., capillary or venous blood, cooling of sample)

• Availability of emergency drugs (nitrogen scavengers, l-arginine) in the hospital

Data presented are pooled from both countries.

Guideline Assessment Using the AGREE Instrument

For content and process evaluation of the guidelines, we randomly selected 46 German and 26 Austrian target users from an alphabetical list of the departments of pediatrics, neurology, and neuropediatrics (Germany only). Heads of departments were asked to complete the AGREE instrument (http://www.agreetrust.org/resource-centre/the-original-agree-instrument/) or to delegate this task to one of their coworkers. The AGREE instrument is a standardized tool, developed in 2003 to “advance the science of guidelines,” and has since then become the “international gold standard for practice guidelines evaluation and development” (Burgers et al. 2004).

Regional Implementation of Guidelines

Finally, we implemented the guidelines following a regional networking approach in Vorarlberg, the most western province of Austria. Due to the geographical structure of the area, medical care for patients from the region is offered mainly by regional medical services. The implementation strategy focused on the three pediatric departments in the region. The implementation process encompassed the following phases:

1. Stakeholder consent: setup of a first contact to the heads of the three pediatric departments to obtain consent and support for the implementation.

2. Awareness raising for UCDs and the guidelines: a short presentation of the main basic topics covered by the guidelines was given to the medical staff of the three departments.

3. Discussion of practical questions in every department, such as facilities for ammonia measurement, clinical symptoms which should prone immediate assessment of ammonia, availability of emergency medications, triage settings for (potentially) affected patients, etc.

4. Provision of assistance concerning practical issues (ordering of drugs, laboratory facilities, which forms to fill, how to get access to specialist advice, etc.).

5. Interactive elaboration of triage and decision trees for each department.

The following outcome parameters for the evaluation of the implementation strategy were measured before and after the intervention:

1. Availability of laboratory facilities

2. Number of ammonia measurements/department (mean per 6 months)

3. Availability of and knowledge about emergency drugs (interview medical staff)

Results

Download Statistics from Online Sources

Download numbers for both online available versions of the UCD guidelines are shown in Table 1.

Table 1.

Numbers of downloads from OJRD and AWMF

Time	Orphanet Journal of Rare Diseases (OJRD)	AWMF server
Last 30 days	1,070	328
Last 365 days	10,896	2,882
Total	18,817	3,894
Total per daya	26.1	6.2

^aTotal days 722 and 632 for OJRD and AWMF, respectively

Remarkably, there was an increase in downloading over the time indicating an ongoing and even growing interest in the guidelines (Fig. 1). However it was impossible by limitations of the method to differentiate between first and repeated downloads from single users.

Fig. 1.

Number of downloads per quarter from the German AWMF guideline server is shown for the period from July 2012 until May 2014

Assessment of Facilities Necessary for the Practical Implementation of the Guidelines in Pediatric Units

The overall response rate to the Internet-based survey was 10.4%; response rates were 14% in Germany (57 responses from 408 addressants) and 4.2% in Austria (10 responses from 238 addressants). Respondents represented primary care hospitals in 30%, secondary in 9%, tertiary in 46%, and tertiary at university in 15%. Detailed results are summarized in Table 2.

Table 2.

Survey on resources in pediatric units (pooled data from the 67 responding hospitals)

Availability of ammonia measurements	24/7	Only during main working hours		In an external lab only
	82%	9%		9%
Time to result after sample collection	<1 h	1–3 h	3–6 h	>6 h
	63%	27%	9%	1%
Mode of ammonia determination	Bedside test	Lab test	Lab test during the day but bedside test outside normal working hours
	1%	96%	3%
Sample requirements a	Capillary		Venous
	9%		94%
Preanalytics	Cooling of sample	Fast transport	Lab informed	Special tubes	Special sampling	Special sample transport
	60%	88%	36%	25%	16%	9%
Availability of emergency drugs	On the ward	Available within 1 h		Not available		Unknown
	44%	13%		34%		9%

^aSome labs accept both types of samples

Remarkably, emergency drugs were not available in 12 of 31 tertiary nonuniversity hospitals (39%) and in 10 of 20 primary care hospitals (50%).

Guideline Assessment Using the AGREE Instrument

After sending out the request to complete the AGREE instrument to a total of 72 colleagues (46 in Germany, 26 in Austria), we did not receive any response from 64 addressants. Two colleagues returned the completed form without queries. Six addressants expressed that they considered themselves not competent to fill the survey because of not being an expert in the metabolic field. Nevertheless, two of these colleagues agreed to fill in the AGREE instrument after having being reassured that the status of being a metabolic expert was not conditionary.

The low number of only four completed surveys hampers systematic evaluation of the results but is in full agreement with the AGREE research group recommending “that each guideline is assessed by at least 2 appraisers and preferably 4” (http://www.agreetrust.org/about-the-agree-enterprise/introduction-to-agree-ii/preparing-to-use-the-agree-ii/). However, although being in agreement with the AGREE research group, we consider the small number of responses a limitation. Nevertheless, there were two points raised by all respondents: the views and wishes of the patients were said to be not ascertained to their full extent by the guideline developers (question 5; score 2 on a scale of 4 to 1 with 4 being best). In addition, costs that possibly result from use of the guidelines were supposedly not taken into account (question 20; score 2 on a scale of 4 to 1 with 4 being best).

Regional Implementation of Guidelines

In the region of Vorarlberg, 52 ammonia measurements had been carried out during 6 months before implementation of the guidelines in two of the three regional departments. In the first 6 months following the intervention, number of ammonia measurements increased to 106.

In the third department, ammonia measurement had not been available in-house but had to be sent to an external laboratory with a turnaround time of >24 h. To the memory of the medical staff, ammonia measurement had “very rarely to never” been ordered. Following the intervention, measurement of ammonia within 1 h time (24/7) has been established in the hospital lab.

Before the intervention, medical staff in all pediatric departments was not familiar with emergency drugs for the treatment of hyperammonemia, and access to the drugs was not regulated. In the context of the intervention, a list with drugs, details of drug preparation, and dosages to be used in hyperammonemic patients has been established on the basis of the guideline recommendations. Drugs have subsequently been stored in one hospital pharmacy and are available on request 24/7 with a maximal delay of two hours for all three departments. On all intermediate and intensive care units, in-house storage of basic emergency drugs has been instituted. With regard to triage and patient pathways, regional resources were discussed, and it was agreed that since the most effective extracorporeal detoxification technology was not available in the region, patients with persistent or progressive hyperammonemia must be referred to a tertiary institution in due time. In addition it was agreed that advice from a metabolic expert should be obtained as early as possible in the process, and contact details were made available for the clinicians.

In the 6 months before the intervention, one child with neonatal hyperammonemia had been diagnosed on day three after admission. Since implementation of the guidelines, one neonate with acute neonatal hyperammonemia has been identified. Ammonia measurement has been conducted within two hours after admission, and subsequently metabolic specialist advice has been obtained. Treatment has been initiated according to the guidelines, and the patient was transferred to a tertiary center due to progressive hyperammonemia.

Discussion

Guidelines are produced in the intention to guide medical decisions in patient care. Readers ideally expect positive effects of specific recommendations on clearly defined (outcome) parameters. In children with glutaric aciduria type 1, a significant benefit is seen when treatment during acute illness adheres to the guideline recommendations (Heringer et al. 2010). Such an evaluation requires detailed knowledge of the natural course of the disease and patients’ perspectives and the formulation of relevant and meaningful outcome parameters. Examples of possible outcome parameters include the survival rate of patients with neonatal presentation, the frequency of hyperammonemic crises during the course, the delay in diagnosis, or the long-term intellectual outcome. Respective data are so far only randomly available for the UCD population. The lack of data underlines the importance of currently ongoing UCD registry projects in the USA, Europe, and Japan intending to pool data from large patient populations systematically (Kölker et al. 2011b; Tuchman et al. 2008).

On the basis of the frequency by which the UCD guideline paper was accessed and downloaded from the open access available online sources, the paper was and still is categorized as “highly accessed” in the Orphanet Journal of Rare Diseases. In the case of the AWMF guideline server, download numbers constantly increased during the first two years after publication (Fig. 1) indicating that the knowledge about the availability of the UCD guidelines was (and still is) growing. Nevertheless, given the rarity of UCDs, the strong and ongoing interest in the guidelines indicated by high and constant download numbers is remarkable. However, it must be kept in mind that repeated downloads from single users could not be differentiated from first downloads. In addition, we cannot infer from these data whether download resulted in any practical consequences. Likewise, it remains open whether readers belong to the expert or nonexpert groups.

The survey sent to pediatric hospitals revealed some remarkable results concerning the availability of ammonia measurements and emergency drugs as well as preanalytical standards. In only 82% of the hospitals, ammonia measurement can be performed at all times, while 9% of the hospitals offer this service only during working hours, and another 9% need to send the sample to an external lab. This seems to be a critical issue since a relevant proportion of acutely hyperammonemic patients will in these circumstances not receive timely analysis of plasma ammonia. This fact automatically results in a delay in diagnosis and treatment initiation. In addition, there seems to be wide variation of preanalytical procedures in use with cooling of the blood sample and requirement to beforehand inform the lab in 60% and 36% of institutions, respectively (Table 2). Another finding from this nonrepresentative survey is even more worrying: in 34% of the hospitals, emergency drugs (nitrogen scavengers and L-arginine) for the treatment of hyperammonemia are not available. Among the hospitals without in-house drug availability, 39% are tertiary nonuniversity hospitals and thus by definition dedicated to care for patients with complex, severe disorders such as UCD. It should be noted however that this survey does not represent a pre- versus post-guideline evaluation. We like to highlight this shortcoming mainly as a suggestion for future guideline groups when authors could (and should) perform a true pre- versus post-evaluation.

Implementation of guidelines into clinical practice resulting in the induction of behavioral change is challenging to achieve (Gross et al. 2001; Legido-Quigley et al. 2012). Publishing guidelines – even on open access platforms – is one of the least effective interventions in that respect (Gross et al. 2001). If enhancement of awareness toward the value of ammonia measurements in a variety of clinical conditions is the target, guidelines only available by searching actively for the term “hyperammonemia” will not be effective. Connecting the guideline to more clinical, symptom-oriented search terms (e.g., impaired consciousness, coma) might help to ameliorate this shortcoming in the future.

Presentation by expert talks likewise often fails to induce changes in clinical practice (Gross et al. 2001). Along this, a recent suggestion to improve surveillance for hyperammonemia by introducing “an electronic medical record-based tool to assist physicians in the detection of hyperammonemia” deserves attention (Vergano et al. 2013). The implementation of such warning systems may prove superior efficacy if compared to education-based strategies classically intended by expert talks.

Our experience in implementing the UCD guidelines in a regional medical system underlined the finding that implementation of guidelines by networking activities and interactive approaches is the most effective strategy to induce behavioral change in everyday practice. This can be attributed to the active involvement of target groups and to the potential to adapt methods to local resources or local resources to yet unmet medical needs as well as to the lowering of thresholds to seek expert advice (Gross et al. 2001).

Conclusion

The data obtained by our evaluation indicate that interest in the UCD guidelines is generally high and sustainable. Nevertheless, more precise targeting of the guidelines is advisable. In the future, registry data may develop into a sound basis for significant outcome parameters the guidelines attempt to change. With regard to implementation of guidelines for a group of rare diseases, our experience suggests that a regional network and awareness campaign approach improve medical care. Sustainability of improved awareness, patient care, and outcome are parameters of interest for future research in this field.

Compliance with Ethics Guidelines

Conflict of Interest

Johannes Häberle and Martina Huemer declare that they have no conflict of interest.

Animal Rights

This article does not contain any studies with human or animal subjects performed by the authors.

Contributions of Each Author

J. Häberle and M. Huemer have together planned and performed the conception, design, analysis, and interpretation of data and drafted the article and revised it.