Using Arden Syntax and Adaptive Turnaround Documents to Evaluate Clinical Guidelines

Abstract

Clinical guidelines translate complex research findings and expert opinion into actionable recommendations. However, the effectiveness of even evidence-based guidelines is rarely tested as a whole in a real clinical environment.

We have developed a decision support system for implementing clinical guidelines in a busy pediatric practice. We have added to this system the ability to randomize patients to receive care with or without system support of the guideline or guideline components. The randomization is part of the Arden Syntax that implements the system logic. The result is a relatively effortless process for testing guidelines, as they are implemented, to assure that they are effective.

We describe the system and the process by which this guideline evaluation functionality was built in, using two guidelines (asthma management and maternal depression screening) both of which have been applied to thousands of patients to date.

INTRODUCTION

Rigorously developed guidelines can translate research findings into actionable recommendations for clinical care.¹ Computer based clinical decision support systems have been recognized as a strategy for implementating clinical guidelines.² However, simply following a guideline may not improve outcomes because many guidelines are based largely on expert opinion.³ Even when a guideline is based on research evidence, the evidence may apply to certain recommendations within the guideline, but the guideline, as a whole, has not been tested.⁴

Most studies of guideline implementation in computer decision support systems address only the effectiveness of the system in leading to guideline implementation,² presupposing that adherence to the guideline will result in improved care. However, computer based guideline implementation, because it can be so effective in changing practice, affords the opportunity to test the guidelines themselves.

We have developed an automated mechanism for studying the impact of guidelines on both clinical care and patient outcomes. It is built into a decision support system that utilizes Arden Syntax and adaptive turnaround documents (ATDs) to deliver guideline driven reminders and tools to clinicians at the point of care.⁵ ATDs are scannable paper forms, tailored to the patient, on which data are recorded and captured through optical character and mark recognition. By randomizing the implementation of the guideline or individual components of the guideline, we are able to set up randomized controlled trials within the clinical practice.

METHODS

We developed a clinical decision support system (CHICA) which implements clinical guidelines for preventive care and disease management.⁵ To test the effectiveness of the guidelines, we built into the implementation a system for randomizing, by patient or physician, whether the decision support was provided. Here we describe the system and how randomization was built into the Arden rules (MLMs) that implement the guidelines.

The CHICA System

CHICA (Child Health Improvement through Computer Automation) is a computer based decision support and electronic record system for pediatric preventive care and disease management. CHICA is used as a front end to the Regenstrief Medical Record System.⁶ However, it is designed to work as a stand alone application or together with another clinical information system.

CHICA's primary user interface consists of two ATDs^{7, 8} that collect the handwritten responses to dynamically generated questions and clinical reminders while easily integrating into care workflows. To determine what information needs to be printed on each ATD, CHICA employs a library of Arden Syntax^{9, 10} rules that evaluate the patient’s records in the underlying Regenstrief and CHICA databases. Since time constraints limit the number of topics that can be addressed feasibly in a given patient encounter, CHICA also employs a global prioritization scheme which limits the printed content to questions with the highest expected value¹¹.

The process begins when registration HL7 messages from our clinic appointment system cue CHICA to begin generating the first of two ATDs. This "pre-screening" form is designed to capture data from both nursing staff and patients immediately prior to the provider encounter. This form has a section for nurses to enter vital signs, and also contains the 20 highest priority questions to ask a child's family at a particular visit. (Figure 1)

Figure 1.

The CHICA Pre-Screener Form (PSF) includes a section at the top with optical character recognition spaces to be completed by clinic personnel and 20 yes-no questions to be completed by the family.

Answers extracted from this form are analyzed alongside previously existing data to generate the content for the second ATD. This provider worksheet (PWS) contains reminders which provide varying levels of patient detail based on the information collected before the encounter. (Figure 2) Each of these reminders consists of a “stem” which explains the reason for the prompt and between one and six check boxes with which the physician can document his or her response to the prompt. The PWS becomes the record of the visit.

Figure 2.

A physician worksheet generated by the CHICA system. The eight prompts in the center of the page include check boxes that are scanned and interpreted by CHICA.

Questions and prompts represent segments of a clinical algorithm ¹², e.g., anemia screening, car safety seat use, screening for domestic violence or lead poisoning (Figure 3). Segments of the algorithm are encoded as Arden rules¹³ in which the “logic” slot represents the decision branches and the “action” slot is either a “do box” or a “decision box” that requires information from the patient or the physician. Finally, a unique priority score is assigned to each rule. For example, an Arden rule representing the outlined portion in figure 3 would be executed for any child between one and three years of age. The logic section would state, “If a lead level exists in the last year conclude false; else if insurance is Medicaid conclude true.”

Figure 3:

Simplified clinical algorithm. Outlined area will become an Arden Rule.

The action of the MLM is to print a physician prompt stating, “The AAP and the CDC recommend drawing a blood lead in children this age who are on Medicaid.” The prompt includes check boxes that document whether a blood lead has been ordered or done previously. The action may also cause CHICA to print a “just in time” handout (JIT). This document can provide additional advice or aid to the physician (e.g., a depression screening tool) or for the patient (e.g., an asthma action plan or educational handout). The JIT is printed along with the PWS and placed on the chart for the physician’s use.

CHICA is used by 12 attending physicians, 16 resident physicians and a variety of medical students on their outpatient pediatrics rotations. The ATD forms are scanned into the computer by nursing or clerical staff after the clinic session. All data collected from both of these forms are ultimately stored as coded clinical observations which are used to drive decision support and can be analyzed retrospectively.

Algorithm Testing Mechanism

CHICA maintains a data dictionary containing a controlled vocabulary of coded terms representing clinical observations. The dictionary maps to standard vocabularies like LOINC or SNOMED when possible. To test algorithms, we created a special study variable term. A study variable is added to the data dictionary any time a guideline or portion of a guideline is to be evaluated. The addition of a study variable causes the system to randomly assign a value to the variable as a “clinical observation” for any patient who has not had the variable assigned previously. This happens in a completely automated fashion. An arbitrary number of values can be assigned to a study variable, such as “control,” “intervention,” or “study-arm-2.”

The value of a study variable can be added to the data and logic slots of an MLM, causing it to fire only for patients who have been randomly assigned a particular value for the study variable. For example, the logic section of an MLM that prints a screening question for depression might include the clause, “if ASTHMA-STUDY = CONTROL, then conclude FALSE” so that the rule will only cause a prompt to print if the patient is not in the control group.

RESULTS

The guideline evaluation mechanism has been pilot tested with two clinical guidelines, dealing with asthma management and maternal depression. Here we describe how these guidelines were implemented and the status of the pilot test.

Asthma

Asthma management guidelines from the National Heart Lung and Blood Institute (NHLBI) recommend (1) that patients’ asthma receive a grade from mild intermittent to severe persistent. The most important treatment implication of the grading is (2) that patients with a grade of mild persistent or higher should use a controller medication such as an inhaled steroid. The guidelines further recommend (3) that all patients with asthma have an asthma action plan that specifies what the patient is to do at different levels of symptom severity. Finally, the NHLBI recommends (4) routine visits for asthma patients at least twice annually.

To implement these recommendations, we began with a case finding rule. For patients who do not have a record of asthma based on the RMRS diagnostic codes, CHICA generates the following question on the PSF: “Does <name> have asthma or wheezing, cough or breathing problems that won’t go away or keep coming back?” If the family answers yes to this question or a diagnostic code for asthma is found in the record, the physician receives a notification and is asked to confirm the diagnosis and grade the asthma (if present) and prescribe a controller if it is persistent (Figure 4). The physician’s confirmation of the diagnosis, asthma grade, and treatment are recorded as observations in the database when the PWS is scanned.

Figure 4.

Asthma case finding and grading uses questions for families and reminders to physicians.

At the same time the PWS prompt to confirm and grade asthma is printed, CHICA generates a JIT handout that reminds the physician how to grade asthma and provides an asthma treatment plan for the physician to give to the patient. (Figure 5)

Figure 5.

The asthma “just in time” handout that shows how to grade asthma and includes an asthma action plan to be given to the patient.

Once it has been confirmed that the patient has asthma, and the asthma grade and use of controller medication has been recorded, CHICA will, at all subsequent visits, check for symptoms (daytime, nighttime or interfering with usual activities) that might suggest the need to upgrade or downgrade the patient’s asthma, evaluate response to therapy or change medication. (Figure 6) If the symptoms suggest the need for a change, the physician is notified with a prompt on the PWS that asks the physician to consider regarding and changing therapy. At this time, a new asthma action plan is also generated.

Figure 6.

PSF questions used to upgrade or downgrade the patient’s asthma, and an example of a PWS prompt that might result from these.

Other rules in CHICA generate PSF questions for families about medication adherence and use of a spacer with metered dose inhalers. Depending on the answers, these may alert the physician.

Since the CHICA asthma guidelines were implemented in August of 2005, 5,185 children have been randomized to either the treatment group (receiving care under the CHICA asthma guidelines, from case finding to disease management) or to usual care without CHICA support for asthma care. Among the 3,860 children in the intervention group, 185 (5%) were confirmed by the physician to have asthma. Among those with asthma, 88 (48%) had persistent asthma. Of those with persistent asthma, 17% had no controller medication, and 8% were non-adherent to their controller regimen.

The current numbers are not sufficient to study outcomes yet. For example, we will need about 200 to see a 10% change in adherence and nearly 300 to see a 10% change in controller use (with 80% power at p=0.05) However, automatic recruitment and randomization will eventually allow us to evaluate clinical improvement.

Maternal Depression Screening

Because maternal depression is prevalent and impairs child development and maternal health behaviors,¹⁴ and because young mothers’ only contact with health services is often their children’s pediatrician,¹⁴ screening for maternal depression is a frequent recommendation. As with asthma, we implemented a maternal depression screening guideline as a series of Arden rules. We randomized families into three groups for whom different levels of guideline support are provided by CHICA to the physician.

The complete guideline implementation includes two screening questions printed on the PSF from the PHQ-2,¹⁵ a screening tool that has been useful in other populations.¹⁶ If either of the screening questions is answered in the affirmative, the physician is alerted to the positive screen and asked to make an assessment of the risk of maternal depression and to document whether a referral to mental health was made. To aid the physician, the program also produces a JIT handout that has patient information about depression and local resources to get help. It also gives the physician a full PHQ-9¹⁷ so that a more compete screening can be done.

Because screening recommendations vary for maternal depression, we have established a more elaborate evaluation approach. We have randomized patients into three groups. The first has all components described above. The second includes the PSF screening questions and the alert to the physician when positive, but no JIT handout. The third includes only a generic reminder to the physician to screen for maternal depression.

So far CHICA has randomized 4,941 patients into three arms of approximately 1,600 patients per arm. Rates of detection are relatively low (a few percent) in each arm, but it is too early to identify differences by study arm.

DISCUSSION

The comprehensive nature of CHICA’s guideline implementation, incorporating data collection, logic modules, physician alerts and data capture from clinicians, creates an ideal environment for testing clinical guidelines, in their entirety, in a real life clinical environment. Alternatively, guidelines can be dissected into their component parts, and these can be tested independently. Moreover, the automated randomization capacity makes it possible to run trials of clinical guidelines relatively effortlessly. Using this approach, we are able to test guideline representations as they are implemented in order to assure that we are getting the clinical benefit we expect. Because the intervention consists of reminders, but clinical decisions remain in the control of the physician, we have not encountered ethical or Internal Review Board issues.

Figure 7.

Screening questions and prompt for maternal depression