Comprehensive Categorization of Guideline Recommendations: Creating an Action Palette for Implementers

Abstract

Transforming guideline recommendations into executable statements for computerized decision support systems requires a clear understanding of what tasks must be performed. We sought (a) to determine whether a limited set of action types could be defined to comprehensively categorize activities recommended by the majority of clinical guidelines, (b) to describe the relative frequency of these action types, and (c) to create a library of recommendations for future validation activities. We randomly selected test and validation sets of 50 recommendations each from the National Guideline Clearinghouse and randomly extracted 3 recommendations from each guideline. We tested the ability of a preliminary palette of action types to categorize guideline-prescribed activities and expanded it to accommodate several unanticipated actions. Ultimately, the following actions were sufficient to categorize all 405 actions: Prescribe, Perform therapeutic procedure, Educate/Counsel, Test, Dispose, Refer/Consult, Conclude, Monitor, Document, Advocate, Prepare, and No recommendation. These action types can be used to construct a framework for design of clinical decision support systems.

Introduction

Implementation of clinical practice guidelines refers to those activities concerned with incorporating guideline knowledge into systems that are intended to influence clinicians’ behavior toward adherence. Implementation of guidelines has been fraught with difficulties (1–3). Informaticians are particularly interested in implementation of guideline knowledge in computer-based clinical decision support systems. A wide variety of representation schemes have been applied to guideline knowledge (4, 5). Moreover, investigators have demonstrated considerable variability in the translation of guideline text into computable formats (6, 7).

The Guideline Elements Model (GEM) is a hierarchical, XML model of guideline content (8) that has been accepted as an ASTM Standard (E2210-02). GEM represents guideline recommendations as conditionals of the form (IF{condition(s)}…THEN {action(s)} statements) or as imperatives (For all eligible patients, users should {action(s)}). In both cases, the actions described in the guideline recommendation must be identified and transformed into executable activities. This transformation is one in a series of activities in the implementation process that we believe is amenable to standardization.

We hypothesize that there are a finite number of actions that may be triggered by an automated guideline. Understanding these actions is important for the design of clinical information systems that include automated guidelines.

The objectives of this work were: 1) to determine whether a limited set of action types can be defined—in effect, an action palette—that will comprehensively categorize activities recommended by the vast majority of guidelines, 2) to describe the relative frequency of each action type in current guidelines, and 3) to create a library of randomly selected guideline recommendations that can be used for testing and evaluation of models and tools. This paper will describe progress toward these objectives.

Methods

On January 21, 2003 we downloaded all 994 current, evidence-based guidelines that were classified as dealing with diseases (n=892) and mental disorders (n=96) from the National Guideline Clearinghouse (NGC) (9). We assigned each guideline a sequential numeric identifier. Using a random number generator and a random number table, we selected 50 guidelines as a test set. Guidelines that were eligible for inclusion in the test set met the following criteria:

1. Availability: Guideline recommendation text had to be available in electronic form

2. Language: Only guidelines written in English were considered.

3. Focus: Guidelines categorized by the NGC as assessment of therapeutic effectiveness or technology assessment and procedural guideline were excluded.

4. Structure: Guidelines had to include a minimum of three distinct and explicit recommendations, clearly identifiable through formatting or numbering systems.

5. Status: Only the current release of the guideline as of January 21, 2003 was used. Guidelines in the process of being updated were not used.

6. Release date: Between January 1, 1998 and January 21, 2003 or earlier if the guideline was revised or updated within the last 5 years.

Local experience in guideline review and implementation had suggested a set of 7 common actions might be used to categorize clinical activities whose execution was called for by guideline recommendations. These included Consider, Test, Prescribe, Perform procedure, Consult, Educate patient, and Dispose. In phase 1 we sought to identify a set of guideline recommendations to test the adequacy of this empiric set of actions and to help define the categories in an unambiguous and standardizable way.

We used a random number table to select 3 recommendations from each of the 50 eligible guidelines to form a test set. Random selection was employed to assure a variety of recommendation types because of the wide variations in guideline length and format. We noted the often “sequential” character of clinical practice guidelines, where the first recommendations address history, then physical examination, then tests, then therapeutic options, etc. and wanted the test sample to reflect all types of recommendations.

For each recommendation, the authors independently categorized the actions according to the preliminary action palette. Many statements were noted to have complex action requirements, i.e., the same recommendation called for multiple action types. In these cases, each unique action type was counted one time, regardless of the number of sub-statements that appeared. We highlighted situations that required adjustment of working definitions and those that did not fit into any existing category. The authors compared their classifications and discussed all disagreements until they were resolved to the satisfaction of all. The first round of classification necessitated increasing the number of actions in the palette to 12.

In phase 2, a validation set of 150 recommendations was selected in the same manner as the test set and classified independently by each investigator. Guidelines were excluded from the validation set if they appeared in the test set. Again, categorization differences were resolved by discussion. Recommendations potentially requiring additional actions were highlighted.

Results (Table 1)

Table 1.

Preliminary Palette	Test Palette	Occurrences		Validation Palette	Occurrences
Consider	Conclude	12	6.2%	Conclude	2	0.9%
Prescribe medication	Prescribe	42	21.8%	Prescribe	61	28.8%
Test	Test	55	28.5%	Test	65	30.7%
Perform procedure	Perform procedure	21	10.9%	Perform procedure	23	10.8%
Consult	Refer/Consult	13	6.7%	Refer/Consult	7	3.3%
Patient Education	Educate/Counsel	24	12.4%	Educate/Counsel	18	8.5%
Dispose	Dispose	5	2.6%	Dispose	12	5.7%
	Monitor	5	2.6%	Monitor	8	3.8%
	Document	4	2.1%	Document	5	2.4%
	Advocate	2	1.0%	Advocate	2	0.9%
	Prepare	4	2.1%	Prepare	7	3.3%
	No recommendation	6	3.1%	No recommendation	2	0.9%
	Total Actions	193	100.0%	Total Actions	212	100.0%

In the randomization process to create the test set of recommendations, we needed to select 66 guidelines in order to obtain 50 that met eligibility criteria for this work. Most exclusions were related to focus (procedural guidelines) or guidelines that were being updated. Three recommendations were randomly selected from each of the 50 eligible guidelines.

In testing the preliminary action set, we had 3 conclusions:

1) We were able to create working definitions for each of the 6 preliminary actions. Definitions and illustrative examples (shown in italics) are:

Prescribe

Order a treatment requiring medication or durable medical equipment. (The guideline developers recommend that every patient who has experienced a noncardioembolic stroke or transient ischemic attack and has no contraindication receives …aspirin 50 to 325 mg daily; the combination of aspirin, 25 mg and extended-release dipyridamole, 200 mg twice per day; or clopidogrel, 75 mg daily…)

Perform therapeutic procedure

Order activities that are therapeutic in nature. (In all of the above situations, intensive phototherapy should be used if {total serum bilirubin} does not decline under conventional phototherapy.)

Educate/Counsel

Inform the patient about means to improve/maintain health, or instruct on how to perform specific activities. (Education about the etiology, prognosis, and risk factors for asthma and prevention of acute exacerbations is recommended.)

Test

Obtain or collect additional data through inquiry (ask patient), laboratory testing (chemistry panel, X-Rays, etc…) or other investigative procedures whose intent is not curative. (Testing for genital Chlamydial trachomatis infection should be performed in…mothers of infants with chlamydial conjunctivitis or pneumonia.”)

Dispose

Initiate an activity to direct the flow of patients, such as Admit, Discharge, Follow-up, Transfer, etc. (Discharge readiness criteria include…family has participated in the planning process and family/patient education is sufficiently complete to assure that prescribed care…can be provided safely and competently at home.)

Refer/Consult

Direct a patient to another clinician for evaluation and/or treatment. (Patients with blepharitis who are evaluated by non-ophthalmologist health care providers should be promptly referred to an ophthalmologist if any of the following occurs…)

2) We needed to add 6 new actions to accommodate recommendations in the test set. They were:

Conclude

Determine a diagnosis or clinical status (Mild traumatic brain injury has defined clinical diagnostic criteria, the hallmark of which is a transient neurologic deficit, along with a diagnostic study confirming the absence of acute skull fracture or pathology.)

Monitor

Make serial observations according to specific criteria and schedule. (All individuals with diabetes should receive an annual foot examination to identify high-risk foot conditions. This examination should include…)

Document

Record one or more facts in the patient record. Document includes situations in which a document (such as a medical report) is to be forwarded to legal authorities or guardians of a minor child to inform or report a condition. (…the emergency physician should initial and time the ECG, noting the presence or absence of changes indicative of acute myocardial infarction.)

Advocate

Argue in support of a policy (…interventions should be directed at one or more of the following areas: advocacy to change public policy to ensure that individuals with {spinal cord injury} have the resources to meet their lifelong needs.)

Prepare

Make ready for a particular guideline-directed activity by training, equipping, or gaining new knowledge (e.g., through research). (All physicians and other health-care providers who administer vaccines should have procedures in place for the emergency management of a person who experiences an anaphylactic reaction.)

No recommendation

A statement that no activity is advised, usually because of insufficient scientific evidence for or against the activity. (There is insufficient evidence to recommend for or against ambulatory electrocardiography screening for patients presenting with stroke or transient ischemic attack.)

3) In attempting to apply the preliminary action palette to the test set of recommendations it became clear that the proposed action “Consider” was not an action but a certainty modifier of other actions. For example, Consider performing a test was really about testing rather than considering. “Consider” appeared in 12 recommendations, but was associated 6 times with prescribing, 6 times with concluding, once with testing, and once with performing therapeutic procedure.

In creating the validation set, we randomly selected 56 additional guidelines to identify 50 that met eligibility criteria. Applying the actions identified in the test set, we found that all 150 recommendations could be categorized successfully.

The proposed action palette successfully categorized all action types required for the validation set.

Discussion

We identified a limited set of action types required to categorize a randomly-selected set of 150 guideline recommendations and verified that they comprehensively describe actions called for by a validation set of 150 different recommendations. In the process, we refined the definitions of the action terms. We describe the set of action types as a palette. This metaphor is intended to convey the concept that guideline implementers might select action types from a palette-like implementation tool—much as artists select colors from theirs—to “paint” implementation activities in a standardized manner.

The preliminary set of action categories defined experientially did not include (Conclude, Monitor, Document, Advocate, Prepare, and No recommendation), which collectively categorized 17.5% of the test set and 12.9 % of the validation set. Somewhat surprisingly, in both the test set and the validation set, the number of recommendations to test exceeded those to prescribe. Several reports on guideline implementation have emphasized the prescription function, paying less attention to other common therapeutic interventions (i.e., perform procedure, educate/counsel) and diagnostic activities (i.e., test, refer/consult, and monitor). In addition, disposition activities (e.g., admit, discharge, follow-up) aim to direct clinical workflow and were responsible for 2.6–5.7% of the recommendations.

Administrative action types include those activities with executive or managerial functions. These include the document and dispose action types. Document variants include patient record updating and editing, as well as various forms of reporting activities to local authorities, other health providers, parents or guardians, and legally mandated disclosures and reports. The dispose type of actions encompasses all those activities that direct the flow of patients within the healthcare enterprise, such as admit, discharge, transfer, follow-up, etc.

Resource-related action types aim at creating an environment (material or otherwise) favorable to implementing the guideline. They are perhaps hardest to model and implement electronically in a computerized Clinical Decision Support System (CDSS). Those calling for Advocate actions appeared only twice in each set, while Prepare appeared slightly more frequently.

Developers of guideline implementation tools, particularly computer-mediated decision support systems, can use these action types to identify recurring situations that may call for similar tasks for operationalization. Specifically, implementation tasks can be modeled using the action palette so that associated beneficial services can be generally provided.

For example, Prescribe requires that a drug be selected from a formulary, its formulation and dosage be chosen or calculated, and its instructions for use specified. Messages may need to be created to place the drug on a medication list, and to test for allergy or drug interaction. Prescriptions or inpatient orders must be generated and transmitted. These associated beneficial services facilitate the Prescribe action and providing them should lead to improved workflow integration. Likewise recommendations calling for Test actions require generation of order messages, abstraction of indications for the test from the health record, institution of systems to assure followup of test results and to facilitate interpretation. Similar recurring beneficial services can be associated with each of the guideline action types. We aim in future work to construct a framework application that will incorporate consideration of these services. When presented with a categorized action, this application will facilitate devising useful decision support tools (10).

We were impressed by the difficulty in implementation posed by a large number of guideline recommendations. It is clear that some action types are considerably more challenging to implement than others—particularly in computer-based decision support tools. Advocate and Prepare actions relate more to the structure of care than to the process (or measured outcomes) and pose considerable difficulty. Recommendations often did not explicitly tell users what to do. Particularly difficult is the No recommendation action. How does a developer operationalize a recommendation to do nothing? Perhaps the No recommendation should be perceived as a “pointer” to areas of future research to clarify or expand our knowledge of the validity of clinical activities and/or their outcomes.

A goal of many implementers is to help guideline authors to create recommendation statements that can be operationalized more readily (11). Work is underway to standardize guideline documentation (12). Identification of recommendations that will be difficult to execute during guideline development may influence recommendation writing in a positive way.}

Furthermore, the categorization of actions called for by recommendations may be used by those charged with selecting and implementing guidelines to assess the implementability of the recommendations. An Advocate action may be more complex to operationalize than a Prescribe action.

This paper is the first of which we are aware that attempts to classify action types of a broad array of real world guidelines into standard categories. The Unified Service Action Model (USAM) has been developed for the HL7 Reference Information Model to integrate guidelines and workflow management into electronic health records, but this work addressed primarily the conceptual rather than the practical level of implementation (13). The random selection of guidelines and recommendations in the work described here should contribute to the generalizability of our findings.

This study resulted in the creation of a library of 300 randomly selected clinical recommendations calling for 405 actions of 12 different types. Each entry in the recommendation library includes a numeric identifier, guideline title, date released, status, NGC category, main focus, intended audience, target population, 3 recommendations randomly selected from the guideline, and the action types represented for each. This collection should be reusable in future efforts to validate guideline models to assure that a representative selection of recommendations has been considered. For example, we plan to examine closely the conditional statements in guideline recommendations to assess their decidability.

It is important to note that the action palette we describe has only been validated with a sample of clinical practice guidelines that met our inclusion criteria, i.e., current, English-language, diagnostic, therapeutic, and management guidelines. Generalizability beyond that group cannot be predicted. Despite the success of categorization of the validation set, the wide variety of topics covered by guidelines suggests that the palette may not be comprehensive. Nonetheless, we believe this categorization will be useful for designers of clinical decision support systems.