Transforming primary medical research knowledge into clinical decision

Abstract

While the utility of computerized clinical decision support (CCDS) for multiple select clinical domains has been clearly demonstrated, much less is known about the full breadth of domains to which CCDS approaches could be productively applied. To explore the applicability of CCDS to general medical knowledge, we sampled a total of 500 primary research articles from 4 high-impact medical journals. Employing rule-based templates, we created high-level CCDS rules for 72% (361/500) of primary medical research articles. We subsequently identified data sources needed to implement those rules. Ourfindings suggest that CCDS approaches, perhaps in the form of non-interruptive infobuttons, could be much more broadly applied. In addition, our analytic methods appear to provide a means of prioritizing and quantitating the relative utility of available data sources for purposes of CCDS.

Introduction

Computerized clinical decision support (CCDS) has the potential to improve the quality and lower the cost of health care.¹ Systematic reviews have concluded that CCDS can improve health care process measures related to performing preventive services, ordering clinical studies, and prescribing therapies.^2,3 They have also been shown to improve practitioner performance with respect to diagnosis of cardiac ischemia⁴ and mood disorder,⁵ identification of at-risk behaviors,⁶ and diabetes care.⁷ While the utility of CCDS in multiple select domains has been clearly demonstrated, much less is known about the full breadth of clinical domains to which CCDS might be productively applied.

Particularly in the face of rapidly increasing medical knowledge, and the "big data" challenges associated with precision health,⁸ markedly expanded CCDS could help address challenges related to clinician cognitive limitations. While transformation of a much greater share of medical knowledge into interruptive pop-up reminders would quickly fatigue clinicians,⁹ non-interruptive approaches such as universal availability of highly relevant patient-specific information in the form of infobuttons¹⁰ would likely be welcomed.

For purposes of testing the feasibility of expanding the scope of CCDS, we determined the extent to which we could transform the study findings of a sample of primary medical research articles into boolean-logic CCDS rules. To further explore the feasibility, we also determined the data requirements of these rules.

Methods

Sample of primary medical research articles. We used Ovid MEDLINE to pull all articles with abstracts for 20182019 from four high-impact medical journals: New England Journal of Medicine, The Lancet, Annals of Internal Medicine, and JAMA Internal Medicine. We excluded review articles identified through Ovid MEDLINE, as well as articles explicitly identified in the title as systematic reviews, meta-analyses, position papers, or guideline articles.

We identified a total of 1,055 primary research articles. We then sorted the entire list of 2018-2019 primary research articles based on the primary author's last name and selected the first 500 for further analysis (a pseudo-randomized approach). The articles spanned primary author last names "Abou-Alfa" through "Lai."

Transformation of article titles and abstract conclusions into boolean-logic CCDS rules. For each of the 500 articles, we attempted to craft a reasonable high-level boolean logic CCDS rule from the article title and abstract conclusion (referred to as "title-abstract combinations" in the discussion below). Our assessment of "reasonable" was a general assessment that the information might be clinically useful under specified circumstances.

For articles that we could transform into CCDS rules, we further categorized the study findings included in the abstract conclusion as follows: positive study results, negative study results, non-inferior study results, or not readily categorized.

For each article, we attempted to rearrange information found in the article title and abstract conclusion into high-level CCDS rules using rule templates as a guide (figure 1):

If patient meets [study population criteria] and if clinician [has/has not/is] [ordered/performed/considering] [intervention], then alert clinician that [overarching study conclusion]

Examples of the rule template components include:

• Meets [study population criteria] - "If patient has [diagnosis]" or "If patient has undergone [surgical procedure]" or "if patient is [age-race-gender]"

• [Intervention] - A particular medication, surgical procedure, nursing protocol

• [Overarching study conclusion] - e.g., "Among [study population], [positive/negative outcomes] among those who received [intervention]"

Identification of relevant CDS-related data sources. As outlined in figure 1, we identified data sources that would either be useful or required for a sample rule implementation¹¹ - i.e., data corresponding to the logical criteria that fires the alert (the "if" portion of if-then logic). We attempted to determine these data needs from the perspective of a CCDS rule-author.

In the case of three data sources, we did apply more stringent criteria for inclusion (i.e., not simply "useful"); These three data sources included "symptoms and signs," "surgical plans," and "working (preliminary) diagnoses." In these cases, we only included them as data sources if the CCDS rules could not otherwise be implemented or if there was particular urgency in relaying the study recommendations. The reason for more stringent criteria with respect to these three data sources is based on our belief that they would necessitate direct data capture from the clinician for CCDS.

Such data capture would be needed due to the fact that clinicians do not keep a running log of their thoughts about "symptoms and signs," "surgical plans," and "working (preliminary) diagnoses." Insofar as a rule is intended to influence the clinician's thought process and requires data that has not been documented yet, then it must be captured directly from the clinician (and the challenges of acquiring coded data from clinicians are well-known¹²). As strictly one example, we judged "symptoms and signs" data as necessary for determining "mild to moderate Alzheimer disease" but not "metastatic prostate cancer." Unlike metastatic prostate cancer that can be determined by a combination of pathology, laboratory tests, and radiology, "mild to moderate" dementia requires clinician examination of the patient (e.g., a standardized cognitive exam). As a second example, we judged recommendations for acute management of stroke as urgent, justifying capture of a working (preliminary) diagnosis from the physician.

In the cases of sociodemographic, allergy, medication claims data, family history, care setting, and health information exchange data, we included them in our list of relevant data sources only if there was a fairly explicit reference to them. For example, even though pregnancy or prostate cancer recommendations are sex-specific, we did not include sociodemographic data (that includes sex) unless the title-abstract combination specifically referred to women or men. Similarly, even though health information exchange or medication claims data might frequently supplement other data sources, we only included them if cross-institutional or medical adherence data, respectively, were referenced by the article title-abstract combination.

Results

Transformation of abstract conclusions into boolean-logic CCDS rules. We were able to transform 72% (361/500) of primary research article title-abstract combinations into high-level boolean-logic CCDS rules.

The percentage of sampled articles that we could transform into CCDS rules ranged from 54% (33/61) for Annals of Internal Medicine to 84% (172/204) for the New England Journal of Medicine.

For the 361 articles that we could transform, we categorized the conclusions as demonstrating positive study outcomes for 70% (251/361) of articles, negative study outcomes for 19% (68/361), non-inferior study outcomes for 5% (18/361), and not readily categorized for 7% (24/361).

For articles reporting negative study outcomes, we were able to create CCDS rules using "blocking" CCDS rules,^{13, 14} which advise against ordering ineffective therapies. As deemed appropriate, for articles with non-inferior study outcomes related to medications, we created CCDS rules that arbitrated the preferred medication based on institutional formulary drug costs (which we considered an independent data source).

With respect to the 28% (139/500) of articles that we could not readily transform into CCDS rules, the primary limitation was that they did not test patient-specific interventions. These articles instead commonly described epidemiologic results, research related to clinicians or programs, policy or spending issues, or basic science findings.

CCDS-related data sources. Based on the logic of the constructed high-level CCDS rules, we determined data sources that would either be useful or required for rule implementation. Table 1 lists these data sources, as well as the percentages of rules that depended on these data sources (e.g., 69% of our constructed CCDS rules depended on the availability of medication order data). The mean number of data sources per rule was 4.05, while the median number was 4.00.

In table 1, we have only listed data sources for which at least 2% of rules depended. Additional data sources identified as useful or required by at least one CCDS rule, but less than 2% of the 361 rules included in decreasing order of frequency: Family history, dietary information, pulmonary function test orders and results, respiratory care orders, EEG orders and results, institutional drug cost data, ophthalmology measurements, durable medical equipment orders, vascular lab orders and results, hemodialysis status, allergy information, inter-institutional health information exchange data, patient insurance data, nerve conduction orders and results, over-the-counter medication data, dietary supplement data, medication claims data, and transfusion orders.

Discussion

Our findings suggest that boolean-logic CCDS models could be readily applied to a large percentage of medical knowledge. Through adoption of a generally formulaic approach using rule-based templates, we transformed 72% of primary medical research article findings into CCDS rules. We found journal-specific differences in the percentages of articles that we could transform, but in all cases it was a majority. We attribute the high level of correspondence between article title-abstract combinations and high-level CCDS rule templates to how abstract conclusions tend to be structured: i.e., most include a succinct description of the study population, the study intervention, and an overarching study conclusion related to study outcomes that provides rule reminder content. We anticipate that automated extraction^{15, 16} of interventions, study populations, and outcomes from abstract conclusions could facilitate generation of precisely-targeted infobutton¹⁰ content.

We also found that high-level CCDS rules based on primary research articles require a relatively small number of data sources, which largely already exist in electronic form. Analyses derived from the medical literature similar to those in the current study provide a promising means of assembling a comprehensive list of data sources important to CCDS implementation. From the perspective of a health care institution, our analytic approach also provides a method for prioritizing and quantitating the relative utility of electronic data interfaces and data capture processes for purposes of CCDS.

As one might expect, we found that many CCDS rules depend on the availability of medication, diagnosis, laboratory, and radiology data. We were also able to quantitate the utility of data sources not commonly captured, such as presenting symptoms and signs, working (preliminary) diagnoses, and surgical plans for implementation. Access to these forms of "real-time" information, or information captured from the clinician during or very shortly after the patient encounter, is required when the study conclusions relate to treatment of acute problems (e.g., acute stroke) or preferred surgical methods (only useful if the surgery has not yet occurred). Such preliminary diagnoses are commonly documented after the encounter, and consequently, frequently after medical decision-making. These preliminary diagnoses are also typically embedded only in clinical notes, requiring natural language processing (NLP) to extract. While approaches similar to the one in this study can assess the utility of capturing such real-time data, the major challenge will be to capture such coded information during encounters without placing additional demands on clinicians.

This study has limitations. We did not attempt to assess the validity of individual article conclusions in the context of similar studies in the medical literature, which would be useful for definitive CCDS recommendations. There was also some degree of subjectivity to assessing needed data sources. As discussed in the Methods section, we included sociodemographic, allergy, medication fill, and health information exchange data only if referenced fairly explicitly. For these particular data sources, our methods based on the medical literature would tend to underestimate their value - e.g., allergy information and medication claims data would arguably be useful in all cases that relate to medications. Finally, the article selection may be biased towards authors with included last names that published multiple articles with a similar writing style.

It has been estimated that outpatient physicians spend nearly twice as much time interacting with an electronic health record (EHR) and performing desk work than in direct clinical face time with patients.¹⁷ Such a time commitment can only be justified if EHRs improve medical decision-making and medical care, presumably through efficient forms of CCDS. As long as rule authors must manually translate select portions of the medical literature into CCDS rules, they will represent a bottleneck between clinicians and actionable medical literature-based recommendations. Our study strongly suggests that article titles and abstract conclusions represent a dense form of information important to CCDS content, which might facilitate automated CCDS development. Such automatically-developed CCDS content could help ensure that every clinician interaction with an EHR would include the availability of relevant content.

Figures & Table

Table 1: The number and percentage of rules dependent on each data source in descending order. (Data sources relied upon by less than 2% of rules are not listed.).

	Number	of
	rules		Percentage
Data sources relevant to rules	(Total n =	361)	of rules
Medication orders	250		69%
Diagnoses	247		68%
Laboratory orders and results	154		43%
Symptoms and signs	120		33%
Radiology orders and results	111		31%
Pathology orders and results	88		24%
Sociodemographic data	73		20%
Surgical history	49		14%
Working (preliminary) diagnoses	47		13%
Surgical plans	45		12%
Genetic laboratory orders and results	31		9%
Patient location/care setting	26		7%
EKGs/Echos orders and results	25		7%
Cardiac cath results	21		6%
Vitals	21		6%
Weight/BMi data	16		4%
Home glucose monitoring	14		4%
Ambulatory BP results	10		3%
Smoking	10		3%
Decision aid orders	10		3%
Social factors	9		2%
Endoscopy orders and results	9		2%
Consult orders	9		2%
Nursing protocol orders	9		2%