An investigation into drug-related problems identifiable by commercial medication review software

Abstract

Background

Accredited pharmacists conduct home medicines reviews (HMRs) to detect and resolve potential drug-related problems (DRPs). A commercial expert system, Medscope Review Mentor (MRM), has been developed to assist pharmacists in the detection and resolution of potential DRPs.

Aims

This study compares types of DRPs identified with the commercial system which uses multiple classification ripple down rules (MCRDR) with the findings of pharmacists.

Method

HMR data from 570 reviews collected from accredited pharmacists was entered into MRM and the DRPs were identified. A list of themes describing the main concept of each DRP identified by MRM was developed to allow comparison with pharmacists. Theme types, frequencies, similarity and dissimilarity were explored.

Results

The expert system was capable of detecting a wide range of potential DRPs: 2854 themes; compared to pharmacists: 1680 themes. The system identified the same problems as pharmacists in many patient cases. Ninety of 119 types of themes identifiable by pharmacists were also identifiable by software. MRM could identify the same problems in the same patients as pharmacists for 389 problems, resulting in a low overlap of similarity with an averaged Jaccard Index of 0.09

Conclusion

MRM found significantly more potential DRPs than pharmacists. MRM identified a wide scope of DRPs approaching the range of DRPs that were identified by pharmacists. Differences may be associated with system consistency and perhaps human oversight or human selective prioritisation. DRPs identified by the system were still considered relevant even though the system identified a larger number of problems.

What this study adds:

• Currently there are no studies which have investigated artificial intelligence applications which have been trained in a commercial environment for real-world assessment of HMR patients.

• This study gauges the capacity of MCRDR to detect potential DRPs using commercial software with real patient data.

• MRM detected a wide variety and larger number of potential DRPs which may be overlooked by human counterparts, in the HMR domain at least, MCRDR is both viable and beneficial.

Background

A DRP can be broadly defined as “…an event or circumstance involving drug therapy that actually or potentially interferes with desired health outcomes”.¹ A HMR is a Commonwealth Government-funded service conducted by accredited pharmacists to identify and address DRPs among eligible patients.² HMRs involve substantial pharmacist-patient interaction and physician collaboration for comprehensive assessment of medication therapy. An important component is the professional skill of the pharmacist to be able to identify clinically relevant DRPs from the available information. This requires a wide scope of knowledge, not only of medications, but of evidencebased guidelines and contemporary management of a variety of medical conditions.

A commercial product developed by Medscope, MRM,³ incorporates an expert system for clinical decision support to assist with the detection of DRPs. MRM utilises a knowledge-based system to detect DRPs and provide recommendations for their resolution. This knowledgebased system uses MCRDR and was based on the work of Bindoff et al. who applied this approach to the knowledge domain of medication reviews.^{4, 5} MCRDR allows an expert in the knowledge domain to dynamically modify and add rules whilst the expert system is in use.⁵ This paper evaluates the similarities and differences between the findings by pharmacists and MRM, by highlighting common findings and extremes of difference. Possible advantages and limitations of the software, as well as areas for potential improvement are discussed.

Method

Data source

Australia-wide data collected during 2008 for a previous project, examining the economic value of HMRs, was used for this study.⁶ The data contained patient demographics, medications, diagnoses and pathology results for 570 community-dwelling patients aged 65 years old and older. The 570 HMRs were obtained from 148 different pharmacists. Supplementing this data were the original reviewing pharmacists’ findings, detailing pharmacistidentified DRPs and recommendations.

Data entry

The HMR data were entered into MRM and DRPs identified by MRM were recorded. MRM utilised a wide range of information including basic patient demographics such as age and gender, medication type including strength, directions and daily dose. MRM could calculate daily dose from strength and directions in many cases. Duration of use of medication could be entered. Medications were assigned Anatomic Therapeutic Chemical classifications (ATC);⁷ ATC is a five-tier hierarchical classification system allowing medications with similar properties to be grouped together in chemical classes which are then grouped into therapeutic categories.

Diagnoses could be entered and were based on the International Classification of Primary Care version 2 (ICPC- 2) classifications.⁸ The ICPC-2 classification system was also hierarchical, grouping diagnoses under similar categories. Diagnoses could be assigned temporal context as recent, on-going or past history. Medication allergies and general observations including height, weight and blood pressure could also be entered. Finally a wide range of pathology readings could be added, including biochemical and haematological data.

At the time of the data entry and collection of results, August 2011, MRM contained approximately 1800 rules.⁹ Rule development was undertaken by a pharmacist with expertise in both clinical pharmacology and HMRs.³

DRP classification

Direct comparison of the DRPs identified by MRM and those identified by the pharmacists was not possible due to the individual textual nature of each DRP. Instead, each DRP identified by either the pharmacist or MRM was mapped to a concept (defined here as a theme) that described the DRP in sufficient detail to allow comparisons of similarity and difference between pharmacists and MRM. The themes often described the type of drug or disease and other relevant factors involved. The development of a list of themes and the mapping of DRPs to themes was performed manually by the primary author, a qualified pharmacist, and validated by another pharmacist.

Examples of the text of two DRPs identified by a pharmacist and by MRM in the same patient are shown in Table 1. These DRPs were assigned the theme Hyperlipidemia under/untreated , which captured the basic problem identified within the text of each DRP.

Table 1. Example DRP text.

MRM	Pharmacist
Patient has elevated triglycerides and is only taking a statin. Additional treatment, such as a fibrate, may be worth considering	Patient’s cholesterol and triglycerides remain elevated despite Lipitor [statin]. This may be due to poor compliance or an inadequate dose

The initial list of themes were created where at least two of three published prescribing guidelines for the elderly were in agreement concerning the same types of DRPs.^10-12 DRPs from MRM and pharmacists were mapped to this list of themes. Further themes were added if both pharmacist and MRM DRPs could be mapped to any remaining prescribing guideline DRPs. New themes were developed for remaining pharmacist and MRM DRPs, where concepts were clearly similar but were not contained within prescribing guidelines. These new themes were very broad such as Vitamin, no indication , and may have included the DOCUMENT DRP classification text such as, Therapeutic dose too high .¹³ The remaining DRPs were unique to either pharmacists or MRM and themes were provided where possible, such as, Skin disease (un)dertreated – pharmacist only DRP. Lastly miscellaneous otherwise unclassifiable DRPs were assigned Other DRP pharmacist and Other DRP MRM .

A list of 129 themes was developed. Many themes described disease states and/or drug classes linked with identified DRPs in general terms.

Data analysis

The number of unique themes found in each patient was considered more important than the raw number of themes found in each patient. That is, where two DRPs matched the same theme in the same patient, that theme was counted once. The reason behind this decision was to compare the number of different types of conceptual problems that could be identified across patients rather than raw numbers across patients.

Each theme identified in each patient was allocated into one of three categories: 1. identified by pharmacists only; 2. identified by MRM only; or 3. identified by both.

Similarity between MRM and pharmacist themes was determined by averaging each patient’s Jaccard Index.¹⁴ A descriptive analysis of the themes was performed, highlighting themes that were common or clearly disparate.

Results

The patient cohort was predominantly female, with a mean age of 80 and a mean of 12 medications and 9 diagnoses, as described in Table 2.

Table 2. Patient demographics.

Patients (N = 570)	Demographics
Age (years)	79.6 ± 6.7
Gender	Male 234 : Female 336
Number of medications	12.0 ± 4.4
Number of diagnoses	9.1 ± 5.2

Pharmacists identified a total of 2020 DRPs, a mean of 3.5±1.8 per patient, with a range of 0 to 13 DRPs. MRM identified 3209 DRPs, of which 256 were excluded due to duplicated findings, leaving 2953 MRM DRPs, a mean of 5.2±2.8 per patient, ranging from 0 to 16 DRPs. A Mann- Whitney test showed a significant difference between the number of DRPs identified by MRM and pharmacists, U = 106461.5, p < 0.001.

The 2953 MRM DRPs were assigned to 100 different themes. Similarly, the 2020 pharmacist DRPs were assigned to 119 different themes. Ninety types of themes identified by pharmacists were also able to be identified by MRM. Within these 90 themes, the software was able to identify the same issues as the pharmacists in one or more of the same patients for 68 particular themes.

The number of different themes identified by MRM or by pharmacists per patient was considered more important than the raw totals. The 2953 MRM DRPs were aggregated into 2854 themes. Pharmacist DRPs which were clearly identifiable as compliance or cost-related problems and outside the scope of MRM’s ability to identify were excluded, this left 1726 pharmacist DRPs which were aggregated into 1680 themes.

MRM was able to identify the same themes as identified by pharmacists in the same patients 389 times, a 23% (389/1680) overlap of pharmacist findings by theme and patient. This then left 1291 themes identified by pharmacists only and 2465 themes identified by MRM only. For each patient a Jaccard Index was calculated, as the number of themes in common divided by the number of different themes found by either MRM or pharmacists. For the 570 patients Jaccard Index ranged from a minimum of 0 to a maximum of 1, with a mean of 0.09 ± 0.12.

The top five themes by number of patients in common are shown in Table 3. Some of the problems that were identified by the software are shown in Tables 3 and 4. Table 3 shows there is some overlap of the ability of MRM to find the same kind of problems as pharmacists in the same patients. However, both pharmacists and MRM find many instances of the same problem in different patients. Table 4 shows examples of some of the themes at the extremes of overlap. Two example themes (calcium-channel blocker and reflux and anti-lipidemic drug, no indication ) were identified in many patients by MRM but only once each by pharmacists. Conversely, the example themes vitamin, no indication and combine medications into combination product illustrate that pharmacists identified many patients with particular problems that MRM could not identify.

Table 3. Top five themes by patients in common.

Top five themes by cases in common	Patients MRM found	Patients pharmacist found	Patients in common	Total Patients: pharmacists + MRM
Osteoporosis (or risk) may require calcium and or vitamin D	137	117	49	205
Renal impairment and using (or check dose for) renally excreted drugs	122	48	24	146
Hyperlipidemia under/untreated	83	31	20	94
Sedatives long-acting or sedative long term	55	31	18	68
NSAID not recommended (heart disease/risk of bleed/other)	59	28	17	70

Table 4. Themes skewed in favour of MRM or pharmacists.

Skewed themes with cases in common	Patients MRM found	Patients pharmacist found	Patients in common	Total Patients: pharmacists + MRM
Calcium channel blocker and reflux	120	1	1	120
Anti-lipidemic drug, no indication	56	1	1	56
Vitamin, no indication	1	6	1	6
Combine medications into combination product	3	10	1	12

Discussion

The majority of the unique pharmacist themes involved mostly drug cost and compliance problems, themes that were not captured in MRM’s knowledge domain model. Although the majority of unique MRM themes could have been identified by pharmacists they were not. This may have been because the pharmacists either had additional knowledge, gathered in their interview with the patient that rendered these issues moot, or missed these particular issues. Alternatively, the software may have produced erroneous findings. The variety of variables encapsulated in the model was manifested in a broad scope of problems that could be identified by the software. For 68 themes the software showed the ability to identify the same issues that pharmacists could find in the same patients. In some circumstances half to all instances of a theme identified by pharmacists was also identified by MRM. However, there were many patients who had particular problems identified by either MRM or pharmacists but not by both. Twenty-two themes were identified by both MRM and by pharmacists without having any patients in common.

MRM found more problems than pharmacists. Pharmacists may have prioritised important issues over lesser ones, or perhaps the pharmacists may have lacked consistency in identifying DRPs. It is not unreasonable to suggest MRM exemplifies consistency, as it is after all computer software that is consistently applied. Calcium channel blockers may aggravate reflux disease, as consistently identified by MRM, yet this issue was identified on only one occasion by a pharmacist, suggesting MRM may be thorough or may be over-exaggerating a minor issue.¹⁵ Several studies examining clinical decision support, including two prototypes on which MRM was based, have identified that humans lack consistency or lack the capacity to identify all relevant problems in contrast with the software.^{4, 5, 16} One might worry that MRM would cause alert fatigue, wherein the system identifies so many irrelevant problems that the user simply ignores it entirely. However, MRM appears to have avoided this pitfall by ensuring that its rules are sufficiently specific, meaning DRPs are only very rarely identified inappropriately. Separate research that we have conducted to evaluate the DRPs found by MRM indicates that experts in clinical pharmacology do believe that MRM’s findings are clinically relevant and appropriate to the case.^17,18 This supports the position that MRM may be more consistent than pharmacists in identifying potential DRPs within its scope of knowledge. Additionally, MRM is likely to save pharmacist time preparing HMR reports for physicians through organising patient information and by identifying DRPs within seconds. Manual data entry took up to 10 minutes per patient although data can be directly imported from physician software.

The strength of this study is in the use of a large volume of real patient data and the ability to compare and contrast software findings with accredited pharmacist findings in the same patients. One limitation, mapping DRPs to themes was undertaken by the author solely. Agreement with the mapping process was not confirmed by an independent person.

An advantage of MCRDR is the use of case-based reasoning, allowing the knowledge domain expert to readily add new rules and refine existing rules. This method incrementally increases the precision of rules in context of the uniquely varied situations encountered through amassing knowledge on individual patients. This is an important point, as the development of new treatments and expanding medical knowledge needs to be incorporated into such software on an ongoing basis to maintain relevance. MRM appears to work well in the HMR domain, but improvements may include a greater extent of variables such as patient adherence or cost-related concepts, to widen the scope of problem detection and increase the accuracy of problem identification. Rule refinement to reduce the occurrence of duplicated DRPs is required. The potential of MCRDR technology has been shown through consistent identification of relevant DRPs, and further development and application of medical software utilising this approach is warranted.

Conclusion

The use of MCRDR software performed well in the complex and detailed HMR knowledge domain. The software identified a varied range of problem types, generally without excess frequency, within its scope of knowledge.

The truly interesting aspect is the software’s capacity to identify more problems than pharmacists. This capacity did not appear to involve lack of relevance, but it is likely to be a strong indication of the methodical ability of the machine to identify problems. This finding alone justifies the use of such a tool. The software cannot replace pharmacists, and is not intended to do so, but may help pharmacists make good decisions and avoid missing important drug-related problems.