Effect of a clinical decision support system on early action on immunological treatment failure in patients with HIV in Kenya: a cluster randomised controlled trial

Summary

Background

A clinical decision support system (CDSS) is a computer program that applies a set of rules to data stored in electronic health records to offer actionable recommendations. We aimed to establish whether a CDSS that supports detection of immunological treatment failure among patients with HIV taking antiretroviral therapy (ART) would improve appropriate and timely action.

Methods

We did this prospective, cluster randomised controlled trial in adults and children (aged ≥18 months) who were eligible for, and receiving, ART at HIV clinics in Siaya County, western Kenya. Health facilities were randomly assigned (1:1), via block randomisation (block size of two) with a computer-generated random number sequence, to use electronic health records either alone (control) or with CDSS (intervention). Facilities were matched by type and by number of patients enrolled in HIV care. The primary outcome measure was the difference between groups in the proportion of patients who experienced immunological treatment failure and had a documented clinical action. We used generalised linear mixed models with random effects to analyse clustered data. This trial is registered with ClinicalTrials.gov, number NCT01634802.

Findings

Between Sept 1, 2012, and Jan 31, 2014, 13 clinics, comprising 41 062 patients, were randomly assigned to the control group (n=6) or the intervention group (n=7). Data collection at each site took 12 months. Among patients eligible for ART, 10 358 (99%) of 10 478 patients were receiving ART at control sites and 10 991 (99%) of 11 028 patients were receiving ART at intervention sites. Of these patients, 1125 (11%) in the control group and 1342 (12%) in the intervention group had immunological treatment failure, of whom 332 (30%) and 727 (54%), respectively, received appropriate action. The likelihood of clinicians taking appropriate action on treatment failure was higher with CDSS alerts than with no decision support system (adjusted odds ratio 3.18, 95% CI 1.02–9.87).

Interpretation

CDSS significantly improved the likelihood of appropriate and timely action on immunological treatment failure. We expect our findings will be generalisable to virological monitoring of patients with HIV receiving ART once countries implement the 2015 WHO recommendation to scale up viral load monitoring.

Funding

US President’s Emergency Plan for AIDS Relief (PEPFAR), through the US Centers for Disease Control and Prevention.

Introduction

At the end of 2014, 10.7 million people living with HIV in sub-Saharan Africa were receiving antiretroviral therapy (ART)—roughly 72% of the 14.9 million people receiving ART globally.¹ In 2014, 1.9 million people were newly initiated on ART and this number is likely to increase as a result of the 2015 WHO guidelines for HIV treatment, which recommend treatment of all HIV-infected people, irrespective of their CD4 cell count.^1,2 With unprecedented ART scale-up comes a major challenge of early identification and management of individuals in whom first-line ART is unsuccessful.

First-line ART regimens comprise the standardised efficacious, cost-effective, widely available, and least toxic drugs. The consequences of ART failure include increased risk of HIV-associated complications, such as opportunistic infections, malignant diseases, and neurocognitive dysfunction. Studies done in sub-Saharan Africa show that 15–25% of people receiving ART experience conditions that define treatment failure.^3–6 Although virological failure is the best predictor of ART failure, access to viral load monitoring for patients receiving ART remains restricted because of inadequate human capacity and laboratory infrastructure in resource-limited settings, especially in rural areas.⁶ Many rural clinics in sub-Saharan Africa therefore opt for WHO clinical staging and the widely available immunological monitoring based on CD4 cell measurement to monitor response to ART,^5,7,8 despite immunological monitoring being an imperfect way to identify treatment failure.⁹

Most adults and children receiving ART in sub-Saharan Africa are enrolled in government-owned HIV clinics, which are often busy and understaffed.^10,11 The challenge of management of a chronic disease with linked data from repeated clinic visits in these circumstances has a negative effect on thorough clinical monitoring.^6,12 Clinical decision support systems (CDSS) are computer programs that apply knowledge, often in the form of sets of rules, to data stored in electronic health records to offer patient-specific and actionable recommendations to improve clinical decisions.^13,14 CDSS applications communicate recommendations to clinicians through alerts and reminders, and have the potential to improve quality of care, patients’ safety, and outcomes in developed countries.^15–17 Systematic reviews^18,19 have shown that very few scientifically rigorous studies have been done in sub-Saharan Africa to show the effects of CDSS on clinical practice or health outcomes.

We did this study to establish whether a CDSS that supports detection of and recommends action on immunological treatment failure in patients with HIV on ART improves timely and appropriate action by clinicians.

Methods

Study design and participants

We did this prospective, cluster randomised controlled trial at HIV clinics in Siaya County, western Kenya. According to the Kenya HIV estimates report for 2013,²⁰ the adult HIV prevalence in Siaya County was 23.7%, which is four times higher than the national prevalence. All clinics in the region, to which the Kenya Medical Research Institute (KEMRI) provides data management support, were invited to participate. The rural clinics were located within the health and demographic surveillance area where KEMRI does various public health studies.²¹ The health facilities comprised three types: dispensaries, categorised by the Kenyan Ministry of Health as level 2 facilities, which provide specific outpatient services focusing on disease prevention and basic curative care, and are headed by a nurse; health centres (level 3), which provide outpatient, maternal and child health, and some inpatient services, and are headed by a clinical officer; and district hospitals (level 4), which are principal referral facilities at the district level; offer a complete range of ambulatory, outpatient, and inpatient services; and are headed by a physician.²² We excluded health facilities if they did not have reliable electric power, a secure location for a computer, or permanent data clerks to help with the regular data management activities as required by the study protocol.

We enrolled adults and children (aged ≥18 months) with HIV who were eligible for, and receiving, ART at the participating clinics. ART eligibility was based on 2011 Kenyan Ministry of Health HIV treatment guidelines, which recommended initiation of ART in patients classified as WHO clinical stage III or IV (appendix p 1), irrespective of CD4 cell count, or as any WHO clinical stage with a CD4 count of less than 350 cells per μL.²³ We included only patients who had at least two recorded CD4 cell counts on or after Jan 1, 2012 (ie, 6 months before implementation of the CDSS), because at least two CD4 cell measurements are needed to define conditions for treatment failure. We did not include HIV-positive pregnant women because the CDSS intervention was not implemented at the maternal and child health clinic where pregnant women are followed up.

KEMRI staff responsible for primary data collection de-identified individual patient data before analysis, and we did not obtain informed consent because data could not be traced back to individual patients. CDC and University of Amsterdam staff had no access to the patients or any information that would identify patients. The study was approved by the Associate Director for Science at the Center for Global Health of the US Centers for Disease Control and Prevention (CDC), and the institutional review board of KEMRI.

Randomisation and masking

The unit of randomisation was the health facility and not individual patients because all patients receiving HIV treatment at any participating clinic were given same level of treatment monitoring based on the type of electronic health record installed. The KEMRI data management team used block randomisation (block size of two) with a computer-generated random number sequence to assign (1:1) eligible health facilities into two groups—electronic health record either alone (control group) or with CDSS (intervention group)—matched by Ministry of Health level and the number of patients enrolled in HIV care.²⁴ For each level, whenever a clinic was randomly assigned to a group, a same-level clinic with the most similar number of patients enrolled was assigned to the opposite group. Health facilities participating in the study could not be masked to allocation because of the nature of the intervention.

Procedures

The health facilities transitioned from use of paper records to use of the Comprehensive Care Centre Patient Application Database (C-PAD) electronic health records for patient data management at varying times from July, 2009, to June, 2012. C-PAD was enhanced to include CDSS functionality and installed at the intervention sites in June, 2012. The electronic health records at control sites were upgraded to include CDSS at the end of the study in the event that positive associations were established. This process is part of the national scale-up of electronic health records in Kenya. For both the control and the intervention sites, clinicians recorded data on the paper-based patient encounter form (the so-called blue card) during the consultation, and the data clerk immediately entered this data into the computer. Laboratory results, including CD4 cell counts, were recorded on the blue card and entered into the electronic health records by the data clerks as soon as they were received from the laboratories. All CD4 cell counts were done at laboratories located within the participating health facilities or referral laboratories within Siaya County. The HIV care and treatment guidelines recommend that all HIV patients, irrespective of their WHO clinical stage, have a baseline and follow-up CD4 cell count.

Immunological treatment failure was defined in the 2011 Kenyan HIV treatment guidelines as any one of three conditions: a CD4 cell count less than the patient’s baseline measurement 6 months after initiation of therapy, a CD4 cell count less than 50% of peak measurement at any time after 6 months of treatment, or a CD4 cell count persistently less than 100 cells per μL after 12 months of treatment. Among children, immunological treatment failure was defined as a CD4 percentage of 10% or less or an absolute count of 200 cells per μL or less 12 months or more after initiation of ART.²³ The C-PAD electronic health record was designed to detect immunological treatment failure if any of the above conditions were encountered.

Health facilities participating in the study had one of the two versions of C-PAD. The version installed at the intervention sites had a CDSS implemented with pop-up information about an individual patient whenever action is needed. An alert was generated when a patient had immunological treatment failure. In such cases, the CDSS offered the clinician advice about appropriate action such as adherence counselling for the patient, doing a repeat CD4 cell count, ordering a viral load measurement, substituting drug, or switching treatment regimen. The CDSS also generated a reminder about overdue CD4 count (ie, more than 6 months after the last test, as defined in the Kenyan Ministry of Health HIV treatment guidelines).²³ More than one action could be recommended (table 1). The condition causing the generation of the alert and the action recommended by the CDSS were printed out by the data clerk and filed on top of the patients’ notes for immediate clinical action either during the current or the next monthly visit. When clinicians deemed immediate action necessary, patients were called back to the clinic before their next appointment. Clinicians had the option to over-ride the decisions recommended by the CDSS, in which case they were asked to record their reason for not following the recommendation and this information was captured in the study case-report form.

Table 1.

Condition and logic for alerts generated by the clinical decision support system

	Alert generated	Suggested actions by clinician
Ordering of CD4 cell count
Baseline CD4 T-cell count not available	Please order baseline CD4 cell test	If result available, write in blue card If result not available, consider the next immediate CD4 cell count as the baseline If none is available, order CD4 test and give reasons why baseline CD4 was not available
The last CD4 cell count was done >6 months before current visit date	The last CD4 test was done >6 months ago	If result available, write in blue card Indicated if CD4 test already reordered Reorder CD4 test if result not available and not yet reordered
Treatment failure in adults
Fall in CD4 count to or to less than pretreatment baseline concentration, ≥12 months after initiation of ART	Check for possible immunological treatment failure	Check on adherence and intervene Check viral load and intervene Order viral load test and intervene Check change of drugs (substitution) Check nutrition status and intervene, if needed Query if there was a drug stock-out Check if patient defaulted and intervene Check if there is any social behaviour that may interfere with adherence—eg, alcoholism Suspect immunological failure Shorten the time to the next appointment for close monitoring Do home visits by the staff to identify gaps at home
Decrease of ≥50% in CD4 count from on-treatment peak value during the follow-up period	Check for possible immunological treatment failure	Same as above
Persistent (two or more) CD4 count less than 100 cells per μL ≥12 months after initiation of ART	Check for possible immunological treatment failure	Same as above Assessment of opportunistic infections (eg, tuberculosis, Kaposi’s sarcoma, meningitis) Treat opportunistic infections Check nutrition status and intervene, if needed
ART eligibility for adults
CD4 count <350 cells per μL (irrespective of pregnancy) and not receiving ART	Patient is eligible for ART	Prepare patient for initiation of ART, via adherence counselling, drug counselling to decrease pill burden Give shorter appointment date Initiate care Check nutrition status and intervene, if needed
Patient co-infected with tuberculosis and HIV is not receiving ART	Patient is eligible for ART	Change or update WHO staging Isolate the patient or refer them to a tuberculosis clinic Start on tuberculosis treatment immediately Prepare the patient for ART Initiation within 2–8 weeks, after tuberculosis treatment has been tolerated Check nutrition status and intervene Order CD4 test Teach the patient about tuberculosis infection control
Treatment failure in children
CD4 percentage of ≥10% or absolute count of ≤200 cells per μL, ≥12 months or more after initiation of ART	Check for possible immunological treatment failure	Thorough screening for opportunistic infections Check or order viral load test and intervene With viral load result, if virological failure switch drugs Check on child adherence by querying caregiver and intervene Shorten time to the next appointment Check dosage of antiretroviral drugs against child’s weight Check nutrition status and intervene, if needed
Child aged >5 years with CD4 cell counts of ≤200 per μL, ≥12 months or more after initiation of ART	Check for possible immunological treatment failure	Same as above
ART eligibility for children
Child aged <18 months, irrespective of CD4 cell counts and not receiving ART	Patient could be eligible for ART; please do confirmatory HIV DNA PCR test	Do antibody test, if positive, do PCR; if PCR is positive prepare for ART initiation and initiate when ready Check nutrition status and intervene
Child aged 18–59 months with a CD4 percentage of <25% or a CD4 count <1000 cells per μL and not receiving ART	Patient is eligible for ART but not on ART	Prepare and initiate Check nutrition status and intervene, if needed
Child aged 5–12 years with a CD4 percentage of <20% or a CD4 count <500 cells per μL and not receiving ART	Patient is eligible for ART but not on ART	Prepare and initiate Check nutrition status and intervene, if needed

Conditions listed are those that were implemented in the clinical decision support system and are not exhaustive of all conditions in the clinical guidelines. ART=antiretroviral therapy.

The alerts were turned off in the version of C-PAD that supports standard care, which was installed at the control sites. Therefore, there were no alerts or recommendations and clinicians relied on weekly summary reports generated at patient level to make decisions about management, which is usual care since the implementation of electronic health records. The clinical staff and data clerks at the intervention sites were given practical training about the additional functionality and how to read the alerts and act on the recommendations. To avoid contamination, the control group received refresher training about use of the standard electronic health record and continued to provide care as usual, but received similar routine support visits by the KEMRI data management team as the intervention group. Training for both groups made reference to the HIV treatment guidelines whenever necessary and was in addition to the routine training provided by the Kenyan Ministry of Health. To generate data for alerts for analyses from the control sites, we retrospectively activated the CDSS alerts at the end of the study.

Outcomes

The primary outcome measure was the difference between groups in the proportion of patients who experienced any of the conditions that define immunological treatment failure and had a documented clinical action. Secondary outcomes were the effect of CDSS on time from detection of immunological treatment failure to clinical action, and time from ART initiation to first CD4 cell measurement. We qualitatively reviewed the most common reasons why clinicians ignore the recommendations of the CDSS, and time from ART eligibility to initiation of ART in patients who became eligible during the study period.

Statistical analysis

Sample-size calculations were based on the intention to include 20 facilities: ten facilities with electronic health records only or standard electronic health records and ten facilities with electronic health records and CDSS. A sample of 730 patients per group was required, with 80% power and a two-sided significant level of 0.05, to detect a difference between the two groups, specifying proportions of patients with clinical action after immunological treatment failure of 35% in the control group and 75% in the intervention group. This sample size was calculated assuming a design effect of 2.0 (for the intrafacility correlation) and that 20% of patients would experience treatment failure.^3,6 This resulted in an average of 73 sampled patients per clinic (1460/20) for each group of ten facilities. The sample-size calculation program was written in SAS statistical software (version 9.2), based on Rochon’s method.²⁴

The data elements required for the analysis in this study were extracted directly from the electronic health record into analysis software. Because the data contained missing values for some variables, we performed Little’s test to assess whether data were missing completely at random.²⁵ On the basis of the significant result obtained (p<0.0001), showing that missing data were not missing completely at random, we fitted a logistic regression model against the observed data to investigate whether data could be assumed to be missing at random. After confirmation that the data could be assumed to be missing at random, we used multiple imputation, which provides more efficient inference than complete case analysis when data are missing at random.²⁶ We used the Markov chain Monte Carlo method to impute missing data. We also did complete case analysis and compared the results with those obtained with multiple imputation.

We calculated means with SDs and medians with IQRs to summarise continuous variables, and frequencies, proportions, and 95% CIs to summarise categorical variables. We used the Kruskal-Wallis test to compare medians. We used logistic regression models to assess the effect of CDSS on detection of and action on immunological treatment failure. Six patient-level variables (age, sex, marital status, baseline CD4 cell count, WHO stage, and treatment regimen) and two centre-level variables (site type or level and duration of electronic health record use) were used as covariates to adjust for differences in case mix between intervention and control groups. We used generalised linear mixed models with random effects to analyse clustered data. Kaplan-Meier survival function plots and hazard ratios from the clustered Cox regression were used to measure statistical differences in the survival curves comparing time-to-event data and were expressed as a p value. Data were censored at the last follow-up visit. We did analyses with SAS (version 9.3) and Stata (version 13.0). This trial is registered with ClinicalTrials.gov, number NCT01634802.

Role of the funding source

The funders of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.

Results

Between Sept 1, 2012, and Jan 31, 2014, 13 clinics were randomly assigned to the control group (n=6) or the intervention group (n=7; figure 1). Data collection took 12 months at each site. 41 062 patients were enrolled in HIV care, of whom 19 662 (48%) patients were in control sites and 21 400 (52%) patients were in intervention sites. Of these patients, roughly half were eligible for and receiving ART (figure 1). Of the patients receiving ART, 1125 (11%) of 10 358 in the control group and 1342 (12%) of 10 991 in the intervention group had immunological treatment failure, of whom 332 (30%) and 737 (55%), respectively, had appropriate action taken (figure 1). Demographic and clinical characteristics of patients receiving ART were similar between groups, except for WHO clinical stage at enrolment in HIV care, for which more patients were classified in stages I and II in the control sites than in the intervention sites (table 2). Median age was 33 years (IQR 26–43), median CD4 cell count at enrolment into HIV care was 181 cells per μL (IQR 89–263), and most patients were in WHO clinical stage III at the time of enrolment in HIV care (table 2). The most common first-line regimens contained nevirapine (table 2). ART adherence at the last visit was 99% (table 2).

Figure 1. Trial profile.

ART=antiretroviral therapy.

Table 2.

Baseline characteristics

	Control (n=19 662)	Intervention (n=21 400)	Total (n=41 062)
Age group (years)
<15	1718 (8.7%, 8.3–9.1)	1674 (7.8%, 7.5–8.2)	3392 (8.3%, 8–8.5)
15–24	3899 (19.8%, 19.3–20.4)	3563 (16.7%, 16.2–17.1)	7462 (18.2%, 17.8–18.5)
25–49	12 264 (62.4%, 61.7–63.1)	13 749 (64.2%, 63.6–64.9)	26 013 (63.4%, 62.9–63.8)
≥50	1781 (9.1%, 8.7–9.5)	2414 (11.3%, 10.9–11.7)	4195 (10.2%, 9.9–10.5)
Sex
Male	6676 (34.0%, 33.3–34.6)	7661 (35.8%, 35.2–36.4)	14 337 (34.9%, 34.5–35.4)
Female	12 986 (66.0%, 65.4–66.7)	13 739 (64.2%, 63.6–64.8)	26 725 (65.1%, 64.6–65.5)
Marital status
Married	10 759 (54.7%, 54–55.4)	11 213 (52.4%, 51.7–53.1)	21 972 (53.5%, 53–54)
Divorced or separated	1427 (7.3%, 6.9–7.6)	1827 (8.5%, 8.2–8.9)	3254 (7.9%, 7.7–8.2)
Widow	3791 (19.3%, 18.7–19.8)	4216 (19.7%, 19.2–20.2)	8007 (19.5%, 19.1–19.9)
Single	3685 (18.7%, 18.2–19.3)	4144 (19.4%, 18.8–19.9)	7829 (19.1%, 18.7–19.4)
Baseline CD4 category (cells per μL)
<350	16 893 (85.9%, 85.4–86.4)	18 690 (87.3%, 86.9–87.8)	35 583 (86.7%, 86.3–87)
350–500	2489 (12.7%, 12.2–13.1)	2504 (11.7%, 11.3–12.1)	4993 (12.2%, 11.8–12.5)
>500	280 (1.4%, 1.3–1.6)	206 (1%, 0.8–1.1)	486 (1.2%, 1.1–1.3)
WHO stage at enrolment
I	5936 (30.2%, 29.6–30.8)	4941 (23.1%, 22.5–23.7)	10 877 (26.5%, 26.1–26.9)
II	6295 (32%, 31.4–32.7)	6027 (28.2%, 27.6–28.8)	12 322 (30%, 29.6–30.5)
III	6950 (35.3%, 34.7–36)	9759 (45.6%, 44.9–46.3)	16 710 (40.7%, 40.2–41.2)
IV	480 2.4%, 2.2–2.7)	673 (3.1%, 2.9–3.4)	1153 (2.8%, 2.6–3)
Transfer out
No	17 289 (87.9%, 87.5–88.4)	19 006 (88.8%, 88.4–89.2)	36 295 (88.4%, 88.1–88.7)
Yes	2373 (12%, 11.6–12.5)	2394 (11.2%, 10.8–11.6)	4767 (11.6%, 11.3–11.9)
First-line regimen
Nevirapine	9097 (86.3%, 85.7–87)	9438 (85%, 84.4–85.7)	18 534 (85.7%, 85.2–86.1)
Efavirenz	1399 (13.3%, 12.6–13.9)	1588 (14.3%, 13.7–15)	2986 (13.8%, 13.3–14.3)
Other	45 (0.4%, 0.3–0.5)	74 (0.7%, 0.5–0.8)	118 (0.5%, 0.4–0.6)
ART adherence at last visit
Satisfactory	10 525 (99.9%, 99.8–99.9)	11 079 (99.8%, 99.7–99.9)	21 604 (99.8%, 99.8–99.9)
Unsatisfactory	15 (0.1%, 0.1–0.2)	20 (0.2%, 0.1–0.3)	35 (0.2%, 0.1–0.2)

Data are n (%, 95% CI). ART=antiretroviral therapy.

2467 (34%) of all 7192 alerts were indicative of immunological treatment failure. Appropriate action based on the national treatment guidelines was taken on 332 (29.5%, 95% CI 26.8–32.2) of 1125 alerts in control sites compared with 737 (54.9%, 52.3–57.6) of 1342 alerts in intervention sites. The unadjusted odds ratio (OR) for taking appropriate action at an intervention site was 3.22 (95% CI 1.02–10.22; table 3). After adjustment for patients’ age, sex, marital status, baseline CD4 cell count, WHO stage, and treatment regimen, the likelihood of clinicians taking appropriate action on conditions indicative of treatment failure was higher with use of CDSS than with no decision support system (OR 3.18, 95% CI 1.02–9.87; table 3). The estimates obtained from multiple imputation and complete case analysis were similar for all variables (table 3, appendix p 2).

Table 3.

Association between clinical decision support system alerts and appropriate action on treatment

	Unadjusted odds ratio (95% CI)	p value	Adjusted odds ratio (95% CI)	p value
Site status
Control	1	··	1	··
Intervention	3.22 (1.02–10.22)	0.047	3.18 (1.02–9.87)	0.05
Age group (years)
<15	0.90 (0.74–1.08)	0.25	0.97 (0.78–1.21)	0.81
15–24	1.25 (0.91–1.74)	0.17	1.38 (1.24–1.53)	<0.0001
25–49	1.32 (1.18–1.46)	<0.0001	1.47 (1.08–1.99)	0.013
≥50	1	··	1	··
Sex
Male	1.11 (1.04–1.19)	0.0016	1.16 (1.09–1.24)	<0.0001
Female	1	··	1	··
Marital status
Married	1.21 (0.98–1.51)	0.077	··	··
Divorced or separated	1.31 (1.14–1.52)	0.0003	··	··
Widow	1	··	··	··
Single	0.93 (0.84–1.03)	0.18	··	··
WHO stage
I	1	··	1	··
II	1.67 (1.34–2.09)	<0.0001	1.62 (1.58–1.67)	<0.0001
III	1.68 (1.57–1.80)	<0.0001	1.64 (1.30–2.06)	<0.0001
IV	1.47 (0.91–2.40)	0.12	1.37 (0.89–2.12)	0.15
Baseline CD4 category (cells per μL)
<350	3.27 (2.87–3.73)	<0.0001	··	··
350–500	2.85 (1.69–4.82)	<0.0001	··	··
>500	1	··	··	··
Transfer out
No	1.25 (0.79–1.97)	0.34	··	··
Yes	1	··	··	··
First-line regimen
Nevirapine	1.85 (1.37–2.52)	<0.0001	··	··
Efavirenz	1.41 (0.89–2.22)	0.14	··	··
ART adherence
Satisfactory	1.59 (0.67–3.79)	0.29	··	··
Unsatisfactory	1	··	··	··

ART=antiretroviral therapy.

Of the 4376 recorded actions taken on patients who had HIV treatment failure, the most common were adherence counselling done (n=537), viral load ordered (n=571), alternative first-line drug substitution (n=73), repeat CD4 cell test (n=65), and a switch to second-line drugs (n=52). In some cases, more than one action was taken for the same patient.

The median time from alert to action was 47 days (95% CI 28–84) in the control group compared with 13 days (11–14) in the intervention group (p<0.0001; figure 2). 507 (90%) of 556 ART-eligible patients in the control group and 669 (93%) of 723 in the intervention group initiated ART after CDSS implementation and had a recorded CD4 cell result. 4493 (62.5%) of the 7192 alerts were indicative of cases for which the last CD4 cell test had been done more than 6 months previously. 162 (33.8%, 95% CI 24.9–44.9) of 480 patients in the control group, and 271 (41.5%, 32.2–52.4) of 652 patients in the intervention group, had CD4 cell counts done 6 months after ART initiation, as recommended in the HIV treatment guidelines. The median time from ART initiation to first CD4 cell count was 12.13 months (95% CI 6.17–not reached) in the control group and 6.76 months (5.13–11.23) in the intervention group (p<0.0001; figure 3).

Figure 2.

Time from immunological treatment failure to appropriate action in patients receiving antiretroviral therapy

Figure 3.

Time from initiation of antiretroviral therapy to first CD4 cell measurements

In 1445 cases, clinicians did not act on the recommendation following an alert. In 1109 (77%) of the cases in which the alerts were ignored, clinicians reported that the appropriate action (such as ordering adherence counselling) had been taken, but was not recorded in the patient charts. There were 225 (16%) cases in which the CD4 cell analyser was out of order or did not have reagents. In such cases, the recorded action was that a CD4 cell test had been ordered. In 15 (1%) cases, the clinician’s perception was that the CDSS alert was incorrectly raised because of errors in the data (eg, an incorrect CD4 cell value entered in the electronic health record). In one case, a provider indicated that “it is possible to fail to act on an alert when the clinic is busy and there are many patients to attend to”.

120 patients were eligible for, but not receiving, ART in the control group compared with 37 patients in the intervention group (figure 1). The CDSS generated 232 alerts for ART-eligible patients who were not receiving ART, which accounted for 3.2% of all alerts generated. Overall, appropriate action, including pre-ART counselling or enrolment on ART, was taken on 103 (44.4%) of the alerts for ART-eligible patients not receiving ART. CDSS use was not significantly associated with appropriate initiation of eligible patients on ART (adjusted OR 1.04, 95% CI 0.41–2.68).

Discussion

Our results provide compelling evidence that a CDSS supports clinicians in timely detection and appropriate action on HIV treatment failure in conformity with HIV treatment guidelines. The odds of clinicians taking appropriate action on immunological treatment alerts were more than three times higher with CDSS than without a decision support system. Additionally, we recorded a 72% relative reduction in the median time taken from detection of conditions for treatment failure to appropriate action, and timely performance of CD4 cell counts, as recommended in the national HIV treatment guidelines.

Timely detection and action on HIV treatment failure can substantially reduce the risk of disease progression, drug resistance, and even death.²³ Furthermore, early actions, such as ART adherence counselling or repeat or confirmatory CD4 cell counts, could help to identify patients in need of additional clinical support and could provide opportunities for prevention and timely diagnosis of opportunistic infections while ensuring timely switches to second-line treatment. This would ultimately improve quality of care, thereby reducing the need for more expensive second-line regimens that are also complex to administer and monitor. Second-line drugs cost about 110% more than first-line drugs; for example, the annual cost of first-line ART for an adult patient in Kenya in 2013 was US$338 compared with $716 for second-line treatment.²⁷

Our findings also show that CDSS marginally improved timeliness of CD4 cell measurements, which is a crucial quality-of-care indicator for immunological response to treatment. Although the proportion of patients who had a CD4 cell test done 6 months after ART initiation was slightly higher in the CDSS group than in the control group, the difference was not statistically significant. Moreover, there was no significant association between use of CDSS and initiation of ART in eligible patients. This result is consistent with findings that attributed an absence of association to factors such as provider practice (clinician’s behaviour) and the patient’s own readiness to initiate ART.²⁸ The overall improvement in quality of care shown in our study is consistent with many studies included in previous systematic reviews^13,19 and adds to the few studies of CDSS done in resource-constrained settings.²⁹ The set of rules programmed in the CDSS to detect and recommend action on immunological treatment failure can also be applied to other laboratory test results, such as viral load. As routine viral load monitoring is implemented, CDSS could provide an opportunity for improved actions such as improved adherence, timely repeat viral load testing, and treatment switches, thereby reducing the accumulation of drug-resistant mutations and improving clinical outcomes.

Because of an absence of reliable electric power, secure location for a computer, or permanent data clerks, seven of 20 invited HIV clinics could not be included in the study. However, at the remaining 13 sites, we were able to include 21 349 patients instead of the minimum required 1460 patients. The cluster randomised design of our study and the large sample size from multiple study sites minimised bias and improved the precision of estimates. As a sensitivity analysis, we used the generalised estimating equations approach to assess the magnitude of association between CDSS and appropriate action and, although the approach narrowed the confidence intervals, it did not qualitatively change the results, nor their significance. Doing both complete case and multiple imputation analyses was reassuring because the results from the two analyses were similar, with only slight variations in p values.

Notably, although the intervention was very effective, there is still much room (about 45%) for improvement in the actions taken on treatment failure. On the basis of studies done in sub-Saharan Africa, which estimated that 15–25% of HIV patients receiving ART have immunological treatment failure,^3–6 the implication of no action taken on the alerts in Kenya, where about 744 000 patients with HIV were receiving ART at the end of 2014,³⁰ is that at least 50 228 individuals could have immunological treatment failure with no action taken. Our study provided strong evidence that CDSS can significantly reduce this number, but there should be adequate capacity to act on the recommendations of the CDSS, including laboratory capacity to ensure timely conduct of CD4 cell tests and an optimum number of clinical staff with skills to act appropriately on immunological treatment failure. Additionally, investments are needed to improve health systems infrastructure, including reliable electric power, information and computer technology resources, and health workers’ information and computer technology capacity, in order to maximise on the benefits of CDSS on a wider scale.

Our study has some limitations. First, because of the high number of patients attending clinics with inadequate numbers of health workers, some actions might have been taken (eg, CD4 tests) but not recorded in the electronic health records, thereby potentially compromising data quality. This issue was particularly evident in control sites, where fewer alerts on overdue CD4 cell counts were noted despite regular supervision at all study sites to ensure high quality of data. This finding could indicate that CDSS alerts remind clinicians to record missing CD4 results. Second, clinical and nursing officers were frequently transferred to and from the study clinics, thereby introducing new health workers with no knowledge of the study. Ongoing training of clinical staff by the study team minimised the effect of such transfers. None of the staff was transferred from one study site to another, thereby eliminating any cross-over effect. Third, randomisation did not uniformly assign patients by severity of illness (WHO clinical stage). To correct for this irregularity, we adjusted for WHO stage in the multivariate analysis. The WHO HIV treatment guidelines launched in July, 2013, and the update released in September, 2015, recommend use of viral load tests to monitor patients’ response to HIV treatment,^2,31 and Kenya is in the process of transitioning from immunological to virological monitoring of patients. The 2013 guidelines were released in the last 6 months of the study and had not taken effect by January, 2014, when our data collection ended. Most countries in sub-Saharan Africa are beginning to roll out viral load monitoring as the preferred strategy for confirmation of ART treatment failure, but the operationalisation and feasibility of this approach still needs close monitoring and evaluation.

In conclusion, our study shows that CDSS can improve the likelihood of timely and appropriate action on immunological treatment failure in patients with HIV in resource-limited settings. We expect that the associations shown will be generalisable to improvement of virological monitoring of patients receiving ART once countries implement the recommendations of the 2015 WHO HIV treatment guidelines to scale up viral load testing. Further studies are needed to assess the effect of CDSS on treatment outcomes such as viral suppression and survival in resource-limited settings.

Research in context.

Evidence before this study

In 2012, we published a systematic review to identify original studies of the effect of clinical decision support systems (CDSS) on HIV care in resource-constrained settings. We searched on MEDLINE, Embase, CINAHL, and Global Health Library databases for original studies published before January, 2012. Our search criteria included MeSH terms relevant to HIV, electronic medical records, and decision support systems. Studies eligible for inclusion had to describe or evaluate implementation of an electronic medical record-based CDSS, include electronic medical records used to provide care to persons with an infectious or chronic disease (including HIV or tuberculosis), and have been done in a resource-constrained setting. Evidence from the above systematic review and more recently published studies and reviews in resource-constrained settings mainly report the effects of CDSS on process outcomes such as patient appointments, waiting time, and laboratory and pharmacy ordering time. Other outcomes of interest that have been associated with CDSS in resource-limited settings are adherence to guidelines, cost-effectiveness, and data quality. Recent systematic reviews concur that most of the studies done in resource-limited settings are premature (ie, they describe effects of early implementations of electronic systems on health outcomes and processes) and of varying and overall low quality. Most of the studies have a weak study design or small sample size. Very few randomised controlled trials have been done in resource-limited settings to assess the effect of CDSS on quality of care and treatment outcomes. None of these trials measured outcomes on treatment failure in patients with chronic diseases in resource-constrained settings. Because our systematic review was done more than 3 years ago, we did a new PubMed search on Oct 15, 2015, with MeSH terms “clinical decision support system” and “treatment failure”, which yielded 177 hits, of which 16 studies described the effect of CDSS on a clinical outcome. None of the 16 studies was done in a resource-constrained setting.

Added value of this study

Our study contributes new knowledge by showing a strong association between an electronic medical record-based CDSS and appropriate action taken by clinicians on immunological treatment failure in HIV patients in resource-constrained settings. We also confirmed findings from previous studies with before-and-after study designs, which showed that CDSS is not associated with timely CD4 cell counts as recommended in the guidelines.

Implications of all available evidence

Although our main objective was to assess the effect of CDSS on appropriate action on immunological treatment failure in patients with HIV, the underlying model for longitudinal tracking of conditions that describe treatment failure and recommendation for appropriate action is applicable to other chronic diseases. We anticipate that the findings from our study will be generalisable to virological treatment failure because conditions defining virological treatment failure can be programmed in a CDSS in a similar way as immunological treatment failure. However, more scientifically rigorous studies are needed to confirm the generalisability of these findings.