Electronic decision protocols for ART patient triaging to expand access to HIV treatment in South Africa: A cross sectional study for development and validation

Abstract

Background

The shortage of doctors and nurses, along with future expansion into rural clinics, will require that the majority of clinic visits by HIV infected patients on antiretroviral therapy (ART) are managed by non-doctors. The goal of this study was to develop and evaluate a screening protocol to determine which patients needed a full clinical assessment and which patients were stable enough to receive their medications without a doctor’s consultation. For this study, we developed an electronic, handheld tool to guide non-physician counselors through screening questions.

Methods

Patients visiting two ART clinics in South Africa for routine follow-up visits between March 2007 – April 2008 were included in our study. Each patient was screened by non-physician counselors using the handheld device and then received a full clinical assessment. Clinicians’ report on whether full clinical assessment had been necessary was used as the gold standard for determining “required referral”. Observations were randomly divided into two datasets – 989 for developing a referral protocol and 200 for validating protocol performance.

Results

A third of patients had at least one physical complaint, and 16% had five or more physical complaints. 38% of patients required referral for full clinical assessment. We identify a subset of questions which are 87% sensitive and 47% specific for recommended patient referral.

Conclusions

The final screening protocol is highly sensitive and could reduce burden on ART clinicians by 30%. The uptake and acceptance of the handheld tool to support implementation of the protocol was high. Further examination of the data reveals several important questions to include in future referral algorithms to improve sensitivity and specificity. Based on these results, we identify a refined algorithm to explore in future evaluations.

Introduction

One of the greatest limiting factors to the expansion of HIV treatment programs in Africa is the chronic shortage of physicians [1]. The demand for physicians to treat HIV patients in many parts of Africa greatly exceeds the number of physicians who are available, limiting the ability to bring HIV treatments to those in need. This problem is compounding, as the numbers of individuals eligible for antiretroviral therapy (ART) increases with patients on treatment living longer and needing routine care for indefinite periods of time.

Additionally, most HIV infected patients must travel a long distance to reach a specialist physician. This travel is expensive, disruptive, and makes the targeted 90–95% adherence rates needed to prevent resistance to first-line medications difficult for patients [2]. Improving adherence through better access provides an enormous incentive to add treatment centers closer to patients’ homes [3]. Given the existing physician shortages, the addition of new treatment centers necessitates the delegation of routine tasks currently performed by physicians to less specialized health personnel, a strategy known as task shifting [4]. However, this task shifting strategy has several key challenges that must be addressed: how to maintain the high quality of care that has been established in ART treatment centers, and how to support, supervise, and report on activities in these new clinics.

Many countries continue policies requiring physician care for health problems to ensure quality, despite the fact that the shortage of doctors makes these policies impossible to follow. At the time of this study, the South African policies required that all patients visiting ART facilities receive a full clinical assessment by a clinician at each visit. Countries are understandably reluctant to support policies that may be perceived as “second class” care implicit in the use of less trained health workers. One of the advantages of the use of clinical protocols to standardize delivery of services is that the protocols support high quality care even when delivered by less specialized health care workers. However, protocols must be rigorously developed and validated to show their equivalence to the current “gold standard” of physician care if countries are to publicly support their use via policy change and incorporate protocols into task shifting activities.

Another factor limiting expansion of HIV treatment is the need to keep accurate records for each patient, each visit, and each program. Accurate records improve patient treatment and care, help clinics identify patients who miss their appointments, and support program monitoring and reporting. In general, the state of health information systems in low income countries is woefully inadequate. Often individual level data is irretrievable after the visit. Without rigorous supervision, aggregate data contain numerous errors [5] and the delay in reporting can undermine the value of the results. Vast resources have been invested into HIV treatment programs to enable a host of workers to collect, enter and process data on patients and program outputs. However, there are substantial questions about whether these costs can be maintained indefinitely or if these systems can grow as programs expand and as demand for information beyond HIV/AIDS is needed.

Use of electronic tools to implement task shifting protocols has the potential to alleviate this data collection and reporting burden. Data captured electronically is easily accessed for long-term care and can be immediately utilized for supervision and program reporting. Further, there is evidence that electronically implemented protocols are more rigorously followed by health care workers and these protocols are easily updated [6]. Finally, electronic systems have been shown to have other benefits including reducing medical errors, saving money (by decreasing the use of unnecessary medical care or training time), increasing accuracy, and improving the overall quality of care [7–14]. However, there are few rigorous evaluations on the effectiveness of electronic systems to improve healthcare in low-income countries [15]. Electronic systems can introduce new errors, require special training, and incur ongoing costs to maintain and support. It is important to assess the usefulness and feasibility of deploying these systems, as well as the costs, in order to inform programs about when their use is appropriate.

This paper presents the results of a 14 month study in South Africa designed to test whether a protocol used by non-medically trained lay counselors could accurately determine which patients were doing well and could continue on their current treatment program and which patients needed further consultation with a physician. We describe the development and validation of this tool. We report the sensitivity and specificity of this protocol for patient referral as well as recommendations for future referral protocols. This triaging protocol reduces the average burden on the physician for managing a patient, reducing the number of physicians required for a fixed number of patients which is necessary for maintain or expanding access to treatment [16]. Further, the data from the screening protocol is captured electronically by the counselors, providing the information needed to track individual patients and ability to accurately aggregate data in a timely way for management and reporting purposes.

Methods

The study was conducted during a 14 month period (March 2007 – April 2008) at two large urban hospitals in South Africa: Tygerberg Hospital outside of Cape Town and Helen Joseph Hospital in Johannesburg. The sites were chosen on the basis of the size of the patient population on ART and the interest of the staff in testing a new approach to patient triage. To be eligible for the study, the patient needed to be over 18 years old, on the same ART regimen for at least three months, and scheduled for a physician visit.

All patients first visited a counselor, who first obtained verbal consent to participate in the study. The consent script, which adhered to standard requirements, was written on the PDA and read to the patient. Verbal consent was noted as a response in the electronic system. There was no compensation and all patients asked agreed to participate. The counselor then led each patient through a series of questions. The initial screening questions were derived from the Integrated Management of Adolescent and Adult Illness (IMAI) guidelines [17], modified based on feedback from physicians and counselors working in these clinics. Questions were added and removed at two different phases in the study based on feedback from site teams. To facilitate adherence to the screening questions and reduce data entry errors, we designed an easy-to-use interface on a handheld device (PDA) in which a summary of each question and response scrolls up the screen (see Figure 1, [18]). Patient self-report of symptoms were entered electronically into the PDA by the counselors. There were three counselors at each of the sites.

Figure 1. PDA Screen Shot.

Screen shot of the electronic tool used to capture self-reported answers from patients.

Subsequent to the counselors’ screening, each patient visited a physician for a full clinical assessment, as per current standard of care. Physicians were blinded to the results captured by the PDA. At the end of these sessions, physicians were asked to fill in a form capturing whether they believed that a full clinical assessment had been necessary and if so, the reasons for this recommended referral and the resulting action taken by the physician.

Data captured on the PDA was downloaded daily into an Access database. The paper forms were entered into an electronic database and linked to the PDA responses using a unique identifier that included the date of the visit and the last three digits of the patient’s clinic record. All data was password protected and the computers and PDAs were kept in a locked room where paper medical records were stored.

We divided the data into two sets. The validation dataset contained 100 patients with physician recommended referral for clinical assessment and 100 patients without physician recommended referral. These were randomly selected from the overall dataset. The remaining patients were included in a dataset used to develop the referral protocol for the PDA system.

Treating the physician recommendations as the gold standard, we used an iterative process to identify the patient responses most strongly linked to physician recommended referral via logistic regression models. At each iteration, we identified the question with the largest odds ratio that was a strong predictor of patient referral (at alpha=0.10) and added this question to the referral algorithm. Then, all patients who answered yes to this question were removed from the protocol development dataset, and models were fit to the remaining data. We repeated these steps until none of the remaining questions were significantly predictive of doctor recommended referral. Any patients with missing data, including non-response or question not included during a phase of the study, were excluded from analysis for that specific question.

Performance of the algorithm in the referral protocol development dataset and the validation dataset were assessed with measures of sensitivity and specificity [19] and corresponding confidence intervals calculated using the Binomial distribution. We grouped physician reasons for referral and following actions into predominant symptom/disease and action categories and explored the linkage between physician and patient responses to identify potential for new or clarified questions in later protocols. All analyses were performed in Stata/MP release 11 (College Station, TX).

Results

1189 individual patient encounters were included in the study. Table I summarizes basic patient characteristics. Of the 1189 encounters, 449 (37.8%) patients were classified by the physicians as needing referral for clinical review. About a third of patients had no physical complaints when prompted by the PDA questions, whereas 190 (16%) had five or more physical complaints. Of the 974 women, 230 (24%) reported reproductive concerns, such as missed periods or suspected pregnancy. The majority of participants (1062, 89.4%) had one or fewer social or psychosocial complaints (problems at home, problems with alcohol, etc). Reasons for physician recommended referral included a variety of diseases and symptoms. The most prevalent are listed in Table I, and include suspicion of TB, skin problems, gastrointestinal problems, complaints of pain, peripheral neuropathy, lipodystrophy or lipoatrophy, genital or urinary problems, and adherence concerns.

Table I. Patient Characteristics.

	Total Dataset		Protocol Developing Dataset		Validation Dataset
	n=1189		n=989		n=200
Doctor Recommended Referral	449	37.8%	349	35.3%	100	50%
Number of Physical Complaints†
0 complaints	403	33.4%	347	35.2%	56	28.0%
1 complaint	196	16.5%	170	17.2%	26	13.0%
2 complaints	134	11.3%	110	11.1%	24	12.0%
3 complaints	169	14.2%	138	13.9%	31	15.5%
4 complaints	97	8.2%	77	7.8%	20	10.0%
5 or more complaints	190	16.0%	147	14.8%	43	21.5%
Reproductive Concerns/Complaints‡	n=974		n=810		n=164
Women Only	230	23.6%	194	24.0%	36	22.0%
Social/Psychosocial Complaints	n=1189		n=989		n=200
0 complaints	728	61.3%	609	61.6%	119	59.5%
1 complaint	334	28.1%	280	28.3%	54	27.0%
2 complaints	97	8.2%	74	7.5%	23	11.5%
3 or more complaints	30	2.5%	26	2.6%	4	2.0%
Reasons for Doctor Requested Referral††	n=449		n=349		n=100
TB related	10	2.2%	5	1.4%	5	5%
Skin Problems	48	10.7%	36	10.3%	12	12%
GI Problems	32	7.1%	28	8.0%	4	4%
Complaints of pain	81	18.0%	61	17.5%	20	20%
Peripheral Neuropathy	43	9.6%	33	9.5%	10	10%
Lypodystrophy or lypoatrohy	28	6.2%	22	6.3%	6	6%
Genital or urinary problems	16	3.6%	12	3.4%	4	4%
Adherence problems	29	6.5%	27	7.7%	2	2%

^†Patient responses to PDA questions;

^‡Pregnancy, Missed Periods, Birth Control;

^††As reported by physicians recommending referral restricted to patients recommended for referral. (Note that these are not exclusive and that percentages are of entire patient population.)

As a first step in the protocol, all patients who required follow-up as indicated in their previous charts were automatically included as a referral. This included patients with pending lab results or serious conditions at the last visit requiring the physician to assess the patient’s current condition. For the remaining patients the final referral protocol is listed in Table II. This referral algorithm is 86.5% sensitive and 48.3% specific in the protocol development dataset. In the validation dataset the referral protocol is 88% sensitive (95% CI: 80%, 94%) and 42% specific (95% CI: 32%, 52%). Combined performance in the full dataset is a sensitivity of 86.9% (95% CI: 82.5%, 89.9%) and specificity of 47.4% (95% CI: 43.8%, 51.1%).

Table II. Final Referral Protocol.

Does the patient record indicate required follow-up?

Does the patient report any of the following?

Feeling ill

Coughing for more than one month

Genital problems

Mouth sores

Other problems

Diarrhea worse

Night chills

Rash

New foot pain

Lost consciousness

Pregnant/Suspected pregnancy

Adherence problems

Changes in shape of abdomen/breast

Abnormal changes in shape of face/arms/legs

Unsafe sex (if sexually active)

We display the trade-off in sensitivity and specificity at the addition of each question in an ROC curve (Figure 2). The 45° line indicates the sensitivity and specificity of a protocol that would randomly recommend referral independent of any patient characteristics. The presence of the ROC lines through the upper left quadrant suggests that the developed protocol has strong predictive properties. The overlap of the lines for the total dataset and the validation dataset suggests that these predictive properties are consistent.

Figure 2. ROC Curve.

ROC curve for the referral protocol in the total dataset and the validation dataset. Area under the curves are 0.788 (full dataset) and 0.785 (validation dataset).

In the two datasets combined, we missed 59 of 449 patients who were recommended for referral by the physician. The Supplemental Table lists these patients who were missed and the problem identified by the physician requiring referral. In an effort to understand why these patients were not accurately screened by the protocol, we matched the patients against their responses to questions intended to screen for this particular problem. For example, if a patient had a rash identified by the physician we looked at whether there was a corresponding question “do you have a rash?” and the patient’s response to that question. For 24 of these patients, the corresponding question was included in the final referral protocol, but we recorded a negative response by the patient to these questions. For nine of the 59 missed patients, the corresponding question was not included in the final protocol but the patient responded negatively to the question so that even if the question had been included, the patient would have been missed. In four patients, the answers to screening questions were positive but the questions were not included in the final protocol. Another thirteen patients had very specific problems for which we did not screen. In nine patients, the doctor’s notes lacked enough detail to infer the medical problem motivating the recommendation for patient referral.

For the 449 patients recommended for referral, we collapsed their reasons for referral given by the physician into 24 major categories. We then identified the questions considered in this study that best matched these major categories. Ten of the reasons for referral categories have no corresponding questions in the original set of questions considered, including general body pains, cardiac problems, eye problems, diabetic complications, dizziness, anemia, general malaise, and weight loss (Table III). These also included patients recommended for referral because they needed further investigation or had problems to discuss with the doctor, where no further detail was provided to allow us to group these patients in other categories. As shown in the table, while these categories did not have corresponding questions, many patients with these problems were captured with the questions relating to general health status (feeling ill, other problems, or other medical problems). For example, 50% or more of patients with diabetes complications, dizziness or general body pains (non-peripheral) were captured with these health questions. Further, some categories of reason for referral had corresponding questions that did not capture all patients in that category. For example, only 52% of patients with skin problems responded yes to the question asking about rash, and 24% of patients with adherence problems responded yes to these related questions (data not shown). The general health questions improve the overall performance, raising the percent captured to 63% and 41% for skin and adherence problems, respectively.

Table III. Summary of missing questions.

Reasons for recommended referral versus patient responses for questions not included in the protocol.

Category of problem	Number with this as reason for referral	Number who answered yes to feeling ill, other problems or other medical problems
Body pains, not peripheral	37	15 (41%)
Cardiac	4	2 (50%)
Eyes	10	2 (20%)
Diabetes	3	2 (67%)
Dizziness	1	1 (100%)
Anemia	5	0 (0%)
General Malaise	13	5 (38%)
Further Investigation	15	1 (7%)
Weight Loss	3	0 (0%)
Problems to Discuss	27	4 (15%)

The PDAs performed well during the study period although two had to be replaced during the study; one due to general failure (did not turn on) and the other due to problems with connectivity with the computer. Since the study was done in hospitals with continuous electricity, charging was not a problem and no data was lost.

Discussion

The initial PDA screening protocol included a list of 50 questions assessing patient reported symptoms, derived from the IMAI protocol. Although using all questions approach would capture nearly all patients requiring referral (Sensitivity = 96.0%), it would also refer most of the remaining patients as well (Specificity = 13.1%) and was therefore felt to be not useful in triaging patients. From this initial protocol, a final triaging protocol that included only fifteen questions performed well, with a sensitivity of 87% in the combined dataset. The protocol has only moderate specificity (47%). Since there is a greater hazard for sending sick patients home than referring well patients to a physician, a prioritized a protocol with high sensitivity (and lower specificity) to one with high specificity and low sensitivity. Although the overall sensitivity could be improved by adding more questions this resulted in a large reduction in specificity. We also found that positive responses to multiple questions (for example, both adherence problems and stomach pains) did not add to the predictive value of the protocol.

We also found significant discrepancies between what patients self-report and what physicians identify as problems necessitating full physician assessment. In a small number of cases, this is because the original PDA screening list did not include a question that matches the problem identified by the physician (Table III). In future protocols, we will add questions to better capture these problems. Further, the next iteration of the protocol development will pilot the question “Do you (the counselor/nurse) believe that the patient needs to see the doctor?” This question may capture patients who are visibly ill but are not responding “yes” to any of the symptom or side effect questions. In most of the misidentified patients with symptoms, the problem was that the response by the patient during the protocol screening was not consistent with the findings of the physicians. It may be that the wording of the question was misunderstood or that patients are reluctant to reveal some of their symptoms to non-medical personnel during the screening process. Further investigations are needed to fully understand these discrepancies.

Interestingly, the questions related to general health such as “How are you feeling today?”, “Do you have any other medical problems?” and “Do you have any other problems?” were originally intended as a transition into and out of the collection of patient self-reported symptoms and were not intended to be used in the referral protocol. However, these questions were very predictive of patients recommended for referral. This indicates that patients can identify poor health although they are often unable to describe or report the specific symptoms, similar to findings in other screening protocols [20].

One limitation of the study is the representativeness of the study setting. We conducted this study at two university affiliated academic centers where there were sufficient staff and patients to achieve an appropriate sample size and where the quality of the physician assessment could more credibly be used as a gold standard. As a result, they are not necessarily representative of the health care system as a whole and the patient population may have more advanced illnesses as compared to peripheral primary health care facilities. Another limitation was the exclusion of children, individuals within three months of ART initiation, and patients who came for an unscheduled visit due to an illness or concern. Future studies should explore the development of other triage protocols for these groups.

Another concern was the addition and removal of questions from during the course of the study. These changes were made at the request of the site study teams based on their expertise. For protocol development, patients screened during the phase when a particular question was not asked were excluded from analysis. This should not greatly affect the selection of questions for the protocol because the presentation of specific patients during a particular phase is “random”. During the validation stage, patients that did not receive a particular question were effectively treated the same as individuals who responded “no”. For this reason, we may underestimate the sensitivity and overestimate the specificity, though we expect this effect to be small.

One additional limitation is the failure to incorporate the full patient record into the protocol development. At the end of the study, we reviewed records of all included patients to identify individuals with laboratory results or indications at the past visit that would require referral. However, we were unable to link other data, such as previous weights to track weight loss. Newer algorithms that have been developed and are currently being validated do incorporate a patient record system including laboratory values, past weights, and other data. The use of PDAs or other electronic tools facilitate the inclusion of this historical data into a decision protocol.

The primary goal of this study was to develop and validate a protocol to support non-medical counselors in determining which patients should be referred to a physician for full clinical assessment in HIV treatment clinics in South Africa. If implemented, then every adult patient who has been on ART for at least three months would be screened at their routine visit. Only those recommended for referral when then be seen by the clinician. The remaining patients would receive their drugs and be allowed to go home without a full clinical assessment. The protocol developed here correctly identifies approximately 47% of the estimated 62% of patients who do not require referral. As a result, we believe this protocol has the potential to lower the physician burden by nearly 30%. Indeed this may understate the impact on physician burden since the patient population in this study is sicker than the general population under ART care. These results contribute to the mounting body of evidence that demonstrate both the feasibility and benefits of shifting tasks to a lower cadre of staff without diminishing the quality of care. Once a final protocol is developed and validated, the true impact of this task-shifting protocol will need to be evaluated in the clinical setting to assess impact on clinic operations.

In addition to developing and validating a screening protocol, we explored the use of an electronic support tool, in this case a PDA, to collect patient responses. Counselors collecting the patient responses reported that the PDAs were easy to use after very brief training sessions (usually one hour) and expressed an overall approval of the technology. Additionally, the PDA helps ensure complete data and implementation of appropriate sequence of questions. Future studies for refining the screening protocol will explore in detail the impact and additional benefits of collection of patient symptoms via a PDA.

Supplementary Material

01. Supplemental Table I - Patients missed for referral.

Summary of patients who were not captured by the referral protocol, including reasons referral was recommended, doctor actions, and patient responses to corresponding questions.

Summary Table.

This table summarizes points contained in this paper.

Current knowledge The low number of qualified clinicians limits the availability to provide universal HIV treatment coverage in Africa. Effective means of shifting tasks to lower cadres of health staff without reducing quality of care can increase the patient capacity at HIV treatment sites. One possible means of task shifting is triaging patients between those requiring full clinical assessment and those stable on treatment and who can go home without needing full clinical assessment.

Contribution of this paper The clinical protocol developed in this research has good discriminating abilities (sensitivity = 87%, specificity = 47%) for patient triaging. The analysis identified areas for improvement in this HIV treatment triaging protocols, including questions not initially asked and questions that were consistently answered incorrectly. Administering the triaging protocol on an electronic handheld device supports complete and thorough assessment by lower cadres of health staff. The PDAs were readily adopted by the clinic staff for this study.

Research Highlights.

• The protocol developed has good discriminating abilities for ART triaging

• The sensitivity of this protocol is 87% and specificity is 47%

• We identified areas for improvement for future ART triaging protocol studies

• The PDA supports complete and thorough assessment by lower cadres of health staff

• The PDAs were readily adopted by the clinic staff for this study