Validating Five Questions of Antiretroviral Nonadherence in a Public-Sector Treatment Program in Rural South Africa

Abstract

Simple questions are the most commonly used measures of antiretroviral treatment (ART) adherence in sub-Saharan Africa (SSA), but rarely validated. We administered five adherence questions in a public-sector primary care clinic in rural South Africa: 7-day recall of missed doses, 7-day recall of late doses, a six-level Likert item, a 30-day visual analogue scale of the proportion of doses missed, and recall of the time when an ART dose was last missed. We estimated question sensitivity and specificity in detecting immunologic (or virologic) failure assessed within 45 days of the adherence question date. Of 165 individuals, 7% had immunologic failure; 137 individuals had viral loads with 9% failure detected. The Likert item performed best for immunologic failure with sensitivity/specificity of 100%/5% (when defining nonadherence as self-reported adherence less than “excellent”), 42%/55% (less than “very good”), and 25%/95% (less than “good”). The remaining questions had sensitivities ≤17%, even when the least strict cutoffs defined nonadherence. When we stratified the analysis by gender, age, or education, question performance was not substantially better in any of the subsamples in comparison to the total sample. Five commonly used adherence questions performed poorly in identifying patients with treatment failure in a public-sector ART program in SSA. Valid adherence measurement instruments are urgently required to identify patients needing treatment support and those most at risk of treatment failure. Available estimates of ART adherence in SSA are mostly based on studies using adherence questions. It is thus unlikely that our understanding of ART adherence in the region is correct.

Introduction

Adherence to antiretroviral treatment (ART) is a crucial determinant of ART outcomes, including survival.^1–7 Nonadherence results in virologic and immunologic failure, clinical deterioration, and the development of viral resistance, complicating further treatment and increasing the risk of transmission of resistant virus.^2,4,8–10 Valid methods to assess adherence are thus an essential component of ART programs.^11,12 Questions on ART adherence are readily accessible, inexpensive, and easily and quickly administered in clinical settings.^9,13 They are thus a feasible method to monitor adherence in sub-Saharan Africa (SSA), where neither the human resources to perform more time-consuming adherence assessment (such as counting of antiretroviral pills or reviewing pharmaceutical records) nor the financial resources to conduct more costly assessment (such as electronic monitoring or monitoring of blood ART concentrations) may be available. In fact, in many of the public-sector ART programs in SSA, where three quarters of the world's four million people receiving ART live,^14–16 questions on adherence are the only measure routinely used to assess adherence.¹⁷

However, these questions are only useful if they perform well in identifying nonadherent patients.^10,18 But questions on ART adherence have rarely been validated as instruments to detect treatment failure in routine clinical settings in SSA's public sector. The few studies that have been conducted found unsatisfactory performance of ART adherence questions in identifying patients who fail treatment. In 34 patients in routine care in Uganda, 3-day recall of missed doses and a visual analogue scale were weakly correlated with viral load level 12 weeks after ART initiation.²⁴ In 238 individuals in a routine clinic in Cameroon, 1-month recall of missed doses had a very low sensitivity in detecting patients with virologic treatment failure.²⁵ The 2006 WHO treatment guidelines for resource-limited settings do not provide clear recommendations regarding adherence measurement.¹² The scarcity of validated ART adherence questions in SSA is all the more surprising because our understanding of ART adherence in the region is largely based on patient responses to adherence questions,¹⁹ and interventions to increase adherence in routine care are commonly evaluated using adherence questions as outcome measures.^20–23

Some studies outside of SSA have demonstrated significant associations between self-reported adherence and viral load¹³ and antiretroviral (ARV) plasma concentrations.²⁶ However, other studies found that self-reported adherence was only weakly correlated with viral suppression and antiretroviral drug concentrations.^27,28 It remains unclear whether questions on ART adherence are adequate instruments to assess adherence in routine clinical settings in SSA.^15,16,29,30 We validate for the first time the performance of five commonly used ART adherence questions against a gold standard of treatment failure in a public sector, decentralized ART program in rural KwaZulu-Natal, South Africa.

Methods

Study setting

Patients were enrolled in KwaMsane clinic, the primary care clinic managing the largest number of HIV patients within a public-sector ART program in the Hlabsia subdistrict of KwaZulu-Natal, South Africa.³¹ The target population of the clinic lives in a rural town, or in the surrounding semiurban and rural areas. ART is initiated at the clinic, free of charge, by a physician to all patients in the program with WHO stage IV disease or CD4 cell count less than 200 cells/mm³. Before ART initiation patients complete three treatment literacy sessions.¹⁷ The ART regimen, a first-line standard triple-drug regimen consisting of stavudine, lamivudine, and either efavirenz or nevirapine, is monitored and dispensed by nurses and counsellors at 2 weeks, 4 weeks, and at 4-week intervals thereafter. CD4 cell count and HIV viral load (VL) are routinely measured every 6 months. Approximately 6000 adult patients were initiated in the program between October 2004 and September 2008.³²

Adherence questions

After written informed consent, trained research staff with prior experience in HIV counseling and treatment administered five questions on ART adherence in isiZulu to patients attending the clinic. The five adherence questions comprising 7-day recall of missed doses, the number of doses missed by more than 2 hours in the past week, a Likert item with six levels of adherence over the past month, a 30-day visual analogue scale (VAS) of the proportion of doses missed, and a multiple-choice question with seven responses eliciting the time of the most recently missed dose within discrete time intervals (Fig. 1). Except for the VAS, which was completed by the patient after receiving instructions, self-reported adherence questions were completed by the research staff. In addition to the adherence questions, the participants answered questions on demographic, socioeconomic, and behavioral factors, as well as on health status and health care seeking. Clinical data, CD4 counts, and VL data were extracted from the clinic files of each enrolled patient.

FIG. 1.

Adherence questionnaire.

Samples

Patients who had received ART for at least 2 weeks, were not pregnant, and were not planning to stop ART in the next 6 months were enrolled between November 2007 and February 2008. Baseline analyses of the performance of adherence questions included all patients with a CD4 count (n = 165) or a viral load (n = 137) within ± 45 days of the date on which the adherence questions were administered.

Outcome definitions

Both immunologic and virologic failure definitions were based on WHO criteria for treatment failure,³³ as relevant for this study. Immunologic treatment failure was defined as less than 100 cells/mm³ after 6 months of ART. Virologic treatment failure was defined as a VL greater than 10,000 copies per milliliter after being on ART for over 6 months, or greater than 400 copies per milliliter after a previously undetectable viral load.

We started evaluating the performance of the different adherence questions, assuming that patients will fail treatment if they do not adhere near-perfectly,³⁴ i.e., considering any adherence level greater than 95% (for the 7-day recall of missed doses, the 7-day recall of late doses, and the VAS), any adherence level less than “excellent” (for the six-level Likert item), and any adherence level less than “never” (for the question about the time of the last missed dose) as “nonadherent.” We then assessed whether the adherence questions performed differently as diagnostic tests predicting treatment failure at alternative non-adherence cut-offs.³⁵ For the six-level Likert item on adherence, we used the additional nonadherence cutoffs less than “very good” and less than “good”; for the multiple-choice question on the time of the last missed dose, we used the additional cutoffs of any time more recent than “1 month” and more recent than “2 weeks.” For the remaining three questions, we used the additional cutoffs less than 85% and less than 75% of prescribed doses taken.

Statistical analysis

We used immunologic or virologic treatment failure as gold standard to evaluate the questions of nonadherence. Sensitivity, specificity, and corresponding 95% confidence intervals for varying cutoffs are reported for each self-reported adherence question. To assess the robustness of results to changes in the time of assessment of treatment failure relative to the date of the adherence questions, we also computed sensitivity and specificity for the samples of patients with biologic measures of treatment outcomes more than 3 months (resulting in samples of n = 165 for CD4 count and n = 139 for VL) or more than 6 months (n = 124 for CD4 count and n = 111 for VL) after the interview date. All analyses were conducted using STATA v11.0 (College Station, TX).

Results

Of individuals in the CD4 count sample, 12 (7%) had immunologic failure, while 13 (9%) in the VL sample had virologic failure. Within the virologic failure group, 44% percent had a VL greater than 10,000 copies/ml after 6 months of therapy. Patient characteristics are described in Table 1.

Table 1.

Sample Characteristics

	CD4 count sample N = 165	VL sample N = 137
Female (%)	76	81
Age (median [IQR])	38 (33–44)	38 (33–44)
Level of education
None (%)	10	9
Primary school (%)	30	28
Secondary school or higher (%)	60	63
Travel time to clinic in minutes [median (IQR)]	30 (15–40)	30 (20–40)
Disability grant for HIV (%)	45	47
Disclosure of HIV status to family or friend (%)	100	100
Time on ARV in months [median (IQR)]	14 (9–25)	14 (9–25)
Treatment failure
Immunologic failure (%)	7	—
Virologic failure (%)	—	9

IQR, interquartile range.

The six-level Likert item performed best in detecting immunologic failure, with sensitivity/specificity of 100%/5%, 42%/55%, and 25%/95% for immunologic failure when individuals reported less than “excellent,” less than “very good,” less than “good” adherence, respectively. The remaining four questions had sensitivities of 17% or less, even when the least strict cutoffs defined nonadherence. For virologic failure, the Likert item again performed best with sensitivity/specificity of 92%/3%, 23%/55%, and 8%/95% for immunologic failure when individuals reported less than “excellent,” less than “very good,” or less than “good” adherence, respectively. The remaining four questions had sensitivities 8% or less, even when the least strict cutoffs defined nonadherence (Table 2). Note that the confidence intervals around the sensitivity estimates are substantially wider than those around the specificity estimates, because the sample sizes for sensitivity estimation were substantially smaller than those for specificity estimation as most individuals in our sample had not failed treatment.

Table 2.

Sensitivity and Specificity of Adherence Questions in Detecting Treatment Failure

Question item	Nonadherence cutoff	Number reporting nonadherence	Sensitivity	95% CI	Specificity	95% CI
CD4 count sample (n = 165)
7-day recall of missed doses	<95%	6	0	0–26	96	92–99
	<85%	1	0	0–26	99	96–100
	<75%	1	0	0–26	99	96–100
7-day recall of missed doses ≥2 h	<95%	5	0	0–26	97	93–99
	<85%	1	0	0–26	99	96–100
	<75%	0	0	0–2	100	74–100
Six-level Likert item	<Excellent	157	100	74–100	5	2–9
	<Very good	73	42	15–72	55	47–63
	<Good	11	25	5–57	95	90–98
30-day VAS	<95%	7	0	0–26	95	91–98
	<85%	0	0	0–2	100	74–100
	<75%	0	0	0–2	100	74–100
Last missed dose	<Never	27	17	2–48	84	77–89
	<1 month	10	0	0–26	93	88–97
	<2 weeks	1	0	0–26	99	96–100
VL sample (n = 137)
7-day recall of missed doses	<95%	4	0	0–25	97	92–99
	<85%	0	0	0–3	100	75–100
	<75%	0	0	0–3	100	75–100
7-day recall of missed doses ≥2 h	<95%	4	0	0–25	97	92–99
	<85%	1	0	0–25	99	96–100
	<75%	0	0	0–3	100	75–100
Six-level Likert item	<Excellent	131	92	64–100	3	1–8
	<Very good	59	23	5–54	55	45–64
	<Good	7	8	0–36	95	90–98
30-day VAS	<95%	3	0	0–25	98	93–100
	<85%	1	0	0–25	99	96–100
	<75%	0	0	0–3	100	75–100
Last missed dose	<Never	22	8	0–36	83	75–89
	<1 month	6	0	0–25	95	90–98
	<2 weeks	1	0	0–25	99	96–100

CI, confidence interval; VAS, visual analogue scale; VL, viral load.

Question performance remained essentially unchanged when assessed against virologic failure or against combined immunologic or virologic failure. Neither sensitivity nor specificity of the questions improved significantly when CD4 counts (or VLs) more than 3 months or more than 6 months after the date of the adherence question were used as gold standard in the analysis. Similarly, when we identified nonadherent individuals through an initial screening test (using the highly sensitive six-level Likert item) and then confirmed the diagnosis of nonadherence with a second adherence question (using one of the four highly specific questions), test performance did not improve significantly in comparison to the individual questions alone. None of the other possible combinations of adherence questions, including combinations of more than two questions, had better performance characteristics than the best-performing individual question included in the combination.

We further tested whether the performance of the adherence questions was better in particular subsamples than in the total sample by stratifying the analysis by sex, age (below 35 versus 35 or older), and education level (below secondary education versus secondary education or higher). While the performance of the questions varied across the strata, it did not improve significantly in any of the strata in comparison to the performance in the total sample. For instance, in the validation against CD4 count the performance characteristics of the best-performing measure (the six-level Likert item with a nonadherence cutoff of less than “very good”) differed between the stratum with higher education level (50% sensitivity (95% confidence interval [CI] 16–84%) and 54% specificity (95% CI 45–64%) and the total population (42% sensitivity [95% CI 15–72%] and 55% specificity (95% CI 47–63%). However, these differences were neither substantial nor statistically significant.

Discussion

Questions on ART adherence are the most widely used instruments to measure adherence in treatment programs in SSA. We find that five questions commonly used in clinical research and practice to assess adherence, including the one currently used in the public sector in South Africa, perform poorly in detecting patients who fail treatment. Estimates of ART adherence in SSA are mostly based on responses to adherence questions.¹⁹ Our findings thus suggest that ART adherence in the region has been overestimated.

Reasons for ART failure other than nonadherence are generally rare in our context.³⁶ However, it is theoretically possible that a proportion of patients failed treatment despite current high levels of adherence, resulting in an underestimate of sensitivity. First, primary viral resistance could, of course, have resulted in treatment failure in perfectly adhering patients.^37–39 But, according both to results from modeling studies⁴⁰ and local empirical evidence^41,42 primary resistance was very rare in South Africa during the time of this study. Hence, we would not expect primary resistance to have affected our results.

Second, concomitant treatment for tuberculosis (TB) can interact with some antiretroviral drugs, leading to subtherapeutic ART concentrations.^43,44 As Boulle et al.⁴⁵ showed, however, TB treatment only affects the probability of virologic failure in patients receiving the standard South African triple-drug regimen with nevirapine, which is only recommended for “pregnant women or women of child-bearing age, not on reliable contraception,”¹⁷ while it does not affect the regimen with efavirenz. The South African treatment guidelines thus specifically recommend the regimen containing efavirenz for all patients with TB coinfection.¹⁷ In all samples in this study, less than one quarter of the patients received a regimen with nevirapine, and less than one fifth of this group was simultaneously treated for TB. It is thus unlikely that TB treatment was responsible for a significant proportion of treatment failures in this study.

Third, HIV-related conditions may reduce ART blood-levels, leading to treatment failure despite perfect adherence. For example, HIV-associated enteropathy can cause vomiting, diarrhea, or malabsorption, reducing ART blood levels and manifesting as treatment failure.^46,47 Finally, it has been suggested that some of the medicines given by traditional healers in KwaZulu-Natal can change the pharmacokinetics of ART.⁴⁸ However, all patients failing treatment according to either CD4 count or VL reported they did not receive care from the two types of traditional healers—sangoma and inyanga—who most commonly prescribe traditional medicines in this area.⁴⁹ Overall, it is thus unlikely that reasons for treatment failure other than nonadherence have biased our findings.

Another potential source of bias in our baseline results are the lag times between adherence and immunologic or virologic response.⁵⁰ But our results remained essentially unchanged when we used only CD4 count or VL in the analyses that were measured 3 or 6 months after the date of the adherence assessment, demonstrating that lags are unlikely to have affected our findings.

A general limitation of most adherence questions is that they provide little information on the longitudinal pattern of adherence. For example, patients answering to the multiple-choice question that they missed the last dose “yesterday” could theoretically have missed one dose the day before but never missed a dose previously or they could have consecutively missed doses over several past days including the day before. Similarly, the VAS does not differentiate between all the different possible time patterns of nonadherence that imply any given proportion of doses missed over the past 30 days. Because the pattern by which patients fail to take their medication can have significant impact on their treatment outcomes and the development of resistance,^37,39,51 the inability of self-reported adherence to capture adherence time patterns limits their utility in detecting patients with adherence problems. Once a patient has been identified as nonadherent, a more extensive dialogue with the patient may be necessary to identify the patterns and causes of non-adherence, in order to determine appropriate adherence support intervention.¹⁴

The location where an adherence question is asked may be an important determinant of the validity of the answer. For instance, when answering an adherence question in a public-sector clinic where ART are free of charge a patient may be more likely to fear negative consequences of reporting nonadherence (such as treatment discontinuation) than when answering the same question in a private-sector clinic where she pays for her ART. In this study in a public-sector clinic, we emphasized in the informed consent procedure that answers to the questions in the study questionnaire would be completely confidential and that reporting nonadherence would not lead to any negative consequences, in order to minimize intentional adherence misreporting. Nevertheless, future studies should examine whether the adherence questions used in this study perform better in identifying nonadherent patients when asked outside the setting of a public-sector clinic.

Adherence is a crucial determinant of ART success. Note, however, that different ART regimens may require different minimum levels of adherence to ensure successful treatment. A minimum adherence threshold of 95% of all prescribed doses taken was established initially as necessary for patients receiving an unboosted protease inhibitor and two nucleoside reverse transcriptase inhibitors to ensure the highest probability of viral suppression.⁷ More recent reports indicate that more moderate levels of adherence may be sufficient to ensure viral suppression for the majority of patients receiving boosted protease inhibitor-based regimens or nonnucleoside reverse transcriptase inhibitor-based regimens.^52,53 Since most patients in public-sector ART programs in South Africa, such as the patients in our sample, receive non-nucleoside reverse transcriptase inhibitors (either efavirenz or nevirapine), it is possible that adherence levels below 95% are sufficient for treatment success in South Africa. Future studies need to establish the precise minimum adherence thresholds for the routine first-line ART regimens in this setting.

Independent of the precise minimum adherence required for treatment success, the ability to simply and validly measure ART adherence is crucial for the long-term success of the recent initiatives to bring ART to scale in SSA. It is necessary to detect patients who are failing treatment and require additional support, in order to prevent the development of resistance. In the absence of resistance testing, it is further important to identify those who despite ongoing and reinforced adherence still fail treatment and thus require regimen change.⁵⁴ Our study shows that simple questions, which are commonly used in routine care, are insufficient for this purpose. It will be important to understand the reasons for the poor performance of these adherence questions in the public-sector treatment programs in South Africa, which could include social desirability bias, imperfect recall, or misinterpretations of the purpose of the questions.^{1,2,14,55–57} It will further be important to reassess our knowledge on ART adherence in SSA. Available estimates of ART adherence in the region are mostly based on studies using adherence questions. It is thus unlikely that our understanding of ART adherence in the region is correct.

Instruments that can be used in ART programs in SSA to identify individuals adhering imperfectly are urgently needed in order to provide additional support to prevent treatment failure and resistance development. Ideally such instruments would be neither costly nor time consuming. However, if instruments that are inexpensive and quick to administer, such as patient adherence questions or health worker estimates of adherence,⁵⁸ do not lead to valid adherence estimates, policy makers need to consider whether more resource-intensive measures, such as pill counts or electronic monitoring, should be routinely employed in public-sector treatment programs in SSA.

Acknowledgments

We thank the patients and program staff in the ART program in Hlabisa subdistrict for their support of this study. The study was financed through a pilot project grant from the National Institute of Aging (NIA), National Institutes of Health (NIH), through the Program on the Global Demography of Aging at Harvard University. The ART program received financial support from the US President's Emergency Fund for AIDS Relief (PEPFAR). Till Bärnighausen and Frank Tanser were supported by Grant 1R01-HD058482-01 from the National Institute of Child Health and Human Development (NICHD), NIH.

Author Disclosure Statement

No competing financial interests exist.