Using a Self-Administered Electronic Adherence Questionnaire to Identify Poor Adherence Amongst Adolescents and Young Adults on First-Line Antiretroviral Therapy in Johannesburg, South Africa

Abstract

Introduction

The best method to measure adherence to antiretroviral therapy (ART) in resource-limited settings has not yet been established, particularly among adolescents and young adults (AYAs). The use of mobile technology may address the need for standardized tools in measuring adherence in this often marginalized population.

Methods

We conducted a cross-sectional validation study among AYAs (18–35 years) attending a South African HIV clinic between 07/2015-09/2017. We determine the diagnostic accuracy of two modes of delivering an adherence questionnaire (self-administered electronic vs interviewer-administered paper-adherence questionnaire) comprising two self-reported adherence tools (South African National Department of Health (NDoH) adherence questionnaire and the Simplified Medication Adherence Questionnaire (SMAQ)) to identify poor adherence compared to; 1) a detectable viral load (≥1000 copies/mL) and 2) a sub-optimal concentration of efavirenz (EFV) (EFV ≤1.00 µg/mL) measured by therapeutic drug monitoring (TDM).

Results

Of 278 included participants, 7.1% and 7.3% completing the electronic- and paper-questionnaires had a detectable viral load, while 14.7% and 16.5% had a sub-optimal concentration of EFV, respectively. According to viral load monitoring, the electronic-adherence questionnaire had a higher sensitivity (Se) in detecting poor adherence than the paper-based version across the NDoH adherence questionnaire (Se: 63.6% vs 33.3%) and SMAQ (Se: 90.9% vs 66.7%). In contrast, when using blood drug concentration (EFV ≤1.00 µg/mL), the paper-adherence questionnaire produced a higher sensitivity across both adherence tools; namely the NDoH adherence questionnaire (Se: 50.0% vs 38.1%) and SMAQ (Se: 75.0% vs 57.1%).

Conclusion

When using more accurate real-time measures of poor adherence such as TDM in this young adult population, we observe a higher sensitivity of an interviewer-administered paper-adherence questionnaire than an identical set of self-administered adherence questions on an electronic tablet. An interviewer-administered questionnaire may elicit more accurate responses from participants through a sense of increased accountability when engaging with health care workers.

Introduction

While HIV incidence has been declining worldwide in recent years,¹ sub-Saharan Africa (SSA) still bears a disproportionate burden of the disease.² There are an estimated 37 million people infected with HIV worldwide.³ Specifically, some 4 million adolescents and young adults (AYAs) aged 15 to 24 years are infected with the virus globally.⁴ This sub-population faces many unique challenges in HIV-treatment access, adherence and subsequently, the achievement of favorable treatment outcomes.

Adolescence is a transitionary period between child- and adulthood, and is often associated with rebelliousness, identity formation, risk-taking behaviour and sexual experimentation.^5–7 Consequently, this group has been identified as being particularly susceptible to HIV infection. Moreover, due to unclear adolescent patient confidentiality policies and the possible judgement and unfriendliness of health care workers, this young population is often marginalized from mainstream health services, subsequently limiting sustained access to treatment and services.⁸ In turn, traditional markers of successful antiretroviral therapy (ART) such as retention in care and virologic suppression remain poorer among AYAs compared to their older adult counterparts.^9–12

While optimal levels of adherence to ART have been defined, a lack of consensus surrounding the best method to measure adherence, particularly in resource-limited settings (RLS), necessitates the need for consistent/standardized measurement tools.^13,14 Modes of measurement can be categorized as direct methods (e.g., biological assays and other markers in the blood, urine or body fluids that measure drug concentrations in the individual patient) and indirect methods (e.g., self-report tools such as visual analogue scales (VAS) and pill identification tests (PIT) as well as missed visits, prescription/pharmacy re-fills and electronic drug monitoring systems).¹⁵ While self-reported indirect methods tend to be commonly used in RLS (through structured patient-interviews), this mode of measurement is often subject to reporting and recall bias resulting in a general over-estimation of true adherence.^16,17

A promising approach identified to specifically address the challenges unique to AYAs in RLS has been the use of mobile technologies. In particular, the use of smartphones and tablets may be especially appealing to this sub-population as they tend to be both early adopters and high impact users of such technology.¹⁸ In particular, when compared to traditional interviewer-administered paper questionnaires, self-administered electronic questionnaires may reduce response bias by allowing for more honest reporting of sensitive information and unprescribed behavior.¹⁹ This coupled with a potential reduction in data entry errors through logic checks, more accurate adherence tracking and increased efficiency in data storage and management may make the use of mobile technology a viable option in measuring adherence to ART.

While virologic and treatment failure can be attributed to drug toxicity or resistance, it is most commonly a function of poor adherence.²⁰ As such, viral load is often considered a marker of poor adherence. However, it is important to note that discrepancies between viral load monitoring (elevated or detectable viral load) and treatment adherence (optimal levels of adherence) have previously been reported.^21,22 This sub-group of patients, who are in fact truly adherent, may then report detectable levels of virus due to possible drug resistance. In South Africa, viral load is routinely used to monitor ART and identify treatment failure (standard of care).²³ In routine practice, patients with a detectable viral load (≥1000 copies/mL) on first-line ART are referred for intensive adherence counselling followed by repeat viral load testing 2–4 months later. Patients who resuppress (<400 copies/mL) continue to receive the standard of care, while those who do not are switched to more expensive second-line regimens.²³ In this way, viral load monitoring is used as a direct measure of treatment failure and provides only a proximal measure of adherence. In most cases, viral load monitoring is only effective after sustained periods of poor adherence which result in an elevated viral load above a pre-determined threshold (early detection of treatment failure but late detection of potential poor adherence – late response).

Therapeutic drug monitoring (TDM), on the other hand, has not been routinely implemented due to high costs, accessibility and complexity which makes its application in RLS challenging.^24–26 TDM does however provide more robust real-time estimates of poor adherence as actual blood drug concentrations are measured (early detection). Therefore, the combined use of both these measures, viral load monitoring and TDM, may provide a better estimate of poor adherence.

In this study we compare the sensitivity (Se), specificity (Sp), positive (PPV) and negative predictive value (NPV) of two modes of delivering an adherence questionnaire (1) a self-administered self-reported electronic tablet-based adherence questionnaire (EAQ) vs (2) standard of care interviewer-administered self-reported paper-based adherence questionnaire (PAQ) comprising two different adherence tools (South African National Department of Health (NDoH) adherence questionnaire and the Simplified Medication Adherence Questionnaire (SMAQ)) to identify poor adherence compared to; 1) a detectable plasma viral load (≥1000 copies/mL) and 2) a sub-optimal concentration of efavirenz (EFV) (EFV ≤1.00 µg/mL) in serum measured by TDM.

Methods

Study Design and Population

We conducted a cross-sectional validation study among HIV-positive AYAs (18–35 years at study enrollment) on first-line ART for ≥12 weeks at a large urban HIV outpatient clinic, located at a secondary hospital in Johannesburg, South Africa. Participants had to be returning to the clinic for a routine viral load test between 07/2015-09/2017 as per national guidelines (first viral load test administered at 6 months post-ART initiation, followed by at 12 months and 12 monthly thereafter).²³ Study eligibility was based on participants’ ability to speak and understand English, willingness to use an electronic tablet for the completion of an ART adherence questionnaire and to provide an extra blood sample for TDM. Participants who were pregnant at enrollment were ineligible to participate due to potentially altered and highly variable pharmacokinetics,^27,28 which may have affected TDM results. Participants who were too sick and those with a previously elevated viral load result in the 12 weeks preceding enrollment were also excluded as we wanted to restrict the study to participants returning for routine viral load monitoring (i.e., only consider the first elevated viral load ≥1000 copies/mL) and not those returning to confirm virologic failure.^26–30

Study Site

HIV care and treatment at the study site follows the South African NDoH ART Guidelines.^23,29–32 Through support of a non-governmental organization (NGO), an adolescent clinic within the study site provides additional counselling to AYAs during clinic visits. All demographic, longitudinal clinical and laboratory data are captured on an electronic patient management system called TherapyEdge-HIV™ (TE).³³ Laboratory data (e.g., CD4 counts and viral load results) were obtained daily through an automated download of electronic records held by the National Health Laboratory Services (NHLS), the primary provider of laboratory services for South Africa’s public sector clinics. During the study period, South Africa introduced a universal test and treat (UTT) policy within its public-sector HIV treatment program (2016) which called for ART initiation among all known HIV-positive persons, irrespective of CD4 cell count.³² While this change may have occurred during study enrollment, first-line ART regimens and treatment monitoring (timing of viral load tests) remained the same during the study period and therefore no direct impact of guideline changes was observed during study procedures.

Procedures

A convenience sample was obtained by approaching patients who presented at the clinic on a daily basis (Monday–Friday). Interviewers screened and confirmed eligibility prior to obtaining written informed consent. Once participants provided informed consent, interviewers assigned a unique study ID and also recorded the medical record ID so that responses to the adherence questionnaire could be linked to the participants’ electronic medical records.

To minimize variability of socio-demographic and clinical characteristics between respective modes of delivery (electronic- vs paper-adherence questionnaire), block randomization using Microsoft Excel was used to assign eligible participants to respective adherence questionnaires (each participant completed only one questionnaire, either the electronic- or paper-adherence questionnaire). To reduce selection bias, the outcome of the randomization was only revealed after participants had provided consent to participate and were enrolled in the study. While study analysts were aware of the randomization pre-consent, patient interviewers and participants became aware of the randomization outcome post-consent, and laboratory personnel processing and managing viral load and TDM results remained blinded throughout the study period.

The two different modes of delivering an adherence questionnaire (self-administered electronic vs interviewer-administered paper-adherence questionnaire) comprised two different adherence tools; 1) South African National Department of Health adherence questionnaire and 2) Simplified Medication Adherence Questionnaire based on the modified Morisky scale of adherence.³⁴ Adherence tools included items for self-reported adherence during the period preceding the assessment (missed doses up to 3 months), a visual analogue scale in which participants classified their adherence as a measure from 0–10 (0 being not good at all, and 10 being very good), forgetfulness, routine, effects of adverse reactions and pill identification tests (i.e., questions about the name of the medication, number of pills per dose, time the medication is taken and if the patient knows any additional instructions such as storage in a refrigerator, take with food or avoid other medications) (see Figure 1).³⁵

Figure 1.

Continued..

To assess the feasibility of the different modes of delivery (self-administered electronic vs interviewer-administered paper-adherence questionnaire), an identical set of six 5-point Likert scale questions (strongly agree, agree, neutral, disagree, strongly disagree) were also included in each questionnaire (see Table S1). Additionally, we explored eight open-ended questions at the end of each questionnaire to understand participants’ likes/dislikes of adherence counselling and experiences with the electronic- or paper-based adherence questionnaire (see Table S1). When analysing qualitative data, we conducted a spell-check and rapid assessment to address consistency of language (this included spelling errors and adjusting from common short message service (SMS) writing format (e.g., bcos=because; hv=have; nd=and; ppl=people, etc.)). We used NVivo 11 and report on the most common responses and provide a few illustrative examples from each question. When appropriate we compared the difference between the electronic- vs paper-adherence questionnaire. All questionnaires were administered in English.

Participants randomized to complete the self-administered electronic-adherence questionnaire did so with minimal assistance from interviewers. Information cards with step-by-step instructions and graphics were available to participants while completing their adherence assessment. Participants completed the electronic-adherence questionnaire in a private study room, away from the general clinic population. The electronic-adherence questionnaire was modified to be administered on a touch screen Samsung Galaxy 3 10.1 inch P5200 Tablet, and designed on iFormBuilder, a universal, cloud-based mobile data collection platform from Zerion software.³⁶ Upon completion of the electronic-adherence questionnaire, participants had to select the “Done” option at the end of the questionnaire at which time all patient-related data were encrypted, uploaded to iFormBuilder and removed from the Samsung tablet through an automated process built-in to the software.

Participants randomized to the counsellor/interviewer-administered paper-adherence questionnaire met with an interviewer who asked adherence questions (NDoH and SMAQ adherence tools) and completed a paper-based form, as is the standard of care at the clinic. As with the electronic-adherence questionnaire, all interviews were conducted in a private room, away from the general clinic population. Patient interviewers later entered data from paper-based forms into iFormBuilder.

After completing either the electronic- or paper-based questionnaire, participants were referred to the clinics blood room for routine viral load monitoring. Clinic staff subsequently conducted the routine viral load blood draw and drew an extra blood sample for TDM. Viral load blood samples were sent directly to NHLS for processing and results were returned generally within 14 days from the date of blood draw. Viral load results were merged with adherence data via unique study identifiers.

TDM blood samples collected at the time of viral load blood draw were stored on site at room temperature and collected daily by Central Laboratory Services (CLS). On arrival at CLS, samples were centrifuged at approximately 1500 g for 10 mins, decanted into labeled cryovials (“P” for primary and “D” for duplicate) and stored at −20 degrees Celsius. Samples were then shipped overnight on dry-ice to the Division of Clinical Pharmacology, University of Cape Town (UCT), and stored until testing. Each sample was tested using a validated liquid chromatography (LC) mass spectrometry (MS)/MS semi-quantitative method for the determination of EFV. Each sample tested was classified into four distinct categories; <0.02 µg/mL; 0.02 µg/mL - 0.05 µg/mL; 0.05 µg/mL - 1.00 µg/mL; and >1.00 µg/mL (personal communication Jennifer Norman, Laboratory Director, Routine TDM Service, Division of Clinical Pharmacology).

All participants (randomized to complete either the electronic- or paper-adherence questionnaire) were reimbursed ZAR100.00 for their time and travel costs. Reimbursement was only disclosed after the screening process to prevent participants from providing inaccurate information regarding their study eligibility.

Study Variables

Patient demographic and clinical characteristics at study enrollment including sex, age, education, employment and current first-line ART regimen were extracted from the electronic patient medical record and respective adherence questionnaires. Time on ART was calculated from the date of ART initiation (obtained from patient medical records) to study enrollment and categorized between 3–12 months and ≥12 months. Monitoring bloods such as CD4 cell count was available from the electronic patient record and measured closest to date of study enrollment (12 months before to 14 days post-study enrollment to capture the latest available results as prescribed by national guidelines).

In keeping with standardized definitions across adherence tools, non/poor adherence according to the NDoH adherence protocol^35,37 and SMAQ protocol³⁴ was defined through a combination of responses (see Figure 1).

Figure 1.

Poor adherence by South African National Department of Health (NDoH) adherence tool vs Simplified Medication Adherence Questionnaire (SMAQ) tool.

Notes: *Pill identification questions appear once in each questionnaire (electronic- vs paper-adherence questionnaire) but are included in definitions of poor adherence in both the NDoH tool and SMAQ tool. ^±Poor adherence defined by the NDoH adherence tool: "Yes" response to Q1, Q2, Q3, Q4 or reported <8 in Q5, or could not identify more than two thirds of their prescribed antiretroviral (ARVs) drugs, nor identify the correct time their medication should be taken or the number of pills to be taken (Q14). ^¥Poor adherence defined by the SMAQ adherence tool: "Yes" response to Q5, Q7, Q8, Q9, Q11 or "No" reponse to Q6 or non- "Never" response to Q10 or "More than 2 days" response to Q12 or non- "Out of 30 tablets more than 27 tablets" response to Q13 or could not identify more than two thirds of their prescribed antiretroviral (ARVs) drugs (Q14). Data from Knobel et al.³⁴

Statistical Analysis

Patient demographic and clinical characteristics at study enrollment were summarized using frequencies for categorical variables and median and interquartile range (IQR) for continuous variables. Where necessary we cross-tabulated proportions of categorical variables. Sensitivity, specificity, positive and negative predictive value of each mode of delivery (electronic- vs paper-adherence questionnaire) and of the different tools (NDoH and SMAQ) were determined against laboratory definitions of poor adherence; 1) a detectable viral load (≥1000 copies/mL) and 2) sub-optimal concentration of EFV in the blood (EFV ≤1.00 µg/mL) (personal communication with Taryn Pillay from NHLS, 26 September 2018).

Results

Cohort Description

During the study period, a total of 290 participants met eligibility criteria and were enrolled. A total of 160/290 (55.2%) and 130/290 (44.8%) participants were enrolled and randomized to complete either the electronic- or paper-adherence questionnaire, respectively. Following enrollment, 3/160 (1.9%) participants completing the electronic- and 3/130 (2.3%) participants completing the paper-adherence questionnaire were found to be ineligible through data/eligibility verification via electronic medical record review (previously elevated viral load in the 12 weeks preceding enrollment/re-enrolled). Of the remaining 157 participants completing the electronic- and 127 participants completing the paper-adherence questionnaire, 3/157 (1.9%) and 3/127 (2.4%) had missing viral load results at study enrollment (unsuitable/contaminated sample) and were subsequently excluded from the analysis. The final analytic sample comprised of 278 participants; 154 (55.4%) completing the electronic- and 124 (44.6%) completing the paper-adherence questionnaire (see Figure 2).

Figure 2.

Screening, randomization and enrollment procedure.

Demographic and Clinical Characteristics

The proportion female was similar between participants completing the electronic- and paper-adherence questionnaire, respectively (78.6% vs 79.0%) as was the median age at initiation (30.7 vs 31.8 years), median time on ART (42.0 vs 42.3 months) and current first-line ART regimen (89.5% vs 92.7% on a fixed-dose combination (FDC)). Participants completing the electronic-adherence questionnaire had higher levels of tertiary education (24.0% vs 12.1%) and employment (55.2% vs 49.2%). The CD4 count categories (closest to study enrollment) were similar between those randomized to electronic- vs paper-adherence questionnaire, respectively (<100 cells/mm³; 0.8% vs 1.1%, 100–350 cells/mm³; 26.9% vs 23.1% and ≥350 cells/mm³; 72.3% vs 75.8%) (see Table 1).

Table 1.

Demographic and Clinical Characteristics of Patients Completing Either a Self-Administered Electronic or Interviewer Administered Paper-Adherence Questionnaire at Study Enrollment (N=278)

	Electronic Adherence Questionnaire (n=154)	Paper Adherence Questionnaire (n=124)	Total (n=278)
	July 2015–September 2017	July 2015–September 2017
n (%)	n (%)	n (%)
Sex
Female	121 (78.6%)	98 (79.0%)	219 (78.8%)
Male	33 (21.4%)	26 (21.0%)	59 (21.2%)
Age at enrollment (years)
Median (IQR)	30.7 (27.6–33.8)	31.8 (26.3–34.0)	30.9 (27.0–34.0)
18–30	69 (44.8%)	51 (41.1%)	120 (43.2%)
≥30	85 (55.2%)	73 (58.9%)	158 (56.8%)
Education
None	6 (3.9%)	0 (0.0%)	6 (2.2%)
Primary	4 (2.6%)	2 (1.6%)	6 (2.2%)
Secondary	107 (69.5%)	107 (86.3%)	214 (77.0%)
Tertiary	37 (24.0%)	15 (12.1%)	52 (18.7%)
Employment
No	69 (44.8%)	63 (50.8%)	132 (47.5%)
Yes	85 (55.2%)	61 (49.2%)	146 (52.5%)
Current first-line regimen
3TC+ABC+EFV	1 (0.7%)	5 (4.0%)	6 (2.2%)
3TC+TDF+EFV/NVP	10 (6.6%)	1 (0.8%)	11 (4.0%)
AZT+3TC+EFV	1 (0.7%)	0 (0.0%)	1 (0.4%)
TDF+FTC+EFV (FDC)	136 (89.5%)	114 (92.7%)	250 (90.9%)
TDF+FTC+NVP	1 (0.7%)	0 (0.0%)	1 (0.4%)
d4T+3TC+EFV/NVP	3 (2.0%)	3 (2.4%)	6 (2.2%)
Other	2 (1.3%)	1 (0.8%)	3 (1.1%)
CD4 cell count (cells/mm3)
Median (IQR)	468.0 (341.0–607.0)	527.0 (353.0–710.0)	489.0 (344.0–652.0)
0–50	1 (0.8%)	0 (0.0%)	1 (0.5%)
51–100	0 (0.0%)	1 (1.1%)	1 (0.5%)
101–200	9 (7.6%)	4 (4.4%)	13 (6.2%)
201–350	23 (19.3%)	17 (18.7%)	40 (19.1%)
≥350	86 (72.3%)	69 (75.8%)	155 (73.8%)
Total time on ART (months)a
Median (IQR)	42.0 (17.8–65.0)	42.3 (18.5–63.6)	42.1 (18.4–64.8)
3–12	24 (15.6%)	19 (15.3%)	43 (15.5%)
≥12	130 (84.4%)	105 (84.7%)	235 (84.5%)

Note: ^aTime from first-line ART initiation to date of study enrollment.

Abbreviations: IQR, interquartile range; 3TC, lamivudine; ABC, abacavir; EFV, efavirenz; TDF, tenofovir; NVP, nevirapine; AZT, zidovudine; d4T, stavudine; FTC, emtricitabine.

Diagnostic Accuracy of Adherence Tools

Detectable Viral Load (≥1000 copies/mL) (Standard of Care – Late Response)

The proportion of participants with a detectable viral load (≥1000 copies/mL) was similar between participants completing the electronic-adherence questionnaire (7.1%) and paper-adherence questionnaire (7.3%). Overall, 7.2% of all participants (20/278) had a detectable viral load at study enrollment (see Table 2).

Table 2.

Measures of ART Adherence Among Patients Completing Either a Self-Administered Electronic or Interviewer Administered Paper Adherence Questionnaire at Study Enrollment (N=278)

	Electronic Adherence Questionnaire (n=154)	Paper Adherence Questionnaire (n=124)	Total (n=278)
	July 2015–September 2017	July 2015–September 2017
n (%)	n (%)	n (%)
Viral load detectable (≥1000 copies/mL)
No	143 (92.9%)	115 (92.7%)	258 (92.8%)
Yes	11 (7.1%)	9 (7.3%)	20 (7.2%)
Therapeutic drug monitoring (MEC)a
EFV (adherent; >1.00 µg/mL)	122 (85.3%)	101 (83.5%)	223 (80.8%)
EFV (poorly-adherent; ≤1.00 µg/mL)	21 (14.7%)	20 (16.5%)	41 (15.5%)

Note: ^an=264.

Abbreviations: EFV, efavirenz; MEC, minimum effective concentration.

The sensitivity of the NDoH adherence tool in detecting poor adherence defined by a detectable viral load (≥1000 copies/mL) at study enrollment was 50% (95% confidence interval (CI): 28.9–71.1). When stratified by mode of delivery, it was higher among participants completing the electronic-adherence questionnaire (63.6% (95% CI: 33.6–87.2)) compared to those who completed the paper-adherence questionnaire (33.3% (95% CI: 9.3–66.8)). The sensitivity of the SMAQ adherence tool was 80.0% (95% CI: 58.5–93.3). As with the NDoH, this was higher among participants completing the electronic-adherence questionnaire (90.9% (95% CI: 62.7–99.5)) when compared to participants randomized to complete the paper-adherence questionnaire (66.7% (95% CI: 33.2–90.7)). The PPV of the adherence tools were low, with the highest reported for the NDoH tool on an electronic adherence questionnaire (11.7%). In contrast, NPVs ranged from 93.1%–98.5% indicating that those AYAs who report good adherence were in fact truly adherent (see Table 3).

Table 3.

Sensitivity and Specificity of Questionnaire Type in Detecting Poor Adherence by Viral Load Response at Study Enrollment (N=278)

Questionnaire Type	Adherence	VL ≥1000 Copies/mL	VL <1000 Copies/mL	Sensitivity (95% CI)	Specificity (95% CI)	Positive Predictive Value (95% CI)	Negative Predictive Value (95% CI)
SA NDoH overall (EAQ+PAQ)	Good	10/20 (50.0%)	171/258 (66.3%)	50.0% (28.9–71.1)	66.3 (60.3–71.9)	10.3% (6.7–15.5)	94.5% (91.6–96.4)
Poor	10/20 (50.0%)	87/258 (33.7%)
SA NDoH PAQ	Good	6/9 (66.7%)	81/115 (70.4%)	33.3% (9.3–66.8)	70.4 (61.6–78.2)	8.1% (3.3–18.8)	93.1% (89.3–95.6)
Poor	3/9 (33.3%)	34/115 (29.6%)
SA NDoH EAQ	Good	4/11 (36.4%)	90/143 (62.9%)	63.6% (33.6–87.2)	62.9% (54.8–70.6)	11.7% (7.5–17.8)	95.7% (91.1–98.0)
Poor	7/11 (63.6%)	53/143 (37.1%)
SMAQ overall (EAQ+PAQ)	Good	4/20 (20.0%)	112/258 (43.4%)	80.0% (58.5–93.3)	43.4% (37.5–49.5)	9.9% (7.9–12.3)	96.6% (92.-98.6)
Poor	16/20 (80.0%)	146/258 (56.6%)
SMAQ PAQ	Good	3/9 (33.3%)	48/115 (41.7%)	66.7% (33.2–90.7)	41.7% (33.0–50.9)	9.5% (6.0–14.7)	94.1% (86.1–97.6)
Poor	6/9 (66.7%)	57/115 (58.3%)
SMAQ EAQ	Good	1/11 (9.1%)	64/143 (44.8%)	90.9% (62.7–99.5)	44.8% (36.7–53.0)	11.2% (9.1–13.8)	98.5% (90.7–99.8)
Poor	10/11 (90.9%)	79/143 (55.2%)

Notes: Sensitivity or the true positive rate measures the proportion of actual positives that are correctly identified as such. Specificity or the true negative rate measures the proportion of actual negatives that are correctly identified as such. A positive predictive value is the number of true positives divided by the sum of true and false positives, a value representing the proportion of subjects with a positive test result who actually have the target condition. The negative predictive value is a numerical value for the proportion of individuals with a negative test result who are free of the target condition—i.e., the probability that a person who is a test negative is a true negative.

Abbreviations: VL, viral load; SA NDoH, South African National Department of Health; SMAQ, Simplified Medication Adherence Questionnaire; EAQ, electronic adherence questionnaire; PAQ, paper adherence questionnaire; CI, confidence interval.

Therapeutic Drug Monitoring (Early Response)

A sixth of participants had a sub-optimal drug concentration (EFV ≤1.00 µg/mL) at enrollment (41/264; 15.5%). When disaggregated by mode of delivery, this was similarly reported among participants completing the electronic- and paper-adherence questionnaire (21/143; 14.7% and 20/121; 16.5%, respectively) (see Table 2).

Similar to viral load response, the sensitivity of the NDoH adherence tool in detecting poor adherence confirmed by TDM (EFV ≤1.00 µg/mL) at study enrollment was 43.9% (95% CI: 28.5–60.3). However, when stratified by mode of delivery, it was higher among participants completing the paper-adherence questionnaire (50.0% (95% CI: 27.2–72.8)) than among those who completed the electronic version (38.1% (95% CI: 18.1–61.6)). The sensitivity of the SMAQ was 65.9% (95% CI: 49.4–79.9). As with the NDoH, this was higher among participants completing the paper-adherence questionnaire (75.0% (95% CI: 50.9–91.3)) when compared to those randomized to complete the electronic-adherence questionnaire (57.1% (95% CI: 34.0–78.2)) (see Table 4). When defining poor adherence through TDM, the PPV of the adherence tools were higher than that of viral load, with the highest value observed for the NDoH tool on a paper-adherence questionnaire (28.6%). Similar to viral load monitoring, NPVs were high when defining poor adherence through TDM, ranging from 85.3%–90.0% (see Table 4).

Table 4.

Sensitivity and Specificity of Questionnaire Type in Detecting Poor Adherence by Therapeutic Drug Monitoring Response at Study Enrollment (N=264)

Questionnaire Type	Adherence	EFV ≤1.00 µg/mL	EFV >1.00 µg/mL	Sensitivity (95% CI)	Specificity (95% CI)	Positive Predictive Value (95% CI)	Negative Predictive Value (95% CI)
SA NDoH overall (EAQ+PAQ)	Good	23/41 (56.1%)	152/223 (68.2%)	43.9% (28.5–60.3)	68.2% (61.6–74.2)	20.2% (14.6–27.4)	86.9% (83.3–89.79)
Poor	18/41 (43.9%)	71/223 (31.8%)
SA NDoH PAQ	Good	10/20 (50.0%)	76/101 (75.3%)	50.0% (27.2–72.8)	75.3% (65.7–83.9)	28.6% (18.7–41.1)	88.4% (82.9–92.3)
Poor	10/20 (50.0%)	25/101 (24.8%)
SA NDoH EAQ	Good	13/21 (61.9%)	76/122 (62.3%)	38.1% (18.1–61.6)	62.3% (53.1–70.9)	14.8% (8.8–23.9)	85.4% (80.3–89.4)
Poor	8/21 (38.1%)	46/122 (37.7%)
SMAQ overall (EAQ+PAQ)	Good	14/41 (34.2%)	97/223 (43.5%)	65.9% (49.4–79.9)	43.5% (36.9–50.3)	17.7% (14.3–21.6)	87.4% (81.5–91.6)
Poor	27/41 (65.9%)	126/223 (56.5%)
SMAQ PAQ	Good	5/20 (25.0%)	45/101 (44.6%)	75.0% (50.9–91.3)	44.6% (34.7–54.8)	21.1% (16.5–26.7)	90.0% (80.3–95.2)
Poor	15/20 (75.0%)	56/101 (55.5%)
SMAQ EAQ	Good	9/21 (42.9%)	52/122 (42.6%)	57.1% (34.0–78.2)	42.6% (33.7–51.9)	14.6% (10.3–20.4)	85.3 (77.2–90.8)
Poor	12/21 (57.1%)	70/122 (57.4%)

Abbreviations: EFV, efavirenz; SA NDoH, South African National Department of Health; SMAQ, Simplified Medication Adherence Questionnaire; EAQ, electronic adherence questionnaire; PAQ, paper adherence questionnaire; CI, confidence interval.

Measures of Agreement – Viral Load Response by Therapeutic Drug Monitoring

For the majority of participants (80.3%), there was an agreement between the viral load measurement and TDM result. A total of 4/264 (1.5%) participants reported a detectable viral load (≥1000 copies/mL) and sub-optimal drug concentrations measured by EFV ≤1.0 µg/mL. In contrast, majority of participants (208/264; 78.8%) had both an undetectable viral load (<1000 copies/mL) as well as drug concentration levels above the minimum threshold (EFV >1.0 µg/mL). Under one-fifth of participants reported an undetectable viral load and sub-optimal drug concentrations (37/264; 14.0%) or a detectable viral load but sufficient drug concentrations (15/264; 5.7%) (see Table 5).

Table 5.

Viral Load Response by Therapeutic Drug Monitoring (N=264) (N, Col %)

		Viral Load (≥1000 copies/mL)		Total
No	Yes
TDM (EFV ≤1.00 µg/mL)	No	208 (84.9%)	15 (79.0%)	223 (84.5%)
Yes	37 (15.1%)	4 (21.1%)	41 (15.5%)
	Total	245 (100.0%)	19 (100.0%)	264 (100.0%)

Abbreviations: TDM, therapeutic drug monitoring; EFV, efavirenz.

Feasibility of Mode of Delivery

Majority of participants strongly agreed/agreed that they would prefer a self-administered questionnaire over a counsellor/social worker administered one (67.5% completing the electronic- vs 71.0% completing the paper-adherence questionnaire). A total of 125 (81.2%) of those completing the electronic- and 64 (51.6%) of paper-adherence questionnaire participants strongly agreed/agreed that they would prefer an electronic questionnaire on a tablet/smartphone instead of a paper form. Additionally, 139 (90.3%) of electronic- and 55 (71.4%) of paper-adherence questionnaire participants were comfortable using a smartphone/tablet. With regard to confidentially, less than half of electronic-adherence questionnaire participants (67; 43.5%) and 63 (50.8%) of paper-adherence questionnaire participants had concerns surrounding the confidentially of their information on a smartphone/tablet (see Table 6).

Table 6.

Feasibility of Mode of Adherence Measurement by Patients Completing Either a Self-Administered Electronic or Interviewer Administered Paper-Adherence Questionnaire at Study Enrollment (N=278)

	Electronic Adherence Questionnaire (EAQ) (n=154)			Paper Adherence Questionnaire (PAQ) (n=124)
	July 2015–September 2017			July 2015–September 2017
	Strongly Agree/Agree	Neutral	Strongly Disagree/Disagree	Strongly Agree/Agree	Neutral	Strongly Disagree/Disagree
N (%)	N (%)	N (%)	N (%)	N (%)	N (%)
Likert Scale Question
You would prefer a self-administered questionnaire over a counsellor or social worker administered one. In other words, fill the answers in yourself without a counsellor or social worker present	104 (67.5%)	27 (17.5%)	23 (14.9%)	88 (71.0%)	10 (8.1%)	26 (21.0%)
You would prefer an electronic questionnaire on a tablet or smartphone instead of a paper form	125 (81.2%)	18 (11.7%)	11 (7.1%)	64 (51.6%)	17 (13.7%)	43 (34.7%)
You are comfortable with using a smartphone or tablet*	139 (90.3%)	10 (6.5%)	5 (3.3%)	55 (71.4%)	8 (10.4%)	14 (18.1%)
I am concerned that my information will not be confidential if I answer questions on a tablet or smartphone	67 (43.5%)	23 (14.9%)	64 (41.6%)	63 (50.8%)	12 (9.7%)	49 (39.5%)
Overall, you liked answering the questions yourself on a tablet or smartphone†	136 (88.9%)	11 (7.2%)	6 (3.9%)	N/A
The tablet was easy to use and the questionnaire easy to complete†	143 (93.5%)	7 (4.6%)	3 (2.0%)	N/A

Note: *PAQ (n=77); ^†EAQ (n=153).

When asked if they would prefer a self-administered adherence questionnaire (in the absence of a counsellor/social worker), participants completing the electronic-adherence questionnaire said, “Yes because it makes me … as honest as possible” (PID 15229, Female, 24y) and “On my own because whenever you speak to a counsellor or nurse you … feel as if you are being judged for not adhering” (PID 15045, Female, 29y). In contrast, some patients also raised concerns of confidentially and misunderstanding saying, “I don’t feel my information is safe on the iPad [tablet], I feel it’s good when I speak to someone” (PID 15214, Male, 33y) and, “No because I needed clarity with some of the questions” (PID 15120, Male, 35y). Patients completing the paper-adherence questionnaire said, “Either way, but if there’s something I don’t understand at least I’ll have someone to explain it to me when being asked by a counsellor” (PID 15048, Female, 30y), “ … I like to interact to a social worker or counsellors who inform me with positive views” (PID 15095, Female, 25y) and, “As long as the person is friendly it’s okay unless the person is not friendly then I would say I prefer to answer on my own” (PID 15004, Female, 29y).

When patients who completed the electronic-adherence questionnaire were asked about their likes of such an adherence measurement tool, the relative ease of use, convenience of typing, flexibility in changing responses and quickness of the tool were cited as primary reasons. In contrast, dislikes included the absence of someone to help clarify misunderstanding and the electronic tablet running out of power while being used (see Table 7).

Table 7.

Qualitative Responses Assessing the Feasibility of a Self-Administered Electronic-Adherence Questionnaire vs Interviewer Administered Paper-Adherence Questionnaire

Question	Aggregated Responses by Questionnaire Arm	Example/Quotation	Main Observed Difference Between Questionnaire Arms
What do you like about the adherence counselling that you receive from counsellors or social workers at this clinic?	Electronic-adherence questionnaire: Patients generally liked the adherence counselling due to supportive, comforting, non-judgmental and well-informed counsellors/social workers.	“It was good because it helps me to understand the reason why must I take the medication and I was able to ask questions about my status and ARVs” (PID 15118, Female, 31y). “They are supportive and they make you feel comfortable when talking to them” (PID 15038, Male, 21y). “Sometimes it feels good to talk to someone who will not judge you, or maybe to your face” (PID 15065, Male, 34y). “Nothing to be honest they don’t know how to talk to people and most of the time they don’t make sure that people know and understand what to do they just don’t have patience for patients” (PID 15132, Female, 33y).	Patients completing the electronic-adherence questionnaire mentioned at least one negative attribute while responses from participants completing the paper-adherence questionnaire were all positive.
Paper-adherence questionnaire: Similar to electronic participants, participants completing a paper-adherence questionnaire generally liked the supportive, well-informed, and considerate counsellors/social workers.	“I liked that they mentioned that I need to eat healthy, use a condom and take my medication on time” (PID 15260, Male, 28y). “They were supportive and the information was great” (PID 15145, Female, 28y). “I like the fact that they were involving everyone as group instead of doing it individuals. Making everyone comfortable” (PID 15248, Female, 24y).
What do you dislike about the adherence counselling that you receive from counsellors or social workers at this clinic?	Electronic-adherence questionnaire: Disconnectedness from counsellors/social workers, long queues, brief counselling sessions and attitudes of counsellors were commonly reported dislikes.	“Sometimes feel like they are just doing their work, we are not connecting” (PID 15065, Male, 34y). “The counselling is very brief and short” (PID 15146, Male, 26y). “The attitude that they have … they have no respect for patients” (PID 15132, Female, 33y). “Waiting for long queue to see doctor” (PID 15008, Female, 23y).	Patients completing the electronic-adherence were primarily concerned with long waiting times and brief counselling sessions while those completing the paper-adherence questionnaire were concern with Tuberculosis exposure during their counselling sessions.
Paper-adherence questionnaire: Integrated counselling sessions with Tuberculosis (TB) patients, the rotation of counsellors at different visits and attitudes of counsellors were commonly reported dislikes.	“I didn’t like them to mix us with TB people because it’s not healthy for us” (PID 15078, Male, 21y). “They were judgemental and they shout at you” (PID 15021, Male, 19y). “Coming to the clinic and find different people all the time who does counselling” (PID 15207, Female, 35y).
Are there ways we could make adherence counselling better? Be as specific as possible	Electronic-adherence questionnaire: A detailed description of potential side effects of new medication, the elimination of language barriers, and friendlier attitudes of counsellors were noted as factors that could improve counselling.	“Yes, before introducing newly improved medication at least give patients information about the side effects of the drugs, the do’s and don’ts … because some people can’t read and understand the leaflets” (PID 15231, Female, 35y). “Yes you could get the counsellor to accommodate everyone in the class by speaking the language that everyone will understand” (PID 15036, Female, 35y). “Be polite all the time that way it makes people more comfortable and open to them” (PID 15284, Female, 24 y). “Have counsellors that come to universities to come and explain better to other young people who are afraid to get tested” (PID 15176, Female, 25y).	The role of mobile counsellors and remote counselling sessions taking place at universities were identified as possibly improvements to counselling among patients completing the electronic-adherence questionnaire. Patients completing the paper-adherence questionnaire stressed the importance of private counselling sessions.
Paper-adherence questionnaire: The non-judgemental attitudes of counsellors and more individualized and private counselling sessions were identified as factors that could improve adherence counselling.	“They must address us in a nice way” (PID 15021, Male, 19y). “I prefer counselling one on one not in a group setting” (PID 15190, Male, 22y). “Not to judge anyone. They must always be friendly and smile to [the] patient” (PID 15163, Female, 32y).
Would/Did you prefer to complete a questionnaire that asks you if you have been taking your medication, on your own and in the absence of a counsellor or social worker? Please answer Yes or No and then explain why you selected Yes or No.	Electronic-adherence questionnaire: Patients who preferred a self-administered questionnaire did so as they could be more honest in their responses. Patients who preferred a counsellor administered adherence questionnaire chose this method as it allowed for interaction with and clarification from counsellors.	“I don’t feel my information is safe on the iPad, I feel it’s good when I speak to someone” (PID 15214, Male, 33y). “Yes because it makes me … as honest as possible” (PID 15229, Female, 24y). “On my own because whenever you speak to a counsellor or nurse you … feel as if you are being judged for not adhering” (PID 15045, Female, 29y). “No because I needed clarity with some of the questions” (PID 15120, Male, 35y).	The ability to provide more honest responses on a self-administered electronic questionnaire was noted among those who were randomized to complete such a questionnaire. Patients completing the paper-adherence questionnaire, for the most part enjoyed their interaction with counsellors and the ability to ask and clarify questions.
Paper-adherence questionnaire Those who completed the counsellor administered paper-adherence questionnaire preferred this mode measurement as it allowed them to interact with health care workers clarify misunderstanding.	“Either way, but if there’s something I don’t understand at least I’ll have someone to explain it to me when being asked by a counsellor” (PID 15048, Female, 30y). “As long as the person is friendly it’s okay unless the person is not friendly then I would say I prefer to answer on my own” (PID 15004, Female, 29y). “No, I like to interact to a social worker or counsellors who inform me with positive views” (PID 15095, Female, 25y).
What did you dislike about using the tablet to complete the adherence questionnaire? Be as specific as possible*	Electronic-adherence questionnaire: The inability to clarify misunderstandings/ambiguous questions and technical challenges were commonly reported dislikes of the electronic-adherence questionnaire.	“There are things I did not understand clearly” (PID 15272, Female, 35y). “The battery went flat while busy” (PID 15147, Female, 35y). “Sometimes I would like to ask some questions” (PID 15024, Female, 29y).	N/A
What did you like about using the tablet to complete the adherence questionnaire? Be as specific as possible*	Electronic-adherence questionnaire: The ability to answer questions independently, the convenience of typing over writing and the flexibility when changing responses were commonly reported likes of the electronic-adherence questionnaire.	“I am able to answer the question by myself and it’s quick. Also with the tablet I know information won’t get lost unlike if it was on a paper” (PID 15045, Female, 29y). “More convenient than writing” (PID 15168, Male, 23y). “If you didn’t get the question and you answered it wrong you can always go back n correct it” (PID 15036, Female, 23y). “It is quick and I am more comfortable in expression“ (PID 15262, Male, 26y).	N/A
Did you find the tablet easy to use? Did you have any trouble using it? If yes, what trouble did you have?*	Electronic-adherence questionnaire: Some participants noted that the electronic tablet was relatively easy to use, while others had difficulty in understanding certain questions in the absence of a counsellor.	“Yes it was easy to use and I did not have any trouble using it” (PID 15055, Female, 31y). “Yes some questions are difficult to understand” (PID 15005, Female, 28y).	N/A
Is there anything else you would like to tell us?	Electronic-adherence questionnaire: The importance and ease of use of technological applications were stressed by participants completing the electronic-adherence questionnaire.	“This is a great survey and it’s making patients’ lives very easy since we use technological apps, this survey must continue every time for everyone including the elderly” (PID 15014, Female, 23).	Responses to this question varied between questionnaire arm. Those completing the electronic-adherence questionnaire enjoyed the ease of use of such technology, while patients completing the paper-based version suggested simpler visit schedules to minimize travel costs.
Paper-adherence questionnaire: The scheduling of medication pick-up and clinical visits was emphasized by those completing the paper-adherence questionnaire. To minimize travel costs and time, these visits should be scheduled on the same day.	“Wish that on the same day that people come for adherence they must also receive medication to minimize traveling cost and time to come to the clinic” (PID 15207, Female, 35y).

Note: *Only completed among patients who were randomized to complete an electronic-adherence questionnaire.

Discussion

Due to the unique challenges faced by adolescents and young adults in resource-limited settings in achieving optimal ART adherence and subsequently favorable HIV-treatment outcomes, we set out to explore the accuracy and feasibility of an electronic ART adherence questionnaire in detecting poor adherence. We compare the diagnostic accuracy of a self-administered electronic-adherence questionnaire vs standard of care patient-interviewer administered paper-adherence questionnaire to identify poor adherence defined 1) as a detectable viral load (≥1000 copies/mL) and 2) as ART blood drug levels lower than the minimum effective concentration measured by TDM (EFV ≤1.00 µg/mL). To our knowledge, no other study in South Africa or Africa has assessed the diagnostic accuracy of different adherence delivery modes (electronic vs paper) across two standard adherence tools (NDoH and SMAQ) to identify poor adherence, defined by either viral load monitoring or TDM, among adolescents and young adults.

Although in widespread use in RLS, particularly due to its feasibility, self-reports are prone to distortion when assessing socially undesirable topics/outcomes.^19,38 This method of assessment has commonly led to the under-reporting of behaviours in contrast to social norms and regulations and the over-reporting of prescribed activities.¹⁹ To increase the validity of self-reports of sensitive issues, computerized/electronic modes of delivery have been suggested to increase respondents’ anonymity, consequently resulting in more truthful responses.³⁹ Moreover, elimination of the patient-interviewer from respective assessments has been repeatedly linked to increases in disclosure of sensitive behaviours and reductions in social desirability responses.⁴⁰

In this study, the electronic-adherence questionnaire (completed independently, with minimal contact with patient interviewers) (SMAQ Se: 90.9%; 95% CI: 62.7–99.5; NDoH Se: 63.6%; 95% CI: 33.6–87.2) had a higher sensitivity in detecting poor adherence by a detectable viral load (≥1000 copies/mL) when compared to the interviewer-administered paper-adherence questionnaire (SMAQ Se: 66.7%; 95% CI: 33.2–90.7; NDoH Se: 33.3%; 95% CI: 9.3–66.8) (although estimates prove imprecise). Participants completing the electronic-adherence questionnaire may then report non-prescribed/undesirable behaviours such as poor adherence more truthfully whilst independently completing an adherence questionnaire, than those being formally interviewed by a patient-interviewer. Similarly, when asked about their preferences of electronic- vs paper-adherence questionnaires, some participants mentioned that an electronic-adherence questionnaire allows them to be as honest as possible in their responses and not have to face possible judgment from counsellors.

However, the use of viral load monitoring may not provide an accurate account of adherence due to the potential lag between periods of non-adherence and the consequential increase in viral load. In a study examining the timing of virologic rebound after treatment cessation, 54% of participants on a non-nucleoside reverse transcriptase inhibitor (NNRTI)-based regimen remained virally suppressed four weeks after treatment interruption (<1000 copies/mL).⁴¹ As participants in this study were on a first-line NNRTI-based ART regimen at study enrollment, the effects of poor adherence to ART, let alone treatment cessation may not necessarily translate into detectable viral load in the weeks following the interruption/period of non-adherence. Viral load then provides a late detection of proximal adherence (importantly, only among those without confirmed drug resistance). A more robust measure of adherence may take the form of TDM which measures blood drug concentration levels. In this study, TDM detected a greater level of poor adherence when compared to viral load monitoring (15.5% vs 7.2%, respectively). TDM provides a more real-time estimate of actual drug concentration and not a measure of the effects (viral load) of sustained sub-optimal drug concentrations/drug resistance. Furthermore, TDM ensures a more accurate account of adherence due to the inadvertent detection of potential drug resistance (i.e., patients who report optimal adherence and whose TDM results detect optimal blood drug concentrations, but whose viral load is in the detectable range).

When assessing the sensitivity of modes of delivery by TDM (EFV ≤1.00 µg/mL), the paper-adherence questionnaire (SMAQ Se: 75.0%; 95% CI: 50.9–91.3; NDoH Se: 50.0%; 95% CI: 27.2–72.8) had a higher sensitivity in detecting poor adherence when compared to the self-administered electronic-adherence questionnaire (SMAQ Se: 57.1%; 95% CI: 34.0–78.2; NDoH Se: 38.1%; 95% CI: 18.1–61.6). Similarly, the positive predictive value of mode of delivery (electronic vs paper) defined by TDM was more than double that of poor adherence defined by detectable viral load (range 14.6%–28.6% vs 8.1%–11.7%, respectively). Unlike the self-administered electronic-adherence questionnaire, patients completing a paper-adherence questionnaire were formally interviewed. Contrary to reports on the effects of social desirability bias/response bias, the presence of the interviewer may then elicit more accurate reporting of adherence as patients may feel more accountable when engaging with health care workers. Additionally, the presence of the interviewer may also allow for the clarification of adherence questions compared to completing an adherence assessment independently. When asked if they would like to complete an adherence assessment independently (in the absence of a counsellor/social worker), patients completing the paper-adherence questionnaire said they prefer the interaction with health care workers as it allows them to clarify misunderstandings. Similarly, among participants completing the electronic-adherence questionnaire, some of the reported dislikes of the questionnaire included; the misunderstanding of questions and the inability to clarify such misunderstandings with someone.

When stratified by adherence tool, sensitivity of detecting poor adherence by detectable viral load was higher in the SMAQ (electronic-adherence questionnaire Se: 90.9%; 95% CI: 62.7–99.5 and paper-adherence questionnaire Se: 66.7%; 95% CI: 33.2–90.7) when compared to the NDoH questionnaire (electronic-adherence questionnaire Se: 63.6%; 95%: 33.6–87.2 and paper-adherence questionnaire Se: 33.3%; 95%: 9.3–66.8), irrespective of mode of delivery (electronic vs paper). A similar trend was observed when poor adherence was defined by TDM; SMAQ (electronic-adherence questionnaire Se: 57.1%; 95% CI: 34.0–78.2 and paper-adherence questionnaire Se: 75.0%; 95% CI: 50.9–91.3) when compared to the NDoH questionnaire (electronic-adherence questionnaire Se: 38.1%; 95%: 18.1–61.6 and paper-adherence questionnaire Se: 50.0%; 95%: 27.2–72.8), irrespective of mode of delivery (electronic vs paper).

The generally high sensitivity of the SMAQ in detecting poor adherence to ART confirmed by virologic outcome has been described previously (>72%).³⁴ Various attributes of the SMAQ could lead to the higher sensitivity, much of which is beyond the scope of this study. However, at a descriptive level, the SMAQ consists of two more items than the NDoH tool (10 vs 8 items, respectively). Furthermore, the SMAQ consists of multi-time point recall adherence questions spanning the three months preceding the assessment (with measurement at various time-points such as the past weekend, week and month), while the NDoH only assesses adherence up to a maximum of four days prior to the assessment. The increased time-span of adherence measurement in the SMAQ may then address the effects of “white-coat adherence”, a well-documented phenomenon in which patients improve their adherence prior to a scheduled appointment with a health care worker.⁴²

Limitations

Although viral load monitoring is considered the standard of care in determining first-line ART treatment failure, it may only provide a proximal delayed measured of ART adherence. Moreover, studies from resource-limited settings indicate that between 18% (74/407) to 52% (33/64) of patients experiencing virologic failure (≥1000 copies/mL) also had one or more NNRTI drug-resistant mutation.^43,44 While resistance testing was beyond the scope of this study and not routinely administered in first-line therapy,²³ the effects of drug-resistance may then be a key determinant of virologic failure among patients who are in fact truly adherent. However, while the cyclic relationship may be difficult to assess here, sub-optimal adherence has often been identified as a risk factor in developing drug-resistance.⁴⁵

The limitations of TDM have been well described in literature and the concentration of the drug can be influenced by other factors (e.g. mal-absorption, concomitant medication, low bioavailability of the drug, incorrect dosage, etc.)⁴⁶ For TDM, blood samples should be obtained at the end of the dosing interval as close to Cmin as possible to enable comparison of a measured concentration with a measure of central tendency (mean or median) for Cmin published in the individual product monographs (using pharmacokinetic data from HIV-infected volunteers). However, we could only obtain samples during the clinic visit (as participants could not return later in the afternoon or after clinic hours), which may have influenced the accuracy of the TDM results and underestimated the number of participants with poor adherence. Therefore, while estimates of poor adherence defined by TDM proved two times higher than that of viral load, this measure is still conservative and a potentially higher proportion of participants in this study could have blood drug concentrations lower than the minimum effective concentration. It should also be noted that although the sensitivities for both the NDoH and SMAQ paper-adherence questionnaire were higher than the electronic-adherence questionnaire, both sensitivities were lower than that observed when compared to the standard of care (i.e. viral load), possibly as a result of misclassification when using TDM method to define poor adherence.

While participants completing the electronic-adherence questionnaire did so with minimal patient–interviewer interaction and for the most part did so independently, those randomized to the complete the paper-adherence questionnaire were formally interviewed by study interviewers. A third arm, self-administered paper-adherence questionnaire was not included in this study, and therefore the effect of response bias could not be assessed here.

Furthermore, results from this study are from a single site which may be dependent on individual staff attitudes, training and willingness to engage and interact with AYAs. Results from this study may then differ from other HIV treatment clinics in Johannesburg and more broadly, Gauteng province.

Although the ability to speak and understand English was considered an eligible criterion, we did not assess the general level of language proficiency among participants. Therefore, participants could have misunderstood particular questions leading to imprecise reporting of their adherence. However, with the higher level of tertiary education recorded among participants randomized to complete the electronic-adherence questionnaire (24.0% vs 12.1%), the effects of lower education among patients completing the paper-adherence questionnaire could have been mediated by the presence of the interviewer.

A total of six participants were excluded from the analysis after study enrollment (unsuitable viral load sample). The suitability of viral load samples only become known post viral load draw and therefore could only be accounted for post-enrollment. The limitations of routinely collected data have been well documented.⁴⁷ Those excluded, either because of a previous elevated load or a missing viral load result were equally distributed between the electronic- and paper-adherence questionnaire arms so we consider the risk of bias to be minimal.

Conclusion

When using more accurate real-time measures of poor adherence such as TDM (EFV ≤1.00 µg/mL), we observe a higher sensitivity of an interviewer-administered paper-adherence questionnaire than an identical set of self-administered adherence questions on an electronic tablet. The electronic- and paper-adherence questionnaire comprised questions from two standard adherence tools (South African National Department of Health adherence questionnaire and the Simplified Medication Adherence Questionnaire, with the latter achieving a higher sensitivity regardless of mode of delivery (electronic vs paper) or definition of poor adherence (detectable viral load vs EFV ≤1.00 µg/mL). Interviewer-administered paper-adherence questionnaire may elicit more accurate responses from participants through a sense of increased accountability when engaging with health care workers.

Funding Statement

This study has been made possible by the generous support of the American People and the President’s Emergency Plan for AIDS Relief (PEPFAR) through USAID under the terms of Cooperative Agreements AID-674-A-12-00029 and 72067419CA00004 to the Health Economics and Epidemiology Research Office and 674-A-12-00020 to Right to Care. The contents are the responsibility of the authors and do not necessarily reflect the views of PEPFAR, USAID or the United States Government. The funders had no role in the study design, collection, analysis and interpretation of the data, in manuscript preparation or the decision to publish.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Ethical approval for the use of data was provided by the Human Research Ethics Committee (Medical) of the University of the Witwatersrand (M141186). All participants provided written informed consent to participate in the study.

Author Contributions

DE, KH and MPF conceptualised the study. KH and NJ facilitated study enrollment while RG, JT and SM facilitated the clinical management of patients. KH and DE analysed the data and drafted the manuscript. NJ, RG, JT, SM, LCL and MPF critically reviewed the analysis, interpreted study data and edited the manuscript. All authors read and approved the final manuscript. All authors are in agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Disclosure

Lawrence C Long report grants from USAID, during the conduct of the study; Julia Turner reports personal fees from Jansen, Viiv, Virology Education, Abbvie, and Mylan, outside the submitted work. The authors report no other conflicts of interest in this work.