Impact of definitions of loss to follow-up (LTFU) in antiretroviral therapy program evaluation: variation in the definition can have an appreciable impact on estimated proportions of LTFU

Abstract

Objective

To examine the impact of different definitions of loss to follow-up (LTFU) on estimates of program outcomes in cohort studies of patients on antiretroviral therapy (ART).

Study Design and Setting

We examined the impact of different definitions of LTFU using data from the International Epidemiological Databases to Evaluate AIDS—Southern Africa. The reference approach, Definition A, was compared with five alternative scenarios that differed in eligibility for analysis and the date assigned to the LTFU outcome. Kaplan–Meier estimates of LTFU were calculated up to 2 years after starting ART.

Results

Estimated cumulative LTFU were 14% and 22% at 12 and 24 months, respectively, using the reference approach. Differences in the proportion LTFU were reported in the alternative scenarios with 12-month estimates of LTFU varying by up to 39% compared with Definition A. Differences were largest when the date assigned to the LTFU outcome was 6 months after the date of last contact and when the site-specific definition of LTFU was used.

Conclusion

Variation in the definitions of LTFU within cohort analyses can have an appreciable impact on estimated proportions of LTFU over 2 years of follow-up. Use of a standardized definition of LTFU is needed to accurately measure program effectiveness and comparability between programs.

1. Introduction

In 2011, 30 years since the first cases of AIDS were reported, 34 million people worldwide were living with HIV [1]. More than two-thirds of the global cases are located in sub-Saharan Africa. Nearly six million people in the region were receiving antiretroviral therapy (ART) in 2011, up from only 100,000 people in 2003 [1]. Across sub-Saharan Africa, the expansion of ART services represents one of the largest pharmacologic interventions to promote population health to date, and there is considerable interest in monitoring the impact of these programs [2]. Given the high mortality rate of untreated HIV infection, death and disease progression have been the major clinical outcomes of interest in ART programs. However more recently, loss to follow-up (LTFU) has emerged as a key indicator of ART program effectiveness [3], and there is a growing emphasis on monitoring programs to improve long-term retention in care [4].

Conventionally within most cohort analyses, LTFU is considered an uninformative censoring event [5,6]; and in practice, relatively little attention may be paid to the definition of LTFU compared with the clinical outcomes of interest (such as mortality in the case of ART programs). However when LTFU becomes an outcome of interest, greater attention to the definition of this phenomenon may be warranted. Within ART programs, consideration of LTFU is of particular importance and is generally higher in settings where mortality ascertainment may be weak [7].

Understanding the impact of varying definitions of LTFU on program outcomes is a central consideration for other chronic disease programs that seek to retain patients in care over the long term. However, there has been relatively little attention to how LTFU should be defined in cohort analyses in ART and other chronic disease services. Within the evaluation of ART programs, the treatment of LTFU in analysis varies widely. The lack of a standard definition hampers systematic reviews of pooled data [8–10]. Previous work has focused on the optimal duration of loss to minimize LTFU misclassification [11,12], but there is little consensus on other aspects of defining LTFU within analyses.

Definitions of LTFU used in the analysis of ART programs vary in their application of eligibility for inclusion in analysis, the date assigned to the LTFU outcome, and whether or not to use the outcome LTFU as defined by the site. In light of this heterogeneity, it is important to investigate whether and how definitions of LTFU impact on observed program outcomes. It is plausible that varying analytic definitions give rise to spurious observed differences in estimates of LTFU as a program outcome. We used data from the International Epidemiological Databases to Evaluate AIDS—Southern Africa (IeDEA-SA) collaboration to explore the impact of varying LTFU definitions on program estimates of LTFU.

2. Methods

IeDEA is an international collaboration comprising seven data centers that pool large HIV data sets [13–15]. The source data set for this analysis included all adults who initiated therapy between 2002 and 2007 at eight public sector ART programs in South Africa; details of the sites and sample have been reported previously [16]. The IeDEA-SA collaboration was approved by the ethics committees of the University of Cape Town, South Africa, and University of Bern, Switzerland and the ethics committees of each participating site.

Within these data, six different scenarios with varying definitions of LTFU were tested to determine how variation in the definition of LTFU may influence the program retention outcomes over the first 24 months after ART initiation. Five definitions generated LTFU under varying assumptions, and the sixth used the site-defined variable of LTFU. Consistent within the five generated definitions were that individuals were defined as LTFU if they had no visit or contact in the previous 6 months. A 6-month period was used to accommodate the largest interval between visits within contributing cohorts. In addition, this time period has been shown to have the least misclassification when defining LTFU [11,12]. Definitions of LTFU differed in: the timing of analysis and database closure, the approach to those who initiated treatment and never returned, and the date assigned to the LTFU outcome (Fig. 1). Variations in these key factors gave rise to five generated definitions of LTFU analyzed here.

Fig. 1.

Decision in defining LTFU in an antiretroviral therapy program cohort analysis. *Bolded option denotes decision illustrated in Definition A. LTFU, loss to follow-up.

2.1. Reference approach–Definition A

In Definition A, the analysis was closed 6 months before database closure. This ensured both sufficient follow-up time among those included as well as a window of observation to observe retention in care. For patients who initiated ART and never returned, 1 day of follow-up was added so that they were included in the time-to-event analysis. The last contact date was assigned to the outcome LTFU [17]. Definition A is the most conservative and likely to result in the highest estimates of LTFU. It is used as the reference to compare with all other scenarios (see bolded options in Fig. 1).

2.2. Variations on the definition of LTFU

Four variations to generating a definition of LTFU were applied. In Variation A1, the analysis and database closure dates were the same and thus all those who initiated treatment in the 6 months preceding the database closure were included in the analysis [18,19]. In Variation A2, participants who did not return after ART initiation were excluded from analysis [20]. A third scenario, Variation A3, assigned the date of LTFU as 6 months after the last contact date, allowing patients defined as LTFU to contribute additional person-time to the analysis [21]. Variation A4 combined Variations A1–A3: the analysis and database closure dates were the same, those who initiated ART and never returned were excluded, and the date assigned to the outcome LTFU was 6 months after the last contact date. The differences between these generated definitions are summarized in Table 1. A sixth definition of LTFU was based on the determination of LTFU by each site. In all of these described previously, those who were transferred out were censored at the date of transfer; deaths were censored on the date of death.

Table 1.

Analytic approaches to defining ART program loss to follow-up

Approach	Inclusion of patients who started ART but never returned to clinic	Analysis closure 6 months before database closure	Date assigned to loss to follow-up
Definition A [17]	Yesa	Yes	Last visit
Variation A1 [18,19]	Yesa	Nob	Last visit
Variation A2 [20]	No	Yes	Last visit
Variation A3 [21]	Yesa	Yes	Last visit plus 6 mo
Variation A4	No	Nob	Last visit plus 6 mo

Abbreviation: ART, antiretroviral therapy.

^aPatients were included by adding 1 day of follow-up.

^bAll patients included, independent of window of observation and duration of follow-up.

2.3. Statistical analysis

For each of the scenarios, patient characteristics at baseline (age, sex, Cluster of Differentiation 4 [CD4] count, and year of enrollment) were described with appropriate summary statistics. Kaplan–Meier estimates of LTFU under the six different definitions were calculated over 24 months and reported at 6-month intervals. Differences in the proportions LTFU were estimated using Definition A as the reference scenario throughout. Data were analyzed using STATA 11.0 (STATA Corporation, College Station, TX).

3. Results

The scenarios included between 43,173 and 45,849 adults with slight variations in baseline characteristics based on the eligibility for the different analyses. Definition A included 44,177 adults with a median age of 35 years and more than 60% of participants initiating therapy in the last 2 years of enrollment (2006 and 2007; Table 2). In Variation A1 with no window between the analysis and database closure, an additional 1,655 participants were included. Patient characteristics (gender, age, baseline CD4, and year of enrollment) of those LTFU were consistent across all scenarios (Table 3).

Table 2.

Patient characteristics at ART initiation in Definition A

Characteristics	N(%)
Adults (≥16yr), n	44,177
Gender
Female	29,904 (67.7)
Age
Age (yr), median (IQR)	35.0 (29.9–41.6)
Age categories (yr)
16–24	2,306 (5.2)
25–34	17,654 (40.0)
35–44	16,177(36.6)
45+	8,040 (18.2)
Baseline CD4 (cells/uL), categorical
<50	9,947 (27.2)
50–199	22,703 (62.1)
≥200	3,899 (10.7)
Year of initiation
2002 and 2003	1,173 (2.7)
2004	5,262 (11.9)
2005	9,909 (22.4)
2006	13,105 (29.7)
2007	14,728 (33.3)

Abbreviations: ART, antiretroviral therapy; IQR, interquartile range; CD4, Cluster of Differentiation 4.

Table 3.

Kaplan–Meier estimates of 12-mo ART program LTFU by covariates reported by LTFU definition

Characteristics	Definition A	Variation A1	Variation A2	Variation A3	Variation A4	Site defined
Gender
Male	17.0	15.8	14.6	13.0	10.6	13.7
Female	14.8	13.8	12.7	10.5	8.5	12.2
Age (yr)
16–24	22.1	20.2	19.0	15.1	12.2	17.3
25–34	15.6	14.4	13.6	11.3	9.3	12.4
35–44	14.6	13.6	12.5	10.9	8.8	12.2
45+	15.4	14.3	13.0	11.1	8.6	13.0
Baseline CD4 (cells/uL)
<50	16.4	15.4	13.2	13.7	11.4	13.7
50–199	13.8	12.8	12.0	9.7	8.0	11.3
≥200	17.7	16.0	15.4	12.4	10.2	12.9
Year of initiation
2002 and 2003	1.1	1.1	1.1	0.8	0.1	1.0
2004	9.0	9.0	7.5	6.4	6.0	7.8
2005	10.8	10.8	9.1	7.8	7.3	8.3
2006	14.0	13.7	11.8	9.6	8.5	9.9
2007	26.7	20.7	23.8	21.4	13.3	21.6

Abbreviations: ART, antiretroviral therapy; LTFU, loss to follow-up; CD4, cluster of differentiation 4.

3.1. Changes in LTFU by definition

Under the reference approach (Definition A), estimated LTFU was 9.3%, 14.4%, 18.8%, and 22.4% at 6-, 12-, 18- and 24-months post-ART initiation, respectively (Table 4). Compared with this definition, all other scenarios underestimated LTFU in the first year on treatment (Fig. 2). When the analysis and database closure dates were the same (Variation A1), the LTFU underestimation increased from 4.3% at 6 months to 12.1% at 24 months compared with Definition A (Table 4). Excluding those who initiate ART and never return (Variation A2) resulted in lower estimates of the LTFU proportion compared with Definition A. At 6 months, the difference was 21.5% decreasing to 7.6% at 24 months.

Table 4.

Kaplan–Meier estimates of cumulative mortality and LTFU reported 6 monthly and relative differences of LTFU^a compared with Definition A

	6 mo			12 mo			18 mo			24 mo
Approach	% Mortality	% LTFU	% Difference	% Mortality	% LTFU	% Difference	% Mortality	% LTFU	% Difference	% Mortality	% LTFU	% Difference
Definition A	4.8	9.3	Ref	6.6	14.4	Ref	7.6	18.8	Ref	8.5	22.4	Ref
Variation A1	4.7	8.9	4.3	6.4	13.5	6.3	7.3	17.0	9.6	8.0	19.7	12.1
Variation A2	4.7	7.3	21.5	6.5	12.5	13.2	7.6	17.0	9.6	8.4	20.7	7.6
Variation A3	4.5	2.5	73.1	6.2	10.7	25.7	7.2	17.2	8.5	8.0	22.8	−1.8
Variation A4	4.5	1.3	86.0	6.0	8.8	38.9	6.9	14.1	25.0	7.6	18.6	17.0
Site defined	4.6	6.8	26.9	6.3	11.9	17.4	7.2	18.5	1.6	7.9	25.6	−14.3

Abbreviation: LTFU, loss to follow-up.

^aDifferences reported are percentage differences compared with Definition A.

Fig. 2.

Antiretroviral therapy program LTFU by LTFU definition 0–24-months follow-up. LTFU, loss to follow-up.

Variation A3 highlighted the effect of assigning the LTFU date 6 months after the date of last contact. The underestimation of LTFU was 73% at 6 months decreasing with time on treatment; and at 24 months, the LTFU proportion was 1.8% higher than Definition A (Table 4). Combining variants A1–A3 into Variation A4 produced the largest deviations from the reference definition with an 86% underestimation of LTFU (1.3% vs. 9.3%) at 6 months decreasing to a 17% underestimation at 24 months (Table 4).

Estimates of LTFU from site-defined LTFU varied appreciably, both in magnitude and direction of LTFU, compared to Definition A. The proportion LTFU at 6, 12, and 18 months was less than in Definition A (26.9%, 17.4%, and 1.6%, respectively) and 14.3% higher at 24 months (Table 4).

4. Discussion

This analysis demonstrates that different definitions of LTFU within an ART cohort analysis have an appreciable impact on estimated LTFU over time. Estimates of LTFU, particularly at 12 months, varied in each scenario when compared with the reference approach Definition A. If the nature of the analytic approach taken to LTFU varies, comparing programs and combining results are not possible.

There is a growing interest in LTFU and program retention in ART programs [4]. Two systematic reviews of ART program retention have combined the results from multiple cohorts despite differences in LTFU definitions [8,9]. There is strong evidence that over time, LTFU is contributing an increasing proportion to those lost to care, and may be a more important indicator of program performance than patient mortality [17]. However, it is difficult to assess trends and differences in LTFU between and within programs as variations in the definitions of LTFU obstruct comparability.

There are many variations as to how LTFU is defined within the literature. This analysis demonstrates empirically the impact of these variations on estimates of LTFU. One approach is to have the analysis and database close at the same time using all of the available data [18,19]. Although it is appealing in its use all available data, this approach, Variation A1, leads to a consistent underestimation of LTFU, as those who had recently initiated treatment did not have sufficient potential to be LTFU.

An alternative approach is to exclude patients who have recently initiated therapy, those without sufficient potential to be LTFU, and keep the analysis and database closure dates the same [22–24]. The common closure of the analysis and database results in an artificial cessation of LTFU in the last 6 calendar months of observation because LTFU is impossible among those with a visit in the final 6 months. Excluding those who initiated treatment in the previous 6 months is either noninformative, or in context of temporal trends of increasing LTFU, contributes to underascertainment of recent LTFU. This variation is not used here, as it is not recommended given that LTFU is a meaningful event often more likely to occur in the final months of observed patient-time when the phenomenon is increasing owing to service expansion.

Another approach uses different dates for database and analysis closure [3,18,25,26]. The analysis is closed the same duration as the LTFU definition earlier than the database closure. As was done in Definition A, restricting the analysis to person-time observed at least 6 months before the database closure gives a window of observation to observe retention in care and ensures sufficient potential observation among those who recently initiated therapy.

Patients who initiate treatment and never return are a group that is often excluded from the time-to-event analyses. Excluding this group, as in Variation A2, potentially underestimates true LTFU and inflates the estimates of program retention. Alternative approaches are to report them as a specific ‘‘no follow-up’’ group [3], or presume a small amount of follow-up time (e.g., 1 day) so that they are included in the analysis (Definition A) [17]. The LTFU in Variation A2 was consistently lower than in Definition A with the relative difference decreasing over the duration of follow-up. Initiation without being confirmed on treatment is debatable, but where data systems are weak, the first visit is often preferentially recorded and those individuals with only the initiation visit recorded may well have been established on treatment.

Approaches to the assignment of a date to the outcome of LTFU vary as well. Most analyses use the last visit or contact date [18,27–32], whereas others choose a date somewhere between the date of last contact and the first missed visit date [21,33]. Adding 6 months to the follow-up time as in Variation A3 lead to an LTFU estimate of 1.3% and 8.8% LTFU at 6 and 12 months compared with 9.3% and 14.4% in Definition A, respectively. Presuming follow-up time is working with the unknown whereas the differential closure approach (where the analysis closes before the database) has no need to make any assumptions. Given these potential variations in how LTFU is defined in analysis, it is unsurprising that few analyses of ART program outcomes use the same definition of LTFU.

The LTFU estimates from each of scenarios A1–A3 appeared only modestly different from Definition A; but when combined in Variation A4, estimates of LTFU were substantially different. This shows that the cumulative impact of small differences in defining LTFU can have an appreciable influence on LTFU estimates.

We found that the site-defined estimates of LTFU varied greatly when compared with Definition A. This variability around a calculated estimate highlights that sites are likely defining outcomes differently, and it may be inappropriate to compare or combine site-defined outcomes across a number of sites [10]. The variability of site definitions suggests that the most efficient approach to standardizing LTFU in combined analyses is to generate LTFU from the data at the time of analysis.

To our knowledge, this analysis is the first to examine how defining LTFU in ART cohort analyses may produce different estimates of program outcomes. This work builds on previous methodological research that proposed using 180 days without contact since the last clinic encounter as an optimal period for defining LTFU to minimize misclassifications [11,12]. Other research has focused on determining risk factors for being LTFU [7], early and late in the program. Herein, we focus on the analytic definition of LTFU in cohort analyses and the impact that this may have on estimates of LTFU.

The objective of the analysis is to demonstrate the impact of LTFU definitions of ART program outcomes, and we have focused on a relatively short duration of 24-months follow-up. LTFU is defined in terms of the absence of death and transfers out and only at the analysis closure in terms of 6 months of nonattendance. We did not focus on treatment interruptions [34,35] or distinguishing observed versus actual LTFU [11,12], but acknowledged that these are important related issues that require additional consideration. Of note, we have used standard Kaplan–Meier methods, which are common in the ART program literature. Competing risks methods have been proposed as an alternative, but given the relatively low mortality observed in these data, the outcomes from the two methodologies are unlikely to differ substantively [36,37].

Concerns around the definition of LTFU are not restricted to ART program analyses, but are likely to emerge in any setting when LTFU may be an outcome of interest. For chronic care programs to be successful, patients need to be adherent and retained in treatment services [38]. For a patient to be adherent, they must be retained within the program and therefore all patients who are LTFU are by definition not adherent. Defining LTFU is therefore relevant in the operational research of all chronic care programs where patient retention is an outcome of interest.

In South Africa, the rate of ART program expansion was so rapid that systems were not in place to track and monitor the outcomes of patients after initiation. There are now renewed efforts to determine the outcomes of patients on ART. To determine the number of patients retained in the national programs, accurate estimates of patients lost to care are needed. Linkage with the National Death Registry has improved the accuracy of mortality data [25,39]. A standardized definition of LTFU to improve the accuracy of LTFU estimates is analogous to the linkage with the National Death Registry to improve mortality estimates necessary to appropriately assess ART program loss.

These results point to the need for a standardized definition of LTFU for ART service evaluation if we are to understand the changes within and the differences between ART programs. They also highlight the need to consider how LTFU is defined and assessed within health systems research involving other long-term therapies. Based on this work, we propose a standardized definition to LTFU that may facilitate comparisons of outcomes within and between the ART programs. Our recommendation is the most conservative Definition A, in which the analysis closes 6 months before the database, 1 day of follow-up is added for those who initiate treatment and never return, and those who do not have contact in the last 180 days be defined as LTFU on the date of last contact.

What is new?

Key findings

• There is growing interest in loss to follow-up (LTFU) as a cohort outcome. We assessed the impact of the definition of LTFU used in antiretroviral therapy (ART) programs in South Africa on estimates of LTFU.

• Different definitions of LTFU led to appreciably different estimates of the frequency of LTFU. Without a standardized definition, it is not possible to determine whether differences observed between analyses are real or an artifact due to different definitions.

What this adds to what was known?

• This study provides an overview of the LTFU definitions used in the literature. These differ in terms of the inclusion or exclusion of patients who started ART but never returned to the clinic, the closure of the analysis with respect to the closure of the database, and the date assigned to LTFU.

• The reference definition closed the analysis 6 months before the database, included those who initiate treatment and never return by adding 1 day of follow-up, and defined LTFU as those who do not have contact in the last 180 days using the date of last contact as the LTFU date. Comparing the reference to alternative definitions in over 40,000 patients followed up in eight ART programs in South Africa showed that estimates of LTFU varied widely. At 6 months the estimated percentage LTFU was 9.3% with the reference definition and 1.3% with the most extreme alternative definition. Corresponding percentages were 14.4% and 8.8% at 12 months and 22.4% and 18.6% at 24 months.

What is the implication and what should change now?

• Any analysis of chronic disease programs that considers patient retention as an outcome should pay particular care in defining LTFU. For ART programs we recommend that the reference definition described here should be adopted.