Novel metric for evaluating pre-exposure prophylaxis programme effectiveness in real-world settings

Summary

Although large-scale provision of HIV pre-exposure prophylaxis (PrEP) is gaining momentum, no systematic method to evaluate or compare the effectiveness of different scale-up strategies in real-world settings exists. To date, estimating the effectiveness of PrEP has relied on clinical trials or mathematical models. We propose a novel and pragmatic metric to evaluate and compare programme effectiveness using routine implementation data. Using South African and Zambian PrEP guidelines, we provide two examples of how to consistently measure PrEP-programme effectiveness with routinely collected data. PrEP effectiveness should account for HIV seroconversion, the variable risk of HIV infection (seasons of risk) estimated with routine risk assessment at each clinic visit (when available), and the persistence of PrEP use. Three criteria should be met in order to be considered a successful outcome: first, a person who initiates PrEP must not seroconvert; second, there should be no more than one period at high risk of HIV infection during the follow-up period when not taking PrEP; and finally, an individual must continue to attend health-care visits or discontinue prophylaxis in consultation with a health-care provider within a specified follow-up period. The number of PrEP successes could then be compared with the total number of people initiating PrEP to establish a success ratio. This outcome is a useful and easily interpretable metric to monitor effectiveness of PrEP programmes with routinely collected clinical data and can be used in cost-effectiveness analyses. These measurements allow for comparisons of scale-up strategies for PrEP programmes and, if widely adopted, will allow comparative studies of different approaches for PrEP service delivery.

Introduction

Oral HIV pre-exposure prophylaxis (PrEP) with tenofovir disoproxil fumarate and emtricitabine has now been recommended by WHO as part of a combination HIV prevention package to be offered to all people at substantial risk for HIV infection.¹ Countries in sub-Saharan Africa have begun to adopt these guidelines^2,3 and PrEP is now moving into the implementation phase, with research shifting from mostly efficacy-based studies and demonstration projects to include PrEP effectiveness and implementation evaluation.^4,5

Daily PrEP has been shown to be highly efficacious when adhered to correctly,⁶ and when implemented in a real-world setting might match clinical trial efficacy because of the strong element of self-selection among PrEP users. The PROUD PrEP effectiveness trial⁷ showed a greater reduction in HIV incidence than in any placebo-controlled trial to date. However, as different target populations across varied service-delivery settings are offered PrEP, implementation effectiveness is likely to vary widely.⁸ This variation has already been observed in Kenya where only nine (5.4%) of 168 adolescent girls and young women (16–20 years of age) who were offered PrEP accepted it.⁹ Under a routine implementation setting in maternal and child health, and family planning clinics, adolescent girls and young women (median age of 24; interquartile range 21–29) in Kenya also dropped out of care early, with the continuation of PrEP among this group reducing at 1 month (38%), 3 months (21%), and 6 months (10%).¹⁰ However, in another evaluation of routine PrEP implementation, predominantly in serodiscordant couples, 65% of participants continued PrEP (≥1 refill in 3 months).¹¹ The WHO monitoring and evaluation module highlights the importance of routine monitoring of PrEP programmes, emphasising that this step will be essential to assess uptake, effective use, and safety, and to predict demand and ensure sufficient and uninterrupted supply of commodities.^12,13

One barrier to rigorous evaluation of PrEP implementation is that no common metric with which to evaluate PrEP success in a real-world setting exists, either for individual people, or for a prevention programme. The obvious effectiveness measure for PrEP is to establish when an infection has been prevented, although this outcome can only be modelled or estimated in a clinical trial setting, and is not readily observed in individuals or prevention programmes during routine implementation. Variations on this traditional metric, such as the measure of averted infections to compare cost-effectiveness of different PrEP regimens,¹⁴ will be useful when comparing new drugs or regimen patterns, but cannot compare and evaluate effectiveness across PrEP implementation programmes. Another proposed metric incorporates changing risk behaviours for HIV acquisition and the use of alternative HIV prevention strategies; however, this metric will be challenging to implement in routine settings as it has data-rich requirements.¹⁵ To meet the demands of national HIV prevention efforts and to appropriately scale different programmes, an operational definition of PrEP success on the basis of routinely available data is essential. We propose a metric that measures programme effectiveness based on current country guidelines. This metric is designed to facilitate the evaluation of different PrEP-based HIV prevention programmes at the national level, but is not designed to measure the effectiveness of PrEP guidelines or assess therapy success in the public health system.

Developing an implementation metric

Defining PrEP success

The first step in developing a new metric is to define the outcome of interest. Unlike male circumcision and antiretroviral therapy (ART), PrEP is neither a one time nor a lifelong intervention, and the timing of this prophylaxis is based on the assessment of each client’s personal risk. Many PrEP clients are likely to go through so-called seasons of risk that require PrEP, and lower risk periods that might not.^1,16–20 Seasons of risk can include the end of a long-term relationship, the beginning of a new sexual practice, immigration to a new city, unanticipated economic hardship, seasonal engagement in sex work, or going on holiday. A change in behaviour can present as a time of risk when PrEP is necessary or when the risk has ended and PrEP is no longer required. Therefore, counting all those people who come off PrEP as an unsuccessful outcome is illogical, as many no longer require the drug because of a lowered risk of HIV acquisition. As a result, health-care workers must either assess potential risk during prevention counselling or use formal risk assessments. Expecting near perfect PrEP adherence regardless of risk status is impractical. Therefore, a meaningful metric for PrEP success must incorporate a measure of persistence during periods of risk and account for changes in behaviour that increase or decrease the risk of HIV acquisition over an individual’s life.

Once an individual client’s risk is taken into account, a metric for evaluating PrEP success must also consider two unsuccessful prophylactic scenarios. The first scenario is if a PrEP client drops out of care without any recorded consultation or reassessment of risk with a health-care provider, as the reason for dropout is unclear and might not reflect a reduction in risk. The second is if seroconversion occurs despite taking or being prescribed PrEP.

Metric inputs

A practical implementation metric should use routine implementation data available in resource-limited settings, where therapeutic drug concentrations monitoring and resistance testing are not commonplace. For most large-scale PrEP programmes, we anticipate that data on HIV status, provider visits, dispensed and pharmacy refill records, and some measure or indication of risk will be available, but information about adherence or behaviour between medication refills will not. Additionally, the definition and determination of risk is dependent on individual country guidelines and this metric assumes that when these guidelines are applied correctly they can identify people who are at a substantial risk of infection. This programme metric is meant to identify the successful application of country guidelines, including their own risk score or patient-identification process, and compare these results with programmes that operate within this guidance.

Finally, the proposed metric should allow comparison of programmes targeting similar population groups. For example, a programme for female sex workers might appear to be less successful than a programme for men who have sex with men, but this observation is not a good reason to favour funding one programme over another. Rather, a useful metric should evaluate programmes within risk groups.

We thus define a successful PrEP outcome as a person who initiates PrEP and does not seroconvert, someone who does not have more than one period at high risk of HIV infection during follow-up (not on PrEP), and finally, a person who remains engaged with care by attending visits or discontinues PrEP in consultation with a health-care provider within a stipulated follow-up period from therapy initiation (eg, 12 months).

Estimating individual scores for PrEP clients

For each PrEP client at every scheduled follow-up visit, we propose to estimate a score using routinely collected data to answer specific questions (table 1) and convert these into client scores. Because the questions used can be applied to any time period, a reasonable initial period of analysis needs to be set. Previously, different time frames have been used by various service-delivery models, making comparisons across programmes difficult. We propose a standard 12-month period for this metric, thereby further improving compatibility and allowing cross-programme evaluation. We suggest that this period starts at a client’s date of PrEP initiation.

Table 1:

Inputs, outputs, and formulas for assessing PrEP success

	Output score	Criterion threshold	Formula
Success criterion 1: maintains HIV uninfected status (a)
What is the PrEP client’s HIV status at the clinic visit? (d)	Negative=n, positive=p, or unknown=u	HIV status negative (n) or unknown (u) at all clinic visits, meets success criterion 1	If (d=n or d=u) then a=1; otherwise a=0
Criterion 2: PrEP persistence (b)
Did the PrEP client return for their follow-up clinic visit? (e)	Yes=1, no=0; if PrEP client does not miss >1 clinic visit during the 12-month period they are considered engaged in care; if client is on PrEP and misses >1 clinic visit they are considered to have dropped out of care	If client returned for follow-up clinic visit (e) and is either on PrEP (f) or stopped PrEP in consultation with the health-care provider (g), meets success criterion 2	If e=1 and (f=1 or g=1) then b=1; otherwise b=0
Was PrEP dispensed to the PrEP client? (f)	Yes=1, no=0	If client returned for follow-up clinic visit (e) and is either on PrEP (f) or stopped PrEP in consultation with the health-care provider (g), meets success criterion 2	If e=1 and (f=1 or g=1) then b=1; otherwise b=0
Was PrEP stopped in consultation with the health-care provider? (g)	Yes=1, no=0; if PrEP-client does not return for follow-up clinic visit this indicator will be scored 0; if client does not return during the follow-up period, but stopped PrEP in consultation with the provider, they are not lost to follow-up; if the PrEP-client does not attend 2 consecutive months of follow-up visits they are lost to follow-up and assumed to have stopped PrEP without consultation with the provider	If client returned for follow-up clinic visit (e) and is either on PrEP (f) or stopped PrEP in consultation with the health-care provider (g), meets success criterion 2	If e=1 and (f=1 or g=1) then b=1; otherwise b=0
Criterion 3: appropriate PrEP use* (c)
According to screening, is the PrEP-client at substantial risk of HIV infection? (h)	Yes=1, no=0; in countries where formal risk screening is not done, a prescription from a clinician is considered adequate to indicate a substantial risk of infection	If client is at substantial risk of HIV infection (h) and dispensed PrEP (f), meets success criterion 3	If h=1 and f≠1 then c=0; otherwise c=1

If f=1 and g=1 (more than once) and h=1 (more than once) PrEP success score=1; otherwise=0. PrEP=pre-exposure prophylaxis.

^*Being appropriately on PrEP depends on a country’s guidelines for determination of risk.

The PrEP success score incorporates all the data and criteria described in table 1 and is based on three criteria: maintenance of HIV-negative status (criterion 1 or a); PrEP persistence (criterion 2 or b); and appropriate PrEP use (criterion 3 or c); success is achieved if all three criteria are met. We note that this metric is binary to avoid weighting unsuccessful outcomes without evidence of how to value these different outcomes appropriately. In addition, table 1 applies only to clients who have already started PrEP and does not comment on the uptake of PrEP among people who have not started prophylaxis but are considered at substantial risk.

Estimating programme success

We have explained how to establish if any individual client is a PrEP success (table 1). To evaluate a programme, rather than an individual client, the number of PrEP successes for a group of clients can be divided by the total number of PrEP initiates during a specified time. This success ratio can then compare PrEP strategies across different sites and programmes, potentially even nationally or across countries, depending on the differences in definitions of a PrEP success. As mentioned, however, evidence has already shown that PrEP risk, uptake, persistence of use, and adherence varies depending on different population groups. We are therefore cautious about comparing programmes that target different populations.

Applications of the proposed metric

Using both the South African and Zambian PrEP delivery and monitoring and evaluation guidelines,^16,18 we provide two examples of how to measure PrEP programme effectiveness with routinely collected data by applying our proposed metric. These examples use hypothetical PrEP client data to show different outcome possibilities (table 2). Both programmes dispense oral tenofovir disoproxil fumarate and emtricitabine (Zambia also dispenses tenofovir disoproxil fumarate and lamivudine) as a fixed-dose combination and require a clinical follow-up visit 1 month after PrEP initiation and then every 3 months thereafter. Monthly prescription refills are recommended for the first 12 months, but the metric only considers refills atthe 3-monthclinicalvisit as othernecessary indicators (HIV test results, risk assessment, and prescription renewal) are usually only done at these appointments. Future use of the metric will need to take visit schedules into account when comparing across programmes. The South African PrEP programme offers oral PrEP to high-risk populations who are at substantial risk of HIV infection (defined as a population group with an HIV incidence greater than three per 100 person-years in the absence of PrEP). Specific populations considered to be at substantial risk of HIV infection include adolescent girls and young women (15–24 years old), men who have sex with men, people with concurrent or more than one sexual partner, people who inject drugs, people with a recent history of sexually transmitted infections, people who recognise their own risk and request PrEP, serodiscordant couples when a partner with HIV is not virally suppressed, and sex workers. South African guidelines indicate that a potential PrEP user should receive counselling in relation to HIV infection risk at their screening visit (before PrEP initiation or reinitiation)—“If the client’s lifestyle is high risk, oral PrEP will be presented as an option [as a strategy to remain HIV negative].” ¹⁶ The South African guidelines provide counsellors with a series of steps and questions to follow in order to ascertain someone’s HIV risk and PrEP eligibility (appendix p 1); although, this process is not routinely documented or translated into a risk score. In South Africa, we assume that a consultation with a clinician that results in a PrEP prescription is considered adequate to indicate an individual at substantial risk. This status might be established by a health-care provider or as a perceived risk based on a client who requests to be on PrEP. The South African PrEP guidelines state that “Users who want to stop PrEP should do so after consultation with the healthcare provider.” ²¹ Therefore, if a client taking PrEP misses a follow-up visit, and for this reason is not prescribed PrEP, they are assumed to still be at risk. However, if a client on this therapy attends their visit and has a consultation with a health-care provider but is not prescribed PrEP, they are no longer considered to be at risk. Additionally, WHO guidelines indicate that PrEP medication should be continued for 28 days after the last potential HIV exposure to ensure protection.¹

Table 2:

South Africa and Zambia’s proposed PrEP initiation and follow-up timeline with routine data collected from hypothetical client examples showing the proposed metric for PrEP success outcomes

	PrEP client 1*	PrEP client 2†	PrEP client 3‡	PrEP client 4§	PrEP client 5¶	PrEP client 6∥	PrEP client 7**
PrEP initiation visit
Month 0
HIV status	Negative	Negative	Negative	Negative	Negative	Negative	Negative
At risk of infection††	1	1	1	1	1	1	1
PrEP dispensed	1	1	1	1	1	1	1
PrEP follow-up visits
Month 1
Returned‡‡	1	0	1	0	1	1	1
HIV status	Negative	Unknown	Negative	Unknown	Negative	Negative	Negative
At risk of infection††	1	1	1	1	1	1	1
PrEP dispensed‡‡	1	0	1	0	1	1	1
Month 3
Returned‡‡	1	0	1	1	1	1	1
HIV status	Negative	Unknown	Negative	Negative	Negative	Negative	Negative
At risk of infectiont††	1	1	1	1	0	0	0 or 1§§
PrEP dispensed‡‡	1	0	1	1	0	0	0
Month 6
Returned‡‡	1	1	0	1	NA	NA	1
HIV status	Negative	Positive	Unknown	Negative	NA	NA	Negative
At risk of infection††	1	NA	1	1	NA	NA	0 or1§§
PrEP dispensed‡‡	1	NA	0	1	NA	NA	0
Month 9
Returned‡‡	1	NA	0	1	NA	NA	1
HIV status	Negative	NA	Unknown	Negative	NA	NA	Negative
At risk of infection††	1	NA	1	1	NA	NA	0 or 0§§
PrEP dispensed‡‡	1	NA	0	1	NA	NA	0
Month 12
Returned‡‡	1	NA	0	1	1	NA	1
HIV status	Negative	NA	Unknown	Negative	Negative	NA	Negative
At risk of infection††	1	NA	1	1	1	NA	0 or 0§§
PrEP dispensed‡‡	1	NA	0	1	1	NA	0
Individual metrics
Engaged-in-care score	1	0	0	1	1	1	1
HIV seroconversion score	1	0	1	1	1	1	1
Appropriately on PrEP score (South Africa’s assumption of risk)	1	0	0	1	1	1	1
Appropriately on PrEP score (Zambian risk assessment)	1	0	0	1	1	1	0
PrEP success score
South Africa	1	0	0	1	1	1	1
Zambia	1	0	0	1	1	1	0

Success ratios can range from 0 to 1. The success ratio is 0·71 (5 of 7 PrEP clients) for the South African programme and 0·57 (4 of 7 PrEP clients) for the Zambian programme. PrEP=pre-exposure prophylaxis. NA=not applicable (because the patient discontinued PrEP in consultation with a provider, or seroconverted).

^*Client 1 was at risk and on PrEP for the duration of follow-up and never seroconverted (success).

^†Client 2 seroconverted at the 6-month visit (unsuccessful).

^‡Client 3 was lost to follow-up after 3 months without indication of risk reduction (unsuccessful).

^§Client 4 missed the PrEP follow-up visit at month 1, but was otherwise on PrEP for the rest of the follow-up period (success).

^¶Client 5 went through seasons of risk and was on PrEP when at risk (success).

^∥Client 6 stopped PrEP after 3 months in consultation with a health provider and is assumed to no longer be at risk of infection (success).

^**Client 7 was at risk of infection according to the Zambian risk assessment, but was not prescribed PrEP at months 3 and 6 (unsuccessful). The South African metric assumed the client was not at risk because they were not prescribed PrEP (success). At months 9 and 12 the risk assessment in Zambia established the client was not at risk and they were not prescribed PrEP (success). As with the South African metric, this client continued to be a success because they were not prescribed PrEP and were therefore assumed to be at low risk.

^††If a client had a visit with a health-care provider (initiation or follow-up) and was prescribed or dispensed PrEP in that visit, the client was considered to be at risk. If a client missed a visit and as a result was not prescribed PrEP, they were assumed to still be at risk.

^‡‡If a client visited a health-care provider and was not prescribed or dispensed PrEP (stopping PrEP) they were considered no longer at risk of HIV acquisition.

^§§The risk score for client 7 differs for the Zambian and South African programmes because of the way risk of infection is measured with routinely-collected data from each country. Zambia documents risk through formal risk assessments, whereas South Africa relies on healthcare counselling to establish risk and this data is not currently recorded. As such, this metric assumes that if a client is prescribed PrEP the health-care provider believes they are at risk of infection

The Zambian PrEP programme, in contrast, uses a baseline risk-assessment tool to offer PrEP to anyone at high risk of HIV infection. PrEP is not exclusively restricted to key populations, although the guidelines state that “PrEP may also be considered for key populations (as defined by the 2017 [National Strategic Framework]) or by persons self-selected as high risk for HIV acquisition.” ¹⁸ Those people who start PrEP continue to be provided prophylaxis on the basis of reassessment of risk at each clinical visit. Health-care providers ask clients on PrEP a series of questions (appendix p 1) regarding their past risk of HIV infection and use these answers to estimate future risk.

We have provided examples for defining successful PrEP outcomes that account for seasons of risk, both with (South Africa) and without (Zambia) an explicit and recorded HIV-screening tool (table 2). For the Zambian programme, the overall number of PrEP successes in this hypothetical cohort was four of seven initiates, leading to a success ratio of 0·57 For South Africa, one additional person on PrEP was counted as a success, resulting in a success ratio of 0·71.

Future applications

We believe that the metric described in this Viewpoint can serve as a measure for implementation at the site, programme, and country level, or for any population that has access to PrEP under the same eligibility and implementation guidelines. As the examples show (table 2), the routine data collected in both South Africa and Zambia allow application of this metric. We foresee this strategy being used to assess the performance of different sites within a single programme or to compare different service-delivery approaches across programmes, such as in community-based and facility-based care. Furthermore, another strength of this simple metric is that it can be applied and compared between a data-rich setting and a data-poor setting if warranted, which could allow for comparisons between trials or demonstration projects in addition to routine care if they are done within the same country or group.

Other potential uses for this metric exist. First, we recognise that cross-country comparison will remain a challenge because of differences in guidelines for PrEP and routine data-collection procedures. However, a country might use another country’s success ratio to benchmark its own against a regional or global standard. Although not all country guidelines call for a formal documented determination of risk before or during PrEP, in cases where a measure of risk is not collected, the new metric allows inference based on prescription records. Second, this metric will ultimately improve the ability to estimate the cost-effectiveness of different PrEP programmes. Cost-effectiveness analysis requires a cost comparison of interventions with the same outcome, be it units of health improvement, life-years gained, or quality of life. It provides a common outcome for PrEP, across what might be widely different implementation approaches. This metric would also weight programmes that target successful outcomes more favourably. For example, if programme 1 has a high cost per person but has good PrEP persistence and engagement in care, and programme 2 has a low cost per person but clients on PrEP rarely return to the clinic for a follow-up appointment after initiating PrEP, programme 1 might be viewed more favourably. This metric is not intended to replace the frequency of averted HIV infections but rather to provide programmes and donor agencies with an alternative measure that can be used within and between programmes to identify cost-effectiveness on a more granular level with routinely collected data. Finally, if this metric were to evaluate PrEP success in a data-rich setting, this metric could allow for the collection of additional data for the assessment of predictors of successful PrEP persistence.

Limitations to the proposed metric

Reliance on existing assessment of HIV infection risk

The hypothetical examples (table 2) show how a programme that measures and records the risk of infection has implications for whether someone is deemed to be appropriately prescribed PrEP or not. Zambia, with their formal risk-assessment tool, has a more explicit documentation of risk that can be used to assess PrEP success. South Africa does not document risk in the same way and, as such, our assessment of PrEP success is based on whether a client was prescribed PrEP. Both have their strengths and weaknesses and both assessments of risk might under or over-report those people at substantial risk of HIV acquisition. Zambia’s measure is standardised and documented,¹⁸ but might be difficult to interpret because of its retrospective nature (ie, asks about past sexual behaviour to infer future risk). South Africa’s guidelines allow for individualised assessment, but assume that being on PrEP in itself is indicative of risk, otherwise the prophylaxis would not be prescribed.¹⁶ This metric relies on national definitions and assessment of risk to measure PrEP success and, depending on the reliability of a country’s risk-assessment strategy, might under or overestimate a programme’s success.

For our proposed metric we have assumed that if an individual is reported as being at low risk of infection and has been prescribed PrEP, then there is a reason for this decision that is not captured in the risk-assessment tool and the outcome is considered a success. The metric does not capture people who do not present for care, but might be at risk, or those people who never start PrEP. Additionally, some people who are assessed as low risk should probably be on PrEP.

Omission of pre-PrEP cascade

Our proposed metric starts with the initiation of PrEP and, as a result, does not account for the initial steps in the PrEP cascade as outlined by Liu and colleagues.²² WHO has recommended PrEP uptake (initiation) as a key indicator of programme performance, but there appears to be slow adoption of this metric.¹³ Like measures of retention in care or viral suppression for people on ART, which indicate outcomes but not uptake, our proposed metric does not take into account a programme’s success at initiating people at high risk onto PrEP. The omission of the pre-PrEP cascade reflects, in part, the relatively new data systems that support national PrEP programmes but do not track cohorts of people from testing negative at the risk screening through to PrEP initiation. A consistently documented metric whose denominator is all individuals who test negative but have a baseline assessment of high-risk infection is still needed to assess PrEP uptake in a standardised metric.

Gaps in PrEP refill visits

Our proposed metric does not specifically account for gaps in PrEP refill or pharmacy visits when they do not coincide with scheduled clinic appointments. With the expansion of differentiated service-delivery models and the introduction of multimonth dispensing in many countries, future clinic appointments might be likely to coincide with pharmacy visits in many programmes. There is also a possibility that individuals might use PrEP intermittently to cover specific exposures, and as a result missing a refill might not indicate a period of uncovered risk. Using the scheduled clinic visit as an indicator, the programme can focus on engagement in care (persistence) rather than adherence, which is challenging to interpret in the context of PrEP therapy.

However, as the metric does not measure length of prophylaxis and is rather a binary result (successful or unsuccessful) at the end of the follow-up period, we do not think that these refill visits will impact the overall programme success score. Additionally, persistent gaps in drug coverage would be identified through missed clinic appointments and HIV infection if periods of high risk continued during these uncovered times.

Conclusion

As countries implement and scale up their PrEP services, much will be learned about what is effective and what is not. To identify best practices, both in terms of clinical and economic impact, a systematic way to evaluate and compare programmes and service-delivery models is needed. Programmes are tasked with increasing coverage of PrEP therapy in priority populations, monitoring the PrEP cascade, and the evaluating effect. This novel metric of PrEP success considers individual risk of HIV infection, PrEP persistence during periods of risk, and HIV acquisition, thereby allowing programmes to assess their performance in a multidimensional way. Use of this metric early on in PrEP scale-up will allow decision makers to make informed and evidence-based choices that will help them achieve national and global goals for PrEP care.