Standardised protocol for a prospective cross-sectional multicentre clinic-based evaluation of two dual point-of-care tests for the screening of HIV and syphilis in men who have sex with men, sex workers and pregnant women

Abstract

Introduction

Dual point-of-care tests (POCTs) for detecting antibodies to HIV and syphilis have been developed for use with venous whole blood, serum/plasma or finger-prick capillary whole blood. Several tests are commercially available showing encouraging performance compared with ‘gold-standard’ reference tests in laboratory-based studies. However, data on their performance in the field are limited. This prospective cross-sectional study will conduct a clinic-based evaluation to assess the performance characteristics and acceptability to end-users of two dual HIV/syphilis POCTs for the screening of HIV and syphilis among men who have sex with men (MSM), sex workers (SWs) and pregnant women (PW). This master protocol outlines the overall research approach that will be used in seven countries.

Method and analysis

MSM, SWs and PW presenting at clinic evaluation sites in high, low and middle-income countries will be enrolled. The (WHO preapproved) POCTs to be evaluated are SD Bioline HIV/Syphilis Duo (Abbott) and Dual Path Platform HIV-Syphilis Assay (Chembio). Finger-prick blood will be collected to perform POCTs and compared with laboratory results (venepuncture blood). Procedures will be carried out by trained healthcare staff and tests performed according to the manufacturers’ directions. Sample size was calculated based on local prevalence of HIV and syphilis. The sensitivity, specificity, positive and negative predictive values for each POCT will be calculated. The study is ongoing with recruitment expected to be completed in all countries by mid to late 2021.

Ethics and dissemination

This core protocol was independently peer reviewed and approved by the Research Project Review Panel (RP2) of the WHO Department of Sexual and Reproductive Health and Research and by the WHO Ethics Review Committee (ERC). The protocol has been adapted to individual countries and approved by RP2, ERC and institutional review boards at each site. Results will be disseminated through peer-reviewed journals and relevant conferences.

Strengths and limitations of this study.

• To our knowledge, the study is the first independent multicentre and multicountry clinic-based evaluation of dual HIV/syphilis point-of-care tests (POCTs) for the screening of men who have sex with men, sex workers and pregnant women.

• The clinic-based evaluation protocol design more accurately reflects real-world conditions and therefore has greater potential to demonstrate the dual POCTs’ practical value as a screening tool, compared with laboratory-based evaluations.

• Although dual HIV/syphilis POCTs can be convenient for patients, treponemal antibodies persist following successful treatment of syphilis. Therefore, additional treponemal and non-treponemal tests may still be required for correctly identifying active infections.

• The public health significance of this study is based on the premise that it will evaluate the performance characteristics of two dual POCTs in field settings as well as enable assessment of operational characteristics by healthcare workers and acceptability to patients.

• The study will identify opportunities to support the WHO Global Health Sector Strategy on Sexually Transmitted Infections, 2016–2021.

Introduction

Serological tests are the diagnostic tests of choice for both syphilis and HIV. For syphilis, two serological tests (treponemal and non-treponemal) are used for laboratory diagnosis. Traditionally, a non-treponemal assay is used for screening and a specific treponemal assay is then used for confirmation. However, with increasing automation and decreasing cost, some institutions use the ‘reverse’ algorithm, whereby initial screening is performed with automated treponemal enzyme-linked or chemiluminescence immunoassays. If reactive, it is followed by a second manual non-treponemal test such as the rapid plasma reagin (RPR) assay to assess disease activity.¹

With respect to HIV diagnosis, two different rapid lateral flow immunochromatographic assays or automated enzyme immunoassays (EIAs) are used in tandem for screening and confirmation. In some settings, a western blot assay is performed for the confirmation of a positive result. However, some of these tests are technically demanding and require laboratory equipment that is not always widely available in resource-limited settings. Therefore, in recent years, there has been considerable effort to develop new diagnostic tools including point-of-care tests (POCTs), which can be used outside of a typical clinical laboratory (eg, a physician’s office, hospital bedside, patient’s home, in the field/community based), by non-laboratory trained healthcare providers (HCPs).² Over the last decade, many POCTs for sexually transmitted infections (STIs) have been developed and are commercially available.³ Guidance or recommendations regarding the placement of these new tests in the diagnostic pathway for STIs are limited, however, interim recommendations on the use of POCTs among pregnant women (PW) are available.⁴ Furthermore, although validated and licensed rapid HIV assays are commercially available and widely used in the field, they are standalone and do not offer the function of dual screening. As a result, syphilis testing coverage may be suboptimal relative to HIV. Integrated testing services will improve testing coverage.^{5 6} The WHO Global Health Sector Strategy on STIs acknowledges the lack of reliable, low-cost POCTs as a major barrier to advancing STI control and prevention.⁷

As HIV and syphilis can be asymptomatic, sensitive diagnostic testing is crucial for early detection and diagnosis and for the guidance of treatment and prevention of onward transmission.⁸ Dual tests that can be used at the point-of-care for simultaneously detecting antibodies to HIV and syphilis (dual HIV/syphilis POCTs) have been developed for use with finger-prick capillary whole blood. To date, they have shown encouraging performance compared with ‘gold-standard’ diagnostic reference tests in laboratory-based studies; but there are limited data on their performance in the field.^{9 10} Field studies in real-world settings are important because the performance of POCTs, including positive predictive values (PPVs) and negative predictive values (NPVs), can be influenced by epidemiological and environmental factors. Both operational characteristics of the test and human factors, such as the ability to follow properly the POCT procedures (eg, taking whole blood finger-prick specimens, correct timing of adding the buffer and reading the result and interpreting results accurately), can interfere with the performance of a POCT.¹¹ Evaluation of the performance of these dual tests in clinic-based settings and their acceptability to patients and HCPs is therefore a high priority. The need for standardised high-quality evaluations of POCTs was identified by two WHO technical consultations as being critical for the development and global uptake of POCTs for STIs.⁷

Consequently, the Sexual and Reproductive Health and Research Department of WHO has established the global ProSPeRo study (Project on Sexually Transmitted Infection Point-of-care Testing). The overall objectives are to (1) advise WHO Member States and other public health institutions on the performance characteristics of commercially available STI diagnostic tests that can be used at the point-of-care, (2) assess the feasibility, acceptability of POCTs by both HCPs and clients/patients and (3) support further implementation and rollout of STI POCTs within national STI programmes by the provision of technical assistance tools.

ProSPeRo comprises three core components: (1) a laboratory-based arm assessing the performance characteristics of STI POCTs that have not yet been evaluated independently in the laboratory¹² (2) a clinic-based component to evaluate STIs POCT performance in the field compared with that of gold-standard laboratory tests among several STI high-risk and vulnerable populations worldwide and (3) a clinical utility component assessing the feasibility and acceptability of STI POCTs among men who have sex with men (MSM) in non-clinical settings in four countries within the WHO European region.

This master protocol refers to the second component of ProSPeRo, a clinic-based evaluation of dual POCTs for the screening of HIV and syphilis in MSM, sex workers (SW) and PW. These groups were identified as those most in need of increased access to testing based on a series of consultations led by WHO, which considered definitions of key and vulnerable populations¹³ and the epidemic trends of HIV/STIs among them.^{14 15} Furthermore, elimination of mother-to-child-transmission of syphilis and HIV is considered to be one of the most cost-effective public health interventions and WHO guidance for country-level action includes targets for the screening and treatment of PW.¹⁶

A clinic-based evaluation is used to determine test performance when the test is performed by clinic personnel who are not trained laboratory technicians. This paper outlines the master protocol within ProSPeRo, describing the overall research approach that will be used for a clinic-based evaluation of dual POCTs for the screening of HIV and syphilis in MSM, SW and PW. The protocol was developed using the QUADAS-2 framework, a Quality Assessment tool for primary Diagnostic Accuracy Studies.¹⁷ The protocol will be adapted accordingly for individual countries taking part.

Objectives

The primary objectives of this clinic-based evaluation are to assess: (1) the performance of two dual POCTs for the screening of HIV and syphilis in MSM, SW and PW using finger-prick capillary whole blood compared with reference laboratory-based serum tests for HIV and syphilis (HIV 1/2 EIA and the treponemal reference test) and (2) the minimal operational characteristics and acceptability of these dual HIV–syphilis POCTs to both patients and HCPs. A secondary objective is to explore the performance and the potential utility of these dual HIV–syphilis POCTs in identifying active syphilis using a combination of non-treponemal and treponemal tests as a comparator.

Methods and analysis

Study setting and design

This clinic-based evaluation is a multisite, observational, cross-sectional study of MSM, SW and PW presenting at sexual health and antenatal clinics for HIV/STI screening according to national guidelines. The study will be implemented across multiple countries and in some countries, at multiple sites, on the basis of locally adapted protocols. For the purposes of this protocol, the term study site or evaluation site refers to an individual clinic (sexual health or antenatal).

This paper is the master protocol and outlines the overall research approach that will be adapted accordingly for individual countries. Seven international sites have been approved by WHO in consultation with in-country researchers and providers as well as local authorities and WHO Country Offices (Italy, UK, Malta, Peru, Uganda, Morocco, South Africa). A standardised site assessment is implemented as part of the approval process for sites expressing an interest to participate. Site-specific protocols are developed with WHO and the in-country principal investigator to agree and delineate the range of parameters and the minor changes needed to adapt the study to the local context while complying with this master protocol. The study is ongoing with recruitment expected to be completed in all countries by mid to late 2021.

Study participants

Inclusion criteria

The target populations are MSM, SW and PW. The term MSM is used to describe those men who have sex with other men, regardless of whether or not they have sex with women or have a personal or social identity associated with that behaviour, such as being ‘gay’ or ‘bisexual’. Sex work is the provision of sexual services for money or goods. SWs are women/men who receive money or goods in exchange for sexual services and who consciously define those activities as income-generating even if they do not consider sex work as their occupation. Antibiotic usage in participants who have been prescribed treatment for syphilis or other infections 3 weeks prior to study entry will be recorded in the data collection form but not used as a criterion for exclusion. All participants have to be at least 18 years old to participate.

Healthcare staff who administer the POCTs will also be asked to complete a short questionnaire to evaluate the operational characteristics (box 1). In order to be eligible to complete this, they need to have been trained in and administered the POCTs and give consent to complete the questionnaire.

Box 1. Acceptability questions and operational characteristics.

Healthcare staff—operational characteristics

1. Clarity of kit instructions (difficult to follow, fairly clear, very clear, excellent).

2. Ease of use (complicated, fairly easy, very easy, excellent).

3. Ease of interpretation of results (difficult, fairly easy, very easy, unambiguous).

4. Rapidity of test results (<20 min, 20–30 min, >30 min).

5. Hands on time (<5 min, 5 min, 10 min, >10 min).

6. Training time required (<30 min, 30 min, 1 hour, >1 hour)

7. Number of tests needed to be performed before being able to feel comfortable administering a point-of-care test.

8. Overall comment/recommendation (free text response).

Patient—acceptability questions

1. Would you be willing to wait for the results at the clinic, directly after the tests are performed? (yes, no, do not know).

2. If yes, how long would you be willing to wait? (up to 20 min, up to 30 min, up to 1 hour, up to 2 hours, do not know, other).

   • 2.1. If other, please describe.

3. Would you prefer two single tests or one dual test? (single, dual, it’s the same, do not know/don’t care).

   3.1. If you prefer a single test, why? (do not want to be tested for HIV, do not want to be tested for syphilis, other).

   • 3.1.1. If other, please describe.

Exclusion criteria

MSM, SW, PW, and healthcare staff who refuse to give consent, are younger than 18 years and/or have previously participated in the study.

Description of the POCTs under evaluation

The tests to be evaluated are SD Bioline HIV/Syphilis Duo (Abbott Diagnostics, USA; hereafter termed Bioline POCT) and Chembio Dual Path Platform (DPP) HIV–Syphilis Assay (Chembio, USA; hereafter termed Chembio POCT). Both are single-use qualitative immunochromatographic assays for the simultaneous detection of HIV and syphilis (treponemal component) in human serum, plasma, whole venous or fingerpicked blood. The Bioline POCT (figure 1) detects antibodies of all isotypes (IgG, IgM, IgA) against HIV specific antigens (HIV-1 gp41, sub O, HIV-2 gp36) and specific IgG and IgM antibodies against a 17 kDa recombinant Treponema pallidum antigen (rTp17 kDa). In 2015, this test was accepted for the WHO list of prequalified in vitro diagnostics.¹⁸ The Chembio POCT detects specific antibodies against HIV types 1 and 2 (HIV 1/2) and Treponema pallidum (figure 2). Recently, the Chembio company developed the DPP Micro Reader (MR) to complete the Chembio DPP technology and minimise error due to subjective visual interpretation (figure 3). The MR is a portable battery-powered cubic reflectance reader with a liquid crystal display. The display shows the status of the instrument and test result to the operator and a multifunction button on top turns on the MR and guides the operator. The device is fitted to the Chembio POCT via a dedicated holder. The MR scans the Chembio POCT cartridge and verifies the presence of line(s) at the control and each of the test line positions and measures numerically the optical density of each test line(s). The device interprets the results comparing the optical density with that of a scoring algorithm and translating it into a numerical value. This number is compared with that of the set cut-off (>20 for the HIV component, >10 for the treponemal component) and if higher, the POCT result is displayed as reactive (R), otherwise, as non-reactive(NR). If the MR reading phase is not successful, the displayed result is invalid (I).

Figure 1.

Bioline point-of-care test.

Figure 2.

Chembio point-of-care test kit. DPP, Dual Path Platform.

Figure 3.

Chembio DPP micro reader. DPP, Dual Path Platform; RFID, Radio-Frequency IDentification.

Reference laboratory tests

For this master protocol, Bioline and Chembio POCTs HIV/syphilis results will be compared with those of the reference or ‘gold-standard’ tests for this evaluation; respectively these are laboratory-based HIV 1/2 EIA confirmed by immunoblot (antibody only), and for syphilis, the reference test is the Treponema Pallidum Particle Agglutination (TPPA). All reference tests will be performed in accordance with the manufacturer’s directions and laboratory staff will be blinded to the POCT results. For the determination of potential utility, a probable active syphilis case is defined as non-Tp (such as RPR test positive at a titre >8), confirmed by TPPA.

Acceptability of dual POCTs

The acceptability of the dual POCTs to patients will be determined as part of the evaluation using a structured questionnaire. Moreover, at the end of enrolment, operational characteristics will be evaluated by healthcare staff who administer the POCTs via a brief self-completion provider questionnaire. These acceptability questions and operational characteristics are displayed in box 1.

Study procedure

Recruitment, enrolment and consent

For each site, patients will be recruited over 9 months (maximum) or until the required sample size is reached. Consecutive MSM, SW and PW presenting to the clinic for HIV/STI screening at the evaluation sites will be informed about the study by HCP 1 (see figure 4 patient flowchart). If they are interested in participating (preconsent), a second HCP (HCP 2) will evaluate the inclusion criteria and provide a participant information sheet and informed consent form. If the potential participant fits the criteria and agrees to participate, HCP 2 will take final written informed consent and HCP 3 will perform the routine care (below) and the additional tests along with completion of the associated case report forms (CRFs).

Figure 4.

Patient flowchart for a clinic-based evaluation of two dual HIV/syphilis POCTs. HCP, healthcare provider; POCT, point-of-care testkit.

Specimen collection

The HCP will undertake a routine examination of the patient according to local clinical procedures. Participants will be asked to complete an acceptability questionnaire. About 3–5 mL of venous blood will be collected for reference testing and placed into labelled collection tubes (with anticoagulant that does not interfere with the assays) and then transported to the laboratory in accordance with standard operating procedures at the local site. The venous blood sample will be processed for reference testing within 12 hours of collection or centrifuged and stored at 2°C–8°C for 5 days maximum.

A capillary whole blood sample will be collected using finger-prick blood to perform both POCTs according to the manufacturers’ instructions, collecting the required amount of capillary blood using the equipment provided in both test kits and waiting a determined time (measured with a timer for each test) before reading the results. With the finger only being pricked once, the first blood drop will be used for the Bioline test and the second drop for the Chembio test. A double reader method (reader 1–reader 2 (R1–R2)) will be adopted for both tests to determine any variability in the interpretation of test results.¹¹ The MR (Chembio) will be read by R2 only. R1 and R2 will be blind to each other’s results and to the clinical history of the patients.

Follow-up procedure

Pretest and post-test counselling will be provided to all participants according to WHO recommendations and local clinical practice. Participants will not be informed of their POCT results and the results will not be used for follow-up because the POCTs are being evaluated in the context of already established clinical testing procedures. Patients with a positive reference test result will be treated according to the standards of care described in the national guidelines for each evaluation site.

Outcomes

Primary outcome: the primary outcome is a measure of the performance of two dual POCTs for the screening of HIV and syphilis. Each test component will be considered separately and compared with the respective gold standard. Readings of the first rater (R1) will be considered as the outcome variable for the comparison. Secondary outcomes: two secondary outcomes are as follows: (1) the acceptability of the two dual HIV/syphilis POCTs to patients and HCPs and (2) the potential utility of two dual HIV/syphilis POCTs in identifying active syphilis using a combination of non-treponemal and treponemal tests.

Sample size

The formula used for the sample size calculation is based on the 2006 WHO/TDR (Special Programme for Research and Training in Tropical Diseases) expert panel document on the evaluation of new diagnostic methods and techniques.¹⁹ As per the standard approach, the sample size is adjusted for the infection prevalence using the formula (100/prevalence × sample size using sensitivity/specificity only). For example, if it is estimated that the sensitivity of a new test is 80% compared with the reference standard, then 200 infected study subjects by the reference standard test would need to be recruited for a CI of +5% around the point estimates of sensitivity and specificity. If the prevalence of infection in the study population is 10%, then there will be 10 infected subjects per 100 healthcare users seen at the clinic. Then 2000 (100/10×200) participants will need to be recruited.

Project and data management

To ensure appropriate implementation of this master protocol, the following actions will be conducted: (1) development of site-specific study management plans including details of the roles and responsibilities of the study/evaluation team (the composition and number of study team members will be adapted at each site according to local need), (2) WHO monitoring visits and monitoring procedures to assess the progress and quality of the study at each evaluation site, (3) an external quality assurance process to standardise reference testing results from different sites by use of proficiency panels sent by the Centres for Disease Prevention and Control in the USA (a WHO reference laboratory), (4) an internal (serum) and external (dried tube specimens) quality assurance process for ensuring accurate performance of the dual HIV/syphilis POC tests and (5) a site-sensitive training programme for all clinic staff in specimen collection and handling including performance and reading of the POCTs as well as familiarisation with the study standard operating procedures.

All data generated will be recorded using WHO standardised CRFs. Paper versions will be stored securely at each study site as per local standard procedures. At regular intervals, data from these CRFs will be entered by a data manager at each site into a WHO provided secured laptop using OpenClinica software. Once data entry is complete, local data managers will be requested to re-enter a random allocation of data to reduce data entry error. Once both enrolment and data entry are complete, the dataset will be deidentified. Archiving (including destruction) of paper versions of the CRFs will be determined by the evaluation sites’ own procedures.

Data analysis

Data analysis will be guided by QUADAS-2 principles.¹⁷ Sensitivity, specificity, PPVs, NPVs, likelihood ratio of a positive test and likelihood ratio of a negative test for each rapid test component will be estimated comparing the POCT results with the gold-standard laboratory test results. Each rapid test component will be then compared with the respective gold standard, namely, POCT treponemal versus laboratory-based treponemal (TPPA) for syphilis and POCT HIV result versus laboratory-based HIV EIA and/or immunoblot. 95% CIs of each estimate will be calculated.

To validate the reading of the POCT outcome variable, the concordance between the R1–R2 readings will be estimated by calculating percentage agreement (concordance) and Cohen’s kappa (κ for binary variables).

Cohen’s κ represents a measure of inter-rater agreement, ranging from −1 to +1, where 0 is the level of agreement that can be expected in case of random chance.

Patient and public involvement

Patients, representatives of communities, and healthcare staff were consulted during the development of this master protocol specifically regarding participant recruitment and approach. Additional consultations were held during adaptation of the master protocol to individual sites.

Ethics and dissemination

This master protocol was independently peer reviewed and approved by the Research Project Review Panel (RP2) of the WHO Department of Sexual and Reproductive Health and Research and by the WHO Ethics Review Committee (ERC). It has also been adapted to individual countries and approved by RP2, ERC and institutional review boards at each site (see online supplemental file). As the study’s procedures generally follow standard of care pathways at each clinic site, minimal risks are foreseen. However, one potential risk identified by the WHO ERC was the recruitment of participants from key populations, particularly MSM, in countries where self-identification as MSM could cause harm for study participants. Therefore, sites will only be included in the study if they have experience of working with key populations, which is reviewed as part of the initial site-assessment. Some patients might experience a small amount of transient discomfort when the finger-prick blood sample is taken for use with the two POCTs. All data and records are to be used only for the purpose of the research project. Names will not be used on any study form or label on laboratory specimens or in any report resulting from the study. At the beginning of the study, a unique study identification number will be assigned to each participant. This number will be used to link the different types of data collected for each participant (demographic and behavioural information, biological samples) and their POCT and reference test results.

All information/data obtained will be kept confidential. Only members of the clinic study team (HCPs) will have access to information linking a name with a study number. All participants will provide written informed consent and may withdraw from the study at any time without affecting the medical care they receive from the clinical team. There will be no immediate benefits to research participants by taking part in the study. However, when the study results are known and if the POCTs are found to be acceptable in terms of accuracy, patients may benefit from being able to access a validated dual POCT available to screen simultaneously for HIV and syphilis and receive treatment if indicated.

Evaluation results will be published in a WHO report for member states and posted on the WHO website. Local and global findings of the HIV/syphilis POCT validation study will be presented during national/international scientific meetings and will be published in peer-reviewed international journals. A further accessible community report will be developed for participants interested in the study and other interested parties.

Discussion

There has been rapid development during the last decade to develop new diagnostic tools that can be administered at the point-of-care and that are Affordable, Sensitive, Specific, User-friendly, Rapid, Equipment-free, and Deliverable (ASSURED).^{20 21} To date, a rich pipeline of promising diagnostic products is emerging although some products are stalled early in the development process, and few have been evaluated robustly in real-life settings. This has arguably not only led to a lag in the integrated scale-up of POCTs in public health strategic planning but also meant that there is still no formal WHO guidance and recommendations available regarding the placement of these new tools in the diagnostic pathway of HIV/STIs.

This paper describes the master protocol of the ProSPeRo study, to conduct a clinic-based evaluation assessing the performance and acceptability of two dual POCTs for the screening of HIV and syphilis in MSM, SW and PW. The study will offer a potentially important contribution to the WHO Global Health Sector Strategy on STIs (2016–2021) as well as the Sustainable Development Goals (particularly goal 3). Specifically, the ProSPeRo findings and publications (expected from 2022) will contribute to the evidence needed to develop the guidance for WHO Member States and other relevant public health institutions on STI diagnostic tests that can be used at the point-of-care. The study will also contribute to the development of a high-quality standardised evaluation approach for POCTs in laboratory and real-world settings. A robust evidence-based framework has been highlighted as a priority for the advancement of development and global uptake of POCTs for STIs.⁷ Longer term, the findings of ProSPeRo may contribute to a gradual shift towards the decentralisation of health systems as STI POCTs within national STI programmes are implemented further and scaled up. Such decentralisation can have considerable advantages regarding accessibility of services and follow-up as well as relieving pressures on overstretched health systems, particularly in terms of personnel, infrastructure and ensuring quality of service.