Minding the Gap: An approach to determine critical drivers in the development of Point of Care diagnostics

Abstract

Introduction

A point of care test (POCT) for Chlamydia trachomatis detection is an urgent public health need. Technology advances in diagnostics have made solutions possible. Yet no reliable POCT exist. Our goal was to address the gap between chlamydia POCT needs and successful POCT development by determining which characteristics of POCT tests are most critical and if any flexibility in the attributes assigned those characteristics exist between technology developer and end user.

Methods

We employed a process known as WALEX (Warfare Analysis Laboratory Exercise) in combination with Design of Experiment (DOE) methodology using discrete choice experiments (DCE), to describe the attributes of the most realistic, rather than the most ideal POCT. The WALEX was conducted as interactive oral and simultaneous electronic discussion among experts with differing expertise, but linked by a common interest in development of a chlamydia POCT.

Results

Our studies demonstrated which features of the ideal chlamydia POCT were considered critical to test acceptance by users and which were open to negotiation. In particular, end users were more lenient on the requirement for the fastest ideal test and the lowest one time instrument costs, if the requirement for higher throughput, lowest cost and vaginal sample source collection were preserved. DOE methods used in forced choice question design provided confirmation of opinions derived from oral and electronic WALEX comments

Conclusions

The WALEX in combination with DCE helped us achieve our goal in identifying the gaps in the chlamydia POCT and determining the most realistic solutions to bridge those gaps.

INTRODUCTION

Despite many years of effort and many valid attempts to develop point of care tests (POCT), there are very few marketed POCT for sexually transmitted infections (STIs) available today. ^{1, 2} This begs the question why there still not a POCT available for many of our common STIs especially when there are such technological advances in other fields of medicine? Why do tests developed for bacterial infections such as Streptococcal pharyngitis and viruses such as HIV, fail to translate to other STIs? What prevents these rapid tests performed in the clinical laboratory from moving to the personal test market i.e. patients waiting at clinics or home testing? Identifying the key gaps in technology, which are preventing the acceptance or development of tests for the personal POCT market for STIs was the focus of our investigations. In particular, we wanted to address the apparent gap between the tests being developed by industry and the user’s expectations of a valid POCT for STIs.³

Our decision to focus on POCT needs for chlamydia diagnostics was based on the input from our large-scale focus group study and large survey of experts in POCT.^4,5 Physicians and clinical staff from public and private clinical sites ranked “on site/within visit” chlamydia testing as the most pressing need for current clinical diagnostic POCT. While the current nucleic acid amplified tests (NAATs) meet the requirements for sensitivity and specificity for chlamydia diagnostic assays, current NAATs, which are the clinically accepted assays for chlamydia diagnosis, they are too complicated, too labor intensive or not appropriately time sensitive to meet the needs for an in-clinic POCT.⁵ In our on-line survey, which invited STI experts and clinicians to “build your own test”, the characteristics selected were high sensitivity and low cost, however, within a fairly narrow range of time (5–25 minutes), there was no clear preference suggesting that time was a factor which could be granted some leeway in order to develop assays to meet the clinical need.⁵

In order to investigate this aspect further and we employed a tool originally developed to bring together experts with a variety of viewpoints in order to address a common problem known as a WALEX (Warfare Analysis Laboratory Exercise). ⁶ As part of the WALEX, anonymous on site surveys employing discrete choice experiments (DCE) were used.^7,8 The goal of these combined processes, WALEX combined with DCE, was to identify the ranges of attributes, which would promote test acceptance and identify the interdependency between them, rather than present desirable attributes as discrete independent single values, which developers need to design with rigid adherence. We wished to define the path to the best achievable design today rather than the best theoretical design for the perfect test.

METHODS

WALEX Description

The WALEX process involves facility design, analytical planning and software to enhance group interaction. As shown in Figure 1, both infrastructure, hardware, software and people are part of the design. The challenge of this process is to capture all of the diverse viewpoints and assemble them into supporting documentation. The Electronic Seminar Support (ESS) System is a suite of software tools that supports collaboration by networking participants through laptop computers. The physical configuration of warfare analysis laboratory (WAL) includes the main seminar area, electronic support of collaborative discussions, easy availability of technical and analytic data, integrated audio and visual infrastructure, and advanced modeling and simulation capabilities. During briefings and moderated discussions, participants can enter comments that can be viewed by all other participants using computer groupware. If necessary, they can enter their comments anonymously. With a large group of participants in this type of setting, the oral debate can range across issues unevenly, leaving some incompletely examined. Some participants dominate discussions by virtue of their authority or personality. Often sidebar conversations take place between certain participants, distracting from the main issue. The ESS can overcome these weaknesses of large group discussion by promoting the exchange of opinions and assessments, with a candor uninfluenced by rank or organizational hierarchy through anonymous comments. The electronic discussion (e-discussion) promotes candid audience input regardless of the participant’s position, personality, or verbal communication skills. It also permits electronic surveying and voting of the audience; and provides sophisticated decision analysis tools designed to quantify subjective assessments from the participants.

Figure 1.

Overview of WALEX process. Schematic of WALEX design (top right) and facility design (lower right). Flow chart of WALEX components interactions and steps (left).

Reproduced with permission from John Nolan.⁶

The WALEX process addressing chlamydia POCT development was initiated by establishing the parameters of the problem to be addressed related to POCT for STIs. The Center for POCT STI experts were interviewed by WALEX team members with regards to the current standards of STI POCT development. Current standards were discussed and specific gaps were identified in the design or in the attributes of specific tests. In addition, information from focus groups organized by Center was used to identify areas in which there perceived gaps in the current need.⁴ These issues were used to design the overall problem by asking participants to identify the most critical characteristics in each of the areas for a successful STI POCT. Based on the needs assessments, from the focus groups chlamydia had been selected as the most important STI to target for development of an STI POCT and was used as the subject of the WALEX.

Participants and Instructions

Participant recruitment for the WALEX was by invitation only and names of 30 participants were recommended by members of the Center for POCT STD including clinicians, laboratory professionals, and regulatory officials, who had sufficient expertise in STIs, experts in engineering sensor designs, or personnel, who interacted with the FDA. One company currently developing POCT for chlamydia was also invited to the WALEX to provide an additional perspective from a commercial developer’s viewpoint. Prior to the WALEX, all participants were sent a reading list with publications grouped into 4 subject areas: the WALEX process, STI characteristics and needs assessments, POCT design needs, and POCT diagnostic technology.

The WALEX was initiated with short instructional briefing and overview of the issues and problems, which needed to be addressed during the 4 hour session. A moderator presented primary topic areas: Characteristics of POCT/Technical Design, Manufacturing Transition, Risk, and Market Acceptance. Each area was supported by a list of questions for group discussion. All discussions and e-discussions were recorded for review and subsequent summation.

WALEX Survey Design and Analysis

Prior to the WALEX, the survey questions were prepared based on interviews with the Center staff and focus group results. These inputs were used to design a parameter list and the ranges at which a POCT test should operate. Regulatory guidelines for test approval were reviewed and used to determine which parameters were not optional and had minimal federal standards for test approval. This information was used to design a scenario for the choice experiments. Since federal regulations require that the specificity and sensitivity of a new test for STD are not defined by end users or developers, these two parameters were fixed in the scenario and not open to choice by the participants as they would be in actual practice. The assumption was that any test FDA cleared and marketed would have to meet these minimal standards and therefore, was not a matter of choice for this WALEX.

Design of Experiments (DOE) and discrete choice experiments (DCE)

DOE was used to develop the extremes of each node (parameter) to be used in the force choice questions.⁹ DCE is a subset of DOE originally used to develop market surveys. Using 2 × 2 matrix factorial design for all the attribute levels listed in Table 1, 648 test points (2³*2²*2⁴*2³*2³*2³) could be represented in as few as 16 questions. Forced choice surveys by design need to be limited to approximately 20 questions as participant fatigue is a key consideration in the use of DCE.¹⁰ Key characteristics important for a POCT for chlamydia were: sample type, assay time, sampling capacity, and assay cost (Table 1). Assay time and cost were sub-divided. Assay time divisions reflected the difference between manual sample manipulation time (hands on time) sometimes referred to as processing time and time to assay result (instrument time). Cost was subdivided into 1) disposable assay costs which are continuous costs and 2) instrument costs which are frequently onetime costs. A range for each characteristic was developed (Table 1) based on input from Center clinical experts interviewed prior to the WALEX. The final choice survey was designed using DCE to derive 16 pairs of forced choice questions (Table 2). Discrete choice experimental methodology has been described as a means to evaluate compromise.¹¹ Most of the choice pairs contained two least ideal combinations and survey participants were forced to select the better of two less desirable choices.

Table 1.

Key POCT Characteristics and their attributes as identified by STD focus groups.

Sample Type:	Hands-on Time:	Instrument Process Time:	No. Samples per Batch:	Instrument Set-up Cost:	Cost per Assay:
Vaginal Swab	5 min	5 min	1	$5000	$10
Urine	10 min	20 min	2	$10000	$25
Cervical/Urethral		1 hr	4	$20000	$50
Swab		4 hr

Table 2.

Forced Choice Selections by WALEX Survey takers

		----------------------------------Attribute Selection------------------------------
Choice Set Number	Participant Choice Frequency (%)	Sample Type	Hands-on Time	Instrument Time	Samples per Batch	Instrument Set-up Cost	Assay Cost
1	91	Cervical/Urethral Swab	5 min	20 min	1	$20000	$10.00
1	9	Cervical/Urethral Swab	10 min	5 min	1	$5000	$50.00
2	18	Cervical/Urethral Swab	10 min	20 min	4	$5000	$25.00
2	82	Cervical/Urethral Swab	10 min	4 hr	1	$20000	$10.00
3	32	Vaginal Swab	5 min	5 min	1	$5000	$50.00
3	68	Cervical/Urethral Swab	10 min	5 min	4	$5000	$50.00
4	48	Cervical/Urethral Swab	5 min	4 hr	1	$10000	$25.00
4	52	Urine	10min	1 hr	1	$10000	$10.00
5	64	Cervical/Urethral Swab	10 min	5 min	2	$5000	$10.00
5	36	Vaginal Swab	10 min	5 min	4	$20000	$50.00
6	95	Urine	10 min	5 min	1	$5000	$25.00
6	5	Cervical/Urethral Swab	5 min	1 hr	1	$10000	$25.00
7	95	Cervical/Urethral Swab	10 min	5 min	2	$10000	$25.00
7	5	Urine	5 min	5 min	4	$5000	$25.00
8	14	Cervical/Urethral Swab	5 min	5 min	4	$5000	$10.00
8	86	Cervical/Urethral Swab	10 min	20 min	2	$10000	$10.00
9	62	Urine	10 min	5 min	2	$20000	$10.00
9	38	Urine	5 min	5 min	1	$10000	$50.00
10	38	Cervical/Urethral Swab	10 min	4 hr	1	$10000	$50.00
10	62	Urine	5 min	4 hr	1	$20000	$25.00
11	29	Cervical/Urethral Swab	10 min	1 hr	2	$20000	$50.00
11	71	Vaginal Swab	10 min	4 hr	2	$5000	$10.00
12	73	Cervical/Urethral Swab	10 min	1 hr	4	$10000	$25.00
12	27	Vaginal Swab	10	20 min	2	$20000	$25.00
13	5	Vaginal Swab	5	1 hr	1	$20000	$10.00
13	95	Urine	10	1 hr	4	$20000	$50.00
14	41	Cervical/Urethral Swab	5	1 hr	1	$20000	$25.00
14	59	Urine	5	4 hr	2	$20000	$50.00
15	86	Urine	5	1 hr	2	$5000	$50.00
15	14	Urine	10	20 min	1	$5000	$50.00
16	14	Vaginal Swab	10	1 hr	1	$5000	$25.00
16	8	Vaginal Swab	5	20 min	2	$5000	$50.00

Data Analysis

Data were collected electronically and analyzed using JMP version 8 (SAS Institute Inc., Cary, NC) with application of Firth’s unbiased estimator. The analysis was conducted by applying Firth’s unbiased estimator available in JMP software and the results are shown in Figure 2.¹² Choice modeling, a form of conditional logistic regression, is a method to use a linear model to model choices based on response attributes and not solely upon subject characteristics.¹³ The Effect of Likelihood is a graphical tool for ranking causes from most significant (Low probability relative to Chi square value) to least significant (High probability relative to Chi square) The Effect Likelihood Ratio Test, similar to a Pareto chart addresses the factors which have the greatest impact on outcome. In this test, however, the null hypothesis states that the attribute or variable has no effect on the probability of success of a POCT.

Figure 2.

Analysis by Firth Unbiased Estimator of the Overall Choices:

Prediction Profiler is a JMP graphical representation of utility and allows one to look variation for each attribute as an independent and interdependent variable.¹² To address the interdependence of each attribute and the effect of compromise, analysis of survey data was evaluated based on its desirability as independent and dependent conditions. Desirability as defined by JMP software is a technique by which multiple attributes can be optimized to obtain the best characteristics for as many attributes simultaneously. As independent variables, desirability for each individual attribute was determined at maximum desirability (1.0). For interdependent variables, desirability is allowed to vary to the optimal level for all factors even if desirability for a single factor is less than its optimal independent value.

RESULTS

Summary of WALEX discussion points

Characteristics and associated attributes for use in the choice surveys were selected from inputs obtain from focus groups conducted among health professionals.⁴ The focus of the WALEX and forced choice surveys was POCT for Chlamydia. In prior work, physicians from public and private sector laboratories selected the need for rapid tests for this organism as the top need.⁵ Under each set of choice questions, Table 1 summarizes the feedback that we received from choices made by health professionals, clinical laboratorians and companies regarding acceptable features of POCT. In each case we included the percent of choices from all participants. An example of this is reflected in the choices for instrument assay time or the time from sample to result. One participant stated that “most focus groups say, yeah, they’d like to have five minutes [for a POCT but], 20 minutes is acceptable. But we all know that people sit in emergency departments for four hours. [In addition,] “the outside requirement of four hours [gives] a lot more acceptability to different manufacturers”.

The target for populating the WALEX was 20–40 participants inclusive of test developers (research engineers and scientists), end users (clinical laboratory directors and physicians) and those who regulate test development ( experts with current and past experience with US Food and Drug Administration and sponsoring agencies). A total of 27 experts participated in the WALEX. The percentages of attendees which self identified in the categories of research engineers and research scientists, clinical laboratory directors and physicians, regulatory personnel and government sponsors or other were 26%, 18%, 33% and 22% respectively. Those that identified themselves in the category of “other” included clinical laboratorians, and industry representatives. Due to conflict of interest restrictions 5 of the government sponsors were not permitted to provide responses to the survey questions. Therefore, there is a slight difference in the makeup of the groups who participated in the WALEX and those who responded to the survey questions (33% versus 18%). The other three groups composed of researchers, physicians and other participants averaged 6 participant per group +/− 1, thus, their contributions to the survey provided nearly equivalent inputs of 31%, 23% and 27 % respectively.

Finally we evaluated the familiarity of the survey participants (n=22) with the issues associated with Chlamydia diagnostics, point of care diagnostics and diagnostic testing. For the most part, the participants stated they were familiar with use of or design of point of care tests (52%). However a much smaller percentage of participants were familiar with Chlamydia; 22% of the survey participants were specifically familiar with Chlamydia as represented by those clinicians (9%) and researchers (13%) with expertise in the treatment or study of this organism. The remaining survey participants (26%) were familiar with requirements for acceptance of diagnostics assays in clinical settings. Prior to administration of the survey questions shown in Table 2, the WALEX was conducted in order to address specific topics associated with the development and transition of successful assays to the clinical market. Shown below are the results from the four topic areas were discussed in detail: technical design of POCT, transition from research and development to testing and manufacturing, risk assessment, and market acceptance.

1. Technical design of POCT

Key factors discussed included patient acceptance of sampling location and patient willingness to self sample, sensitivity of sample types based on organism load associated with sampling location, differences in processing requirements based on site, sensitivity versus specificity and time to result (Table 3). The interdependency of these factors in the design of an appropriate test was evident from the ensuing discussion. Sampling location (vaginal, cervical, penile, urine) and type (swab versus liquid) were impacted by inferred sensitivity due to organism load and dilution factor. Urine while preferred by patient community and for the home POCT, was not ideal for Chlamydia as stated by clinical professionals. To obtain the necessary sensitivity, urine concentration or fractionated sample collection (mid stream, first burst or total) is needed which adds to the complexity, equipment and training needed to administer the POCT. Sensitivity and time to result are also impacted by administration of Chlamydia POCT as a screen or for confirmation of a diagnosis as acknowledged by participants. A two tier test composed of a rapid but less sensitive POCT followed by a slower but more sensitive confirmatory test for all first tier negative samples to be modeled after the methodology used for “streptococcal” testing was suggested. An assumption of this model was that higher false positive rates would be tolerated; this assumption was rejected by some participants due to variations in geographic prevalence rates and due to higher costs. One participant stated “using a specificity under 96% and a prevalence of 1%”resulted “in an estimated PPV of 26%” making “the value of such a test nearly irrelevant”. A final viewpoint expressed was that a “one size fits all” Chlamydia POCT may not be the correct solution.

Table 3.

Highlights From Eech Topic Area Discussed During WALEX

Topic Area: POCT Technical Requirements	Variables discussed
Patient acceptance of sampling location	Cervix, vaginal, penile, urethral and urine as sites Male preference for urine
Patient willingness to self sample	Acceptance of self collection of vaginal swabs by women 14 Acceptance and preference of urine (in a cup) by men Fewer bathrooms for collection in clinics for urine Home testing acceptance of urine
Sensitivity of sampling site	Organism load of urine, cervical swabs. Organism load associated with first urine versus entire urine stream  Glans (male) is an insensitive site to collect by swab
Sample processing	Requirements for concentration of urine: more equipment, more steps, more training  Use of self collection sponge or urine pellet on swab
Specificity and sensitivity	FDA regulates test approval  FDA states new test must perform >95% sensitivity of current NAAT Chlamydia test15  Use two test system to capture and immediately treat positives and confirm negatives by more sensitive test  Asymptomatic nature of Chlamydia is problematic in physician choice for testing and syndromic treatment model  Lower specificity tests may result in faster detection and treatment of positives  No false positives required for fast test for Chlamydia  False positive and negative rates are driven by prevalence of disease in population  Different Chlamydia disease prevalence rates need different specificity and sensitivities
Time to result	Patients want result in 5 minutes but will accept 20 minutes  ED physicians will accept up to 4 hours to fit with clinic flow  Public health laboratories want POCT to fit with clinic visit flow (<1hr)  Many patients tested for Chlamydia do not return for results
Other	Use of screening tests versus confirmatory testing  Unacceptablity of terms: screening and confirmatory tests since results are used in the same way for treatment
Topic Area: POCT Transition to Commercial Project	Variables Discussed
CLIA/FDA approval	Both FDA has oversight in categorization of CLIA tests  User community for CLIA waived and FDA approved tests have different levels of implied expertise  Clearance/approval requires new test have 95% agreement with reference test  Low complexity tests are candidates for CLIA waived tests  Reference test can be selected from any clinically approve tests performed by trained professional
Unit Size Packaging	Devices need to run tests in parallel  Batch tests for POCT require high volume and may not be appropriate for all settings  Flexibility in number of samples is needed to address variations in patient flow
Quality controls	If quality controls are built into every run method they must be run with each batch as defined by manufacturer’s instructions  CLIA inspectors not users define “a run” or required periodicity of controls  Manufacture’s definitions of periodicity do not support clinical assay work flow  Manufacturer should design controls and stability testing to support “lot level” testing not “shipment level”.
Topic Area: Risk Assessment and Cost	Variables Discussed
Cost structure	Costs per test are dependent on volume of tests produced  Cost of POCT are dependent on whether another test is required to confirm result  Costs of $30 are too high for screening test  If number of people were being tested that CDC recommends, volume of tests would increase  Current NAATs test cost approximately $15
Multiplexed test	Combining tests is a strategy for reducing cost  Multiplex tests are only good if co-prevalence rates are high  Multiplex tests costs which are three times the cost of a single assay are unacceptable
Topic Area: Market Acceptance	Variables Discussed
End users defined	POCT end users defined as physicians, clinicians and patients  Clinical laboratorians defined “the lab” (themselves) as end users in order to provide greatest cost control  CLIA waived tests would be appropriate  Internet can be used to market tests directly
Drivers for Chlamydia POCT	POCT should be available as off the shelf product “at the supermarket”  Privacy is driver for Chlamydia tests  Package inserts describe the target for use of the test (who should be tested) but do not describe where test should not be used.
Home testing	Patient focus groups indicate they would accept test with high false positive rates if they could perform themselves  Problems in home testing are seen when patients self medicate based on test results  Home sampling combined with mailers for testing at central location would be acceptable

2. Transition from research and development to testing and manufacturing

In the next part of the WALEX the regulatory requirement and preferences regarding the format in which tests are packaged was the focus (Table 3). The most important detail regarding the approval and clearance of the test by FDA is the newest requirements for medical devices inclusive of CLIA waived test to exhibit a performance of at least 95% of a currently accepted reference test.¹⁴ The ability of immunoassays to meet the 95% agreement with clinically accepted reference tests was considered to be challenging when the reference test is nucleic acid amplification tests (NAATS) noted to be the current clinical standard for Chlamydia diagnostics by several participants. Even CLIA waived POCT must be compared against performance conducted by a trained professional. Participants made the point that faster POCT with lower sensitivity may have a bigger public health impact for chlamydia than a perfect test that is not POCT. Two other issues relating to manufacturing of the final POCT design -the number of tests which could be run simultaneously and quality control frequency, were discussed in detail. While developers intended to package POCT as one test per patient, clinicians noted that parallel (multi-sample) processing would be needed for POCT to be used in the clinic. Flexibility to meet uneven flow of patients through a clinic was noted as a negative impact in batch processing for POCT. Participants stated that this would defeat the advantage of rapidity in POCT. Clinical laboratorians in the group noted with some irritation the problems encountered in the packaging guidance for quality controls noting that manufactures need to put more thought in addressing this. For some rapid test for other STIs, controls need to be run so frequently that significant amounts of time and labor are devoted to quality control rather than sample testing.

3. Risk assessment and cost

Risk in the development and marketing of POCT for Chlamydia was not seen as an issue though cost per test was seen as a significant factor in successful test development (Table 3). Cost per tests in the $20–$30 range per sample or for multiplex tests per analyte suggested by developers were summarily rejected by clinicians as unacceptable, particularly if the test was used as a screening test. It was noted by clinician laboratorians that the current NAATs were about $15 per test. Multiplexing analytes as presented by developers as a way to reduce costs was not universally accepted by clinicians attending the WALEX. One clinician noted that unless disease prevalence was similar for all the analytes, multiplex POCT would not be advantageous and could be difficult to justify for cost savings.

4. Market Acceptance

When queried the WALEX group, defined the end user of POCT as clinicians, clinical laboratories and patients indicating that the design of POCT for Chlamydia has many potential targets (Table 8). One research stated that adolescent focus groups indicated that they would accept POCT for Chlamydia even if they had higher false positive rates. Patient privacy issues were noted to be a key driver for POCT test acceptance and home testing. Participants described the potential complications of patient based home testing as a concern when patients self medicated based on the outcome of home tests. The potential for misinterpretation and operator errors were seen as negative influences on home testing for Chlamydia. Notably home sampling with mail in testing at established centers was viewed as a favorable solution.

Force Choice Survey Results

The frequency of each forced choice within the pair is shown in the second column and reflects the selection by all WALEX survey participants (Table 2). The survey data was modeled using Firth’s unbiased estimator ( JMP version 8.0, SAS Inc) (Figure 2). By posing a set of choices which carefully balances the choices in a balanced designed matrix of questions, a quantitative statistical evaluation of the respondents’ true desires is estimated using the bias-corrected maximum likelihood estimator. ¹⁶ The parameter estimate indicating the coefficients on a linear regression and the chi square probability are shown in the lower table. As can be seen in the Effect Likelihood Ratio Tests, the smallest Chi-Square > probability is assigned to the cost per assay, followed by instrument processing time, number of samples per batch (also defined in WALEX as run) and instrument set-up cost and sample type Hands-on time just barely made the cut off for significance (<0.05), while sample type was not determined to be a significant factor based on its attributes choices of 5 or 10 minutes.

The largest negative coefficient in the Parameter Estimates table is shown for urine as a sample type. This negative bias may be reflective of the WALEX discussion in which the value of urine samples may be compromised by the absence of clinical data indicative of sufficient organism load and absence of appropriate sampling preparation methods for current POCT.

As shown in Figure 2, statistically significant characteristics (<0.05), based on failure of the null hypothesis for each encompass choice variation, according the attributes listed in Table 1, are likely candidates to impact the success of a POCT test. The independent effect for the choices of each variables are often called marginals. The sum of the marginals in a particular combination of variables chosen is called the utility of that choice. A graphical representation of this utility function is called the prediction profiler, shown below as Figure 3. ¹² The prediction profile calculates the utility of a choice set by being able to able to interactively change the test variables or factors independently from each other in JMP 8. Thus, for this analysis with the following 6 characteristics at the following most desirable independent attribute levels (Figure 3A): Vaginal Swab, a 5 minute Hand-on Time, a 5 minute Instrument Process Time, 4 samples per batch, a $5000 Instrument Set-up Cost and a $10 Cost per Assay, the Utility (also called response) were calculated as:

$$\text{Utility}=5.209893-0.272061\ast (5)-0.07308899\ast (5)+5.3264127\ast (4)-0.00051979\ast (5000)-0.5308509\ast (10)=16.88$$

Figure 3.

Prediction Profiler of the STD survey with the settings for the maximum utility:

A. Independent attribute choices (no compromise)

B. Interdependent attribute choices (with compromise)

In contrast when choices were forced to select the best combination of attributes, survey participants made their selections based on compromise. Less important ideal attributes were sacrificed in order to preserve key characteristics. This compromise is represented by the prediction profiler graph shown in Figure 3B. In this case maximum utility for the attributes of Vaginal Swab, a 10 minute Hand-on Time, a 240 minute Instrument Process Time, 4 samples per batch, a $10000 Instrument Set-up Cost and a $10 Cost per Assay were selected. When desirability was maximized, there is no compromise for the desired attributes of higher throughput (4 samples per batch), lower cost per assay (< $10), and sample type (vaginal swab). In contrast survey takers were willing to compromise where acceptance of hands-on time increased to 10 minutes, overall assay time (instrument process time) increased to 240 minutes and instrument costs acceptability slightly increased to $10,000.

Essentially this tool provided a means to determine exactly where the WALEX participants drew “a line in the sand” beyond which POCTs lacking essential attributes would be rejected and provided a “broader bulls eye” for developers to target an acceptable POCT. The idea of a “broader bulls eye” is illustrated in Spider chart which provides a graphical representation of relative target size for an acceptable POCT (Figure 4). The areas within the boundaries define the ranges of attributes or trade space for assay development. Notably the configuration and area bounded in the spider chart varied by expertise supporting the WALEX quote that a “one size fits all” Chlamydia POCT may not be the correct solution.

Figure 4.

Spider Chart of Participant Preference for Chlamydia POCT:

Each characteristic is shown as a spoke on the wheel. Spokes are organized by preferences where 6 is the most preferred or most significant choice and 1 is the least preferred choice.

DISCUSSION

Our studies demonstrate one approach which can be used to 1) address the causes for the gap between technology solutions and needs and 2) find practical and acceptable solutions by partnering industry, regulatory, and health professionals as unified stakeholders in identifying the most practical path forward. Based on our research focus, we used the process of WALEX in combination with DCE to address the needs for better Chlamydia tests. The need for improved methods for detection of chlamydial infections has been well established by our Center and a number of other groups over the last decade. ^1–5 The recommendation for POCT has been described and the implementation of more robust screening has been addressed as a national priority.¹⁷

A key advantage of the WALEX is the opportunity provided participants to educate and inform each other from different expert viewpoints. For the WALEX to be successful the right number and mix of people is critical to a successful outcome. This tool can provide participants with a better understanding of the issues or restrictions faced by developers (industry) and end users (clinicians), while giving sponsors (funding agencies, payers) a voice in the development of a solution. Rigid adherence to narrow performance characteristics by the user community is likely off-putting to commercial developers and may well be a significant cause for the gap between the need for and the development of POCTs. The addition of DCE to the WALEX assisted us in capturing the process of consensus and in validating key parameters in potential technical solutions. In this way the WALEX process combined with DCE developers may view the challenge of POCT development for targets like chlamydia more favorably thus attracting more developers and promoting greater success.

Unlike focus groups, the WALEX intentionally mixes experts with differing and sometime opposing views not for the purpose of “group thinking” but for the purpose of achieving consensus on the best solution or path forward. As a result, the solutions achieved in the WALEX may be different than that of a focus group as demonstrated by our WALEX and DCE methodology. As described in our study, acceptable test characteristics could support testing times of up to 4 hours if the cost of the test could be kept under $10. In contrast, our center has conducted studies among focus groups and used surveys conducted among clinicians involved in STI testing who indicated a time window of only 20 minutes ⁵. This difference may be attributed to the interaction among WALEX members to compromise on the best practical solution rather than the ideal solution. This trade off provides developers flexibility during the development process to select the most cost effective technology options and guidance with regards to technical decisions not only in making a test but in market acceptance in an appropriate target population.

In summary, we have described a new methodology for developing compromise concessions to a problem using inputs from diverse viewpoints in the affected community. To use this tool effectively it is important to do adequate preparation before the actual WALEX and include all who may be stakeholders in the outcome. All survey participants had the basis for an in-depth understanding of the characteristics of diagnostic tests and 74% of the participants were specifically familiar with the needs for POCT or POCT for Chlamydia.

Our study has limitations. One caveat to our studies is that payers (i.e. HMOs, federal medical insurance programs and insurers) who are stakeholder in this process, were not included in our WALEX. We recognize their absence as a source of potential bias in the outcome and are attempting to include them in future WALEX.

We conclude that the WALEX process combined with forced choice surveys 1) is an effective means to rapidly evaluate technological gaps, 2) provides an opportunity for cross fertilization and better understanding of technology needs from multiple viewpoints and 3) helps focus a multidisciplinary team on providing practice solutions to technology challenges. While this WALEX focused participants on the needs of one particular POC technology under development, we will investigate the appropriateness of this tool in providing broader industry guidance and education on critical test attributes by including multiple developers in future WALEX meetings with the possibility that the best solutions may lie in the innovation among multiple technologies. It is hoped that this tool will prove to be accurate and useful in bridging the technology gaps which are inhibiting adoption of successful POCT for targets important to public health such as chlamydia.