More Than Just Accuracy: A Novel Method to Incorporate Multiple Test Attributes in Evaluating Diagnostic Tests Including Point of Care Tests

Abstract

Current frameworks for evaluating diagnostic tests are constrained by a focus on diagnostic accuracy, and assume that all aspects of the testing process and test attributes are discrete and equally important. Determining the balance between the benefits and harms associated with new or existing tests has been overlooked. Yet, this is critically important information for stakeholders involved in developing, testing, and implementing tests. This is particularly important for point of care tests (POCTs) where tradeoffs exist between numerous aspects of the testing process and test attributes. We developed a new model that multiple stakeholders (e.g., clinicians, patients, researchers, test developers, industry, regulators, and health care funders) can use to visualize the multiple attributes of tests, the interactions that occur between these attributes, and their impacts on health outcomes. We use multiple examples to illustrate interactions between test attributes (test availability, test experience, and test results) and outcomes, including several POCTs. The model could be used to prioritize research and development efforts, and inform regulatory submissions for new diagnostics. It could potentially provide a way to incorporate the relative weights that various subgroups or clinical settings might place on different test attributes. Our model provides a novel way that multiple stakeholders can use to visualize test attributes, their interactions, and impacts on individual and population outcomes. We anticipate that this will facilitate more informed decision making around diagnostic tests.

Proposed new conceptual model for incorporating test availability, test results, and test experience, as well as the interactions between these three elements and their effects on the diagnostic pathway and subsequent individual patient and population outcomes.

I. Introduction

New diagnostic technology has led to many advances in modern health care in terms of the precision, speed, and scope of diagnoses, and the global diagnostics industry is growing rapidly [1]. Although tests represent only 5% of the costs of health care, they inform the majority of subsequent health care decision-making and thus have a major impact on overall costs [2]. Several frameworks for evaluating diagnostic tests have been proposed, most include four phases of technical efficacy, diagnostic accuracy, diagnostic efficacy, and patient outcome [3]. Others propose five components of test delivery, test results, diagnostic decisions, management decisions, and treatment implementation. [4]. Yet none have become universally understood or adopted in a similar way to phases of new pharmaceutical development. Accuracy in particular is the ‘center of attention’ at many levels. Indeed, most test regulatory bodies (e.g. the US Food and Drug Administration or FDA) focus on test accuracy, the only reporting guideline for diagnostic studies is limited to accuracy (Standards for Reporting of Studies of Diagnostic Accuracy, or STARD), and most guidance for patients and clinicians is related to accuracy. In contrast, impacts of tests on patient outcomes are rarely evaluated or reported [5], and typically include impact on clinical management while ignoring effects on other aspects of patient well being [6]. Finally, most frameworks assume that all aspects of the testing process and attributes of tests are discrete and equally important.

Quantifying the balance between benefits and risks of interventions is well established. Typically, this involves measuring the clinical effectiveness and harms associated with a new intervention, and assessing their relative importance [7]. However, this approach has largely been overlooked in diagnostic testing. In reality, test attributes have different value depending on the viewpoint. For example, test developers often need to balance investment in enhancing test accuracy over the increased test costs and the impact on sales and market share that this could incur. Clinicians on the other hand, might need to weigh up the advantages of improving test access (for example a faster or easier test) over potentially lower accuracy. Finally, patients might want to balance the advantages of a more accurate yet more invasive test, over one that is less accurate but less invasive.

A shift to a more patient-centered approach to device regulatory approval is now emerging, and the FDA recently proposed a framework for incorporating information on patient preferences into regulatory assessments of medical technology including diagnostics [8], [9]. In this article we propose a novel approach which potentially allows multiple stakeholder preferences to be incorporated into the assessment of benefits and risks of diagnostic tests. We hypothesized that the interactions that occur between test attributes are currently overlooked in test evaluation, yet are critical in order to fully appraise the value of diagnostic tests.

II. Methods and Procedures

We conducted a scoping search of PubMed to identify published models or frameworks of diagnostic test evaluation, reviewed selected textbooks on diagnostic test evaluation, reviewed methods guidelines from the Agency for Health Care Research and Quality, the Patient Centered Outcomes Research Institute, and Cochrane Collaboration, as well as a focused search for diagnostics methodology guidance from key experts in the field. Using a group process, the authors listed all attributes of diagnostic tests (including screening, monitoring, prognostic tests) based on existing literature and authors’ experience as developers, evaluators, implementers, and users of diagnostic tests. We categorized different attributes of diagnostic tests into three broad groups: the availability of a test, the experience that patients have when they are tested, and the results that a test produces (Table I). We used an iterative process to develop a model that visualized the balance between test attributes and their impact on health outcomes. The model went through several iterations, with input from colleagues involved in multiple aspects of test development, research, and implementation.

TABLE 1. Key Attributes of Diagnostic Tests.

Test availability

•Cost or reimbursement •Storage, technical feasibility (e.g., climate control, shelf life, electricity, bench space) •Knowledge/expertise required to perform and interpret test •Time taken to conduct test •Need for and supply of reagents/supplies •Maintenance, calibration, quality control

Test experience

•Preparation required (e.g., fasting, bowel preparation •Speed of getting results •Physical effects of the test (e.g., pain, bleeding) •Psychological effects of the test itself (e.g., anxiety, improved knowledge, change in behavior

Test results

•Accuracy (e.g., sensitivity, specificity) •Precision/Reproducibility •Ability to act on result (e.g., whether the test result allows a clinical decision to be made). •Diagnostic confidence (e.g., the impact of a test on patient or clinician’s willingness to take action) •Interpretability (e.g., understanding the meaning of the test)

The final model (Fig. 1) encompasses the following key features: 1) Tests used in clinical settings involve attributes within all three of these categories (test availability, results, and experience), 2) Test attributes directly interact with, and can influence each other in positive and negative ways, and 3) Test attributes and their interactions ultimately influence actions in terms of diagnostic decision making and subsequent effects on therapeutic management, patient health outcomes (e.g. improved health outcomes, effects on emotional, cognitive or social functioning), or population-level outcomes (such as cost effectiveness).

FIGURE 1.

Model demonstrating the three main types of test attributes, the interactions between them, and impact on diagnosis, treatment decisions, and patient and population outcomes.

We used a scoping review to identify examples of test evaluations that demonstrate interactions between test attributes. We searched PubMed, Web of Science, and Google Scholar to identify examples of interactions between test attributes and/or test outcomes. The goal of the search was to find illustrative examples of these impacts, rather than to perform a systematic review. Two broad strategies were employed. The first used a list of terms that were related to test attributes. We then conducted searches regarding this attribute using varying degrees of specificity. A general search such as invasive diagnostic testing might give us a list of topics/procedures to then further investigate. For example, we might see articles cited in the search regarding patient discomfort during routine cervical smear tests, so we would then search for “pap smear” refusal pain, etc. This search would then lead us to more relevant articles that addressd our target criteria. The second strategy was to search literature based on our prior knowledge. We searched databases for specific procedures or tests that we were aware of in order to find articles that discussed the outcomes of these tests. For example, we searched for articles about HIV home tests using search terms such as HIV “home test” acceptability or HIV “home test” accuracy. For both strategies, we checked the reference list for related articles and searched for other articles that cited the original article.

In the following sections, we use examples of diagnostic test evaluations illustrate several aspects of the proposed model: i) how test attributes (availability, experience, results) interact or influence each other, ii) how these interactions can influence patient outcomes or population health outcomes, and iii) the potential complexity of multiple such interactions.

III. Results

A. Interactions Between Test Availability and Outcomes

Changes in the availability of a test can impact subsequent outcomes (Table II). A trial in Zimbabwe compared two strategies (mobile van and door-to-door testing) for improving the rates of tuberculosis case finding [10]. Both strategies led to significant declines in rates of tuberculosis, suggesting that the increased test availability improved screening rates and led to positive effects on health outcomes at a population level. Improving the availability of blood pressure monitoring devices by providing them for patients’ home use led to increased patient confidence about monitoring their own blood pressure (i.e. positively impacted their test experience), and patients felt that home readings were more accurate than those taken by their doctor (i.e. impact on test results) [11].

TABLE 2. Interactions Between Test Availability and Other Test Attributes and Outcomes.

	Test availability	Potential impact on outcomes
Mobile van for TB testing in low resource setting (10)	Increased access to TB testing by screening in community settings.	Decline in tuberculosis rates in communities using this strategy (i.e., impact on population outcomes)
Patients’ experiences of self-monitoring blood pressure and self-titration of medication. (11)	Self monitoring of blood pressure with self titration of medication	Improved patient knowledge about hypertension (i.e., impact on test experience). More accurate blood pressure results (i.e., impact on test results)
Impact of the rapid diagnosis of influenza on physician decision making (12)	Increased availability of point of care testing for influenza compared to not having this test	Physicians who were aware of the influenza test result ordered fewer tests, care was less costly, fewer antibiotics, and shorter length of stay, and treated more children with antiviral drugs (i.e., impact on patient management, patient outcomes)
User acceptability and feasibility of self-testing with HIV rapid tests (26)	Increased availability of HIV testing using self-test kits	More private testing experience (i.e., test experience better). Worse test procedure (i.e., test experience worse). High proportion invalid results (i.e., test results worse).

Improving test availability can also increase clinicians’ diagnostic confidence, thus improving therapeutic management. A trial of rapid influenza tests in a pediatric emergency department found that rapid diagnosis of influenza led to fewer tests ordered for other conditions, lower inappropriate antibiotic use, and less costly care, than when this test was not available [12]. However, increased availability does not necessarily improve test experience or test results. A study of feasibility of self-testing for HIV in Singapore found that 89% of users preferred the privacy of testing, but 85% had difficulty performing all steps of the test correctly implying that test experience was worse. More concerning was that 56% of individuals had invalid results due to incorrect test performance, suggesting that improved availability led to less accurate or valid test results [13].

B. Interactions Between Test Experience and Outcomes

Attributes of the experience of undergoing a test can have both positive and negative effects on outcomes (Table III). A study of patients’ perspectives about colorectal cancer screening found that fear and bowel preparation requirements were the most important barriers to screening (i.e. potentially worse individual and population outcomes) [14]. On the other hand, improved patient experience of a testing procedure can lead to higher test uptake or fewer complications. For example using HbA1c rather than fasting glucose as a screening test in adolescents at high risk for type 2 diabetes found that screening rates increased (albeit modestly) due to removing the inconvenience of having to fast [15]. However, in this case, improving patient experience may have negative effects on test results (i.e. lower test accuracy). For example, although oral glucose tolerance tests require fasting and up to 4 hours of multiple glucose tests, they are more sensitive than HbA1c or single fasting glucose tests [16]. A more dramatic illustration is the potential use of noninvasive measures of liver elasticity to screen for cirrhosis, with considerably fewer adverse effects (e.g. pain, bleeding), compared to liver needle biopsy (i.e. improved patient experience leading to potentially higher cirrhosis screening rates) [17]. Finally, the increased speed of obtaining cytopathology results using intra-operative frozen sections, has led to improved ability to act on results and potentially reduce adverse outcomes from the need for repeat surgery or delayed treatment [18].

TABLE 3. Interactions Between Test Experience and Other Test Attributes and Outcomes.

	Test Experience	Potential impact on outcomes
Adverse effects of colorectal cancer screening (14)	Fear and anxiety related to colonoscopy	Potentially lower uptake of colorectal cancer screening. Potentially higher rates of undiagnosed cancer and worse morbidity and mortality (i.e., worse individual and population outcomes)
Use of HbA1c increases rates of diabetes screening for at-risk adolescents in primary care settings (15)	Use of non-fasting blood test rather than the need for fasting samples	Increased screening rates for diabetes. (i.e., improved individual and population outcomes) Lower accuracy of results than oral glucose tolerance tests (i.e., negative impact on test results)
Noninvasive measures of liver elasticity rather than liver biopsy to diagnose or screen for cirrhosis (17)	Using non-invasive measures of liver elasticity causes less pain, discomfort and risks for the patient, is more rapid than liver biopsy, and is also less invasive.	Fewer complications caused by liver biopsies, which occur in up to 5% of patients. (i.e., improved individual outcomes)
Intraoperative frozen-section diagnosis of ovarian tumors (18)	Intraoperative test results can improve surgical management and increase diagnostic accuracy of non-borderline ovarian tumors during surgery	Obtaining immediate results will enable patient to receive prompt access to appropriate treatment, ideally improving patient outcomes and reducing mortality. (ie improved speed of getting test results and ability to act on results)

C. Interactions Between Test Results and Outcomes

The impact of various components of test results themselves on patient or population outcomes is perhaps the most recognizable interaction. Test results that are insufficiently accurate, or difficult to interpret or act on can have direct (usually negative) effects on other test attributes and outcomes (Table IV). Screening tests in particular are prone to negative effects from false positive results. For example, screening athletes for sudden cardiac death is enhanced when using athlete-specific ECG criteria rather than routine criteria, leading to fewer false positive tests, and potentially lower anxiety or need for more sophisticated imaging tests (i.e. higher sensitivity leading to improved patient outcomes and lower costs potentially) [19]. Yet, even small false positive rates can have profound adverse effects. One cohort study that compared women who had abnormal screening mammography with those who had normal mammograms, found that even 6 months after final diagnosis, women with false positive findings experienced similar psychosocial harms to those women who had a true diagnosis of breast cancer (i.e. inaccuracy leading to worsened patient outcomes, and negative test experiences) [20].

TABLE 4. Interactions Between Test Results and Other Test Attributes and Outcomes.

	Test Results	Potential impact on outcomes
Use of specific EKG criteria to screen athletes for cardiomyopathy or risk of sudden cardiac death, compared to regular EKG criteria (19)	Using a small subset of athlete-specific EKG parameters can improve the interpretability of EKG screening and reduce the frequency of diagnosing false positives	Improved interpretability reduces false positive screening tests in athletes, and potentially Save resources, time, and patient distress and discomfort by eliminating unnecessary further tests and treatments of false positives (ie improved patient experience)
Suboptimal accuracy of screening mammography (20)	Significant proportion of false positive mammograms in women undergoing screening	Psychological harms persisted for 3 years in those with false positive tests (i.e., negative impact on test experience)
Improved accuracy of diagnosis of malaria in febrile patients (21)	More accurate diagnosis of malaria, fewer false positives based on solely clinical diagnosis	More appropriate treatment with antimalarials (i.e., improved patient management outcomes.) Possibly lower antimalarial resistance rates in future (i.e., population outcomes). Possible difficulty maintaining test supplies and cost of tests longer term (i.e., negative effects on test availability)
Diagnosis of acute pharyngitis without a confirmatory test after a negative rapid antigen detection test (RADT) result (22)	Low diagnostic confidence in RADT tests can lead to overprescribing unnecessary antibiotics in patients	Low diagnostic confidence in result, Increased antibacterial resistance among the population as well as unnecessary treatment costs for the patient and insurer; potential antibiotic side effects (ie

In contrast, other tests can markedly improve diagnostic accuracy compared to current clinical practice. For example, malaria is difficult to diagnose based on clinical features alone, so rapid malaria tests have been introduced to increase accuracy and treatment targeting. A trial in Afghanistan compared clinical diagnosis with that using malaria rapid tests on the proportion of patients treated appropriately with antimalarials [21]. Compared to patients with a clinical diagnosis, those who had rapid testing were five times more likely to be treated appropriately for malaria. However, the improved test accuracy could potentially come at the cost of maintaining malaria test supplies and availability (e.g. test reagents, expertise to conduct the test). Indeed, other situations using POCTs with suboptimal accuracy can have negative effects on clinician behavior and health care costs. For example, rapid antigen tests are used to improve diagnostic accuracy over and above clinical features for identifying group A streptococcal pharyngitis. Yet, although most rapid antigen tests have higher sensitivity than clinical features, their false negative rates (approximately 10-20%) are sufficiently high that clinicians (and some regulators) use a back up bacterial culture test for negative rapid tests. This leads many clinicians to ignore negative rapid streptococcal antigen test results and prescribe antibiotics, while incurring costs of both rapid tests and microbiological culture (i.e. inaccuracy of test leading to worse clinical outcomes and higher test costs) [22].

D. Multiple Interactions Between Test Attributes and Outcomes

Some diagnostic test scenarios involve multiple interactions between test attributes, and with subsequent impacts on individual and population outcomes. One example is the use of point of care HbA1c tests in ambulatory care settings for screening and diagnosis of type 2 diabetes. Patients may be more satisfied with the ease of a finger stick sample compared to venous phlebotomy, or not having to fast prior to testing, or to not need repeated samples (for oral glucose tolerance tests), compared to other diabetic screening or diagnostic tests (i.e. improved test experience and test availability) (Fig. 2) [23], [24]. In addition, having a test result available during a consultation might influence patient motivation or satisfaction with their clinical interaction (i.e. improved test experience) [24]. This might lead to improved rates of screening or diagnosis (i.e. improved patient and population outcomes). However, some of the interactions between test attributes may be negative. For example HbA1c tests currently show greater variability than fasting glucose tests, and the diagnostic threshold for HbA1c is less clearly defined than that for fasting glucose or glucose tolerance tests (i.e. lower accuracy, lower interpretability) [24], [25], [28], [29]. In addition, HbA1c may be less accurate at predicting microvascular complications (i.e. lower interpretability, lower diagnostic confidence) than other tests [24]. A systematic review comparing point of care HbA1c vs. lab-based HbA1c tests on glycemic control in individuals with type 2 diabetes failed to incorporate many of the attributes, interactions, and impacts noted above, and failed to show significant differences between these two approaches to testing [30]. The complexity of these attributes, their interactions, and impact are typically not incorporated in existing approaches to evaluating or comparing diagnostic tests.

FIGURE 2.

Examples of potential multiple interactions between test attributes and outcomes, using an example of point of care testing for Hemoglobin A1c test for diabetes.

IV. Conclusions

A. Strengths and Limitations of the Model

The above examples illustrate the complexity that multiple stakeholders face when evaluating diagnostic tests, and provides evidence for our hypothesis that different test attributes not only interact, but can also influence individual and population health outcomes. We acknowledge that the evidence for the clinical test scenarios was not identified systematically, and there may be some tests where more complex and bidirectional interactions occur, whereas in others these may be single or unidirectional interactions.

Furthermore, some test attributes and interactions may differ in their relative importance to different stakeholders.

B. Proposed Uses of the Model

A key question with any proposed new framework, is ‘How could or should it be used?’. In its present form, our model provides a way to visualize all possible attributes, interactions, and impacts of diagnostic tests from multiple perspectives. This provides a global outline of ‘what research or evaluations need to be done’. For example, it could be used to direct systematic searches for existing evidence and then prioritize generation of new research or evidence for various attributes, interactions and impacts, based on the needs or viewpoints of different stakeholders. The model also potentially provides a way to incorporate the relative importance that various subgroups or clinical settings might demand for different (or even the same) diagnostic tests. For example, tests designed to be used in low income country settings might prioritize test attributes that minimize cost, equipment maintenance, water or power supply, expertise, consumables etc., as in this setting these be more critical than optimizing patient experience with the test or even test accuracy [31], [32].

In its present form, this new approach has potential relevance to multiple stakeholders:

• •Test Developers: Market realities and the commercialization landscape often dictate development efforts from the beginning of the process. Investment is required to develop tests that offer advantages (e.g. improved accuracy, higher acceptability) over existing tests. However, these development costs are likely to result in higher test costs, which could paradoxically lead to lower rates of testing (lower test availability) if fewer individuals or health care payors can afford the new test. For test developers this could mean lower return on investment from a new test. Understanding which test attributes are worth investment within the overall planned test use is critical to such R&D decisions.
• •Regulators of Tests: For regulatory or health technology assessment agencies, our model provides a framework to appraise or compare new tests using a more comprehensive method than currently exists. This builds on current efforts to incorporate patient benefit/risk information in the assessment of new devices proposed by the FDA [8], by addressing the methodologically challenging area of diagnostic tests (rather than other devices), and providing a qualitative and visual way to do this.
• •Diagnostic Test Researchers: While most clinical research (and reporting guidelines) around diagnostics focuses on technical or clinical accuracy, our model clearly illustrates the need for a far broader approach to identifying, prioritizing and prioritizing research around diagnostic tests in order to understand the relevant test attributes, their relative importance, interactions and impact from multiple perspectives.
• •Clinicians and Patients: As end-users of diagnostic tests, patients and clinicians need better information to guide choice of tests. At present there are few options for informed decisions, with the exception of some decision aids which provide tools for pictorial representations of test accuracy. Many clinicians think of tests using a paradigm of the test as an intervention, in other words ‘test and act/treat’. To some extent the elements in the outer ring of our model’s figure represent the multiple aspects of the ‘test’ component(s), while the inward facing arrows and central part of the figure represent the ‘action’ or ‘treat’ components. Having a model which outlines the various test attributes, their tradeoffs, and their impacts on patient outcomes could facilitate more informed decision making within a ‘test and act/treat’ paradigm.
• •Healthcare Provider Organizations: Information or evidence on the comparative advantages and disadvantages of different tests informs decision making at provider organizations. Again, moving away from a focus largely on accuracy, to one that involves multiple attributes that our model outlines, could provide an better framework to evaluate and compare diagnostic tests.

As it is, the model cannot be used to quantify the impact of attributes or their interactions, on clinical actions or other downstream effects. However, we believe more quantitative methods could be used to rank or weight preferences from different perspectives, drawing on methods used in risk benefit preference evaluations and health economic modelling, and attempt to quantify (or model) the relative importance (and economic impact) of attributes within particular settings.

Balancing the benefits and risks of medical interventions is not new, yet for diagnostic tests this has largely been overlooked. Our model provides a novel way to visualize test attributes, interactions, and impacts on actions arising from these, from multiple perspectives. We aim to further refine this model, evaluate how useful it is to various stakeholders, and in particular incorporate patient centered outcomes of diagnostic tests. We encourage further input from other researchers. Ultimately, we anticipate this could lead to more informed decision-making across the ‘bench to bedside’ process of needs assessment, design, development, testing and implementation of diagnostic tests.

Biographies

Matthew Thompson leads the Primary Care Innovation Lab at the University of Washington which aims to identify and implement new technology solutions in primary care settings. He is currently a Professor and the Interim Chair with the Department of Family Medicine, University of Washington, with adjunct appointments in Global Health, Mechanical Engineering and Pediatrics. He has particular research interest in advancing methods related to diagnostics and monitoring, and developing, testing and implementing point of care tests in both high, and low income settings globally.

Bernhard Weigl is currently a Principal Investigator with Intellectual Ventures/Global Good (IV/GG) and an Affiliate Professor with the Department of Bioengineering, University of Washington, where he leads work to develop highly sensitive, rapid diagnostic assays for diseases of relevance in developing countries. From 2014 to 2015, he was a Senior Technology Officer with PATH, and a Portfolio Leader for non-communicable disease diagnostics and the former Director of the Center for Point-of-Care Diagnostics for Global Health. His work focuses on diagnostic technology research and development, including strip-based immunoassays, instrument-free molecular assays, and a variety of isothermal amplification assays for low resource settings. He has led projects across the diagnostics value chain, from invention and proof of principle though product introduction and support.

Annette Fitzpatrick is currently a Research Professor of Family Medicine, Epidemiology, and Global Health with the University of Washington. She is currently an Assistant Dean for Graduate Education with the School of Public Health. She has been involved in large epidemiological studies of cardiovascular disease and dementia in the USA and is now directing her focus to implementation studies in low-resource countries to address the global burden of non-communicable diseases. She is interested in primary care interventions that can directly impact hypertension, diabetes, and cognitive decline in community-based settings.

Nicole Ide received the Master’s degree in Public Health from the University of Washington. He is currently a Global Health Researcher. Her work has been focused on non-communicable disease systems research in low resource settings. She recently completed a Fulbright Fellowship in Nepal with a focus on barriers to diabetes management and had previously conducted research on death certification using Verbal Autopsy. Her work has been focused on non-communicable disease systems research in low resource settings.