Toward a Framework for Outcome-Based Analytical Performance Specifications: A Methodology Review of Indirect Methods for Evaluating the Impact of Measurement Uncertainty on Clinical Outcomes

Abstract

BACKGROUND

For medical tests that have a central role in clinical decision-making, current guidelines advocate outcome-based analytical performance specifications. Given that empirical (clinical trial-style) analyses are often impractical or unfeasible in this context, the ability to set such specifications is expected to rely on indirect studies to calculate the impact of test measurement uncertainty on downstream clinical, operational, and economic outcomes. Currently, however, a lack of awareness and guidance concerning available alternative indirect methods is limiting the production of outcome-based specifications. Therefore, our aim was to review available indirect methods and present an analytical framework to inform future outcome-based performance goals.

CONTENT

A methodology review consisting of database searches and extensive citation tracking was conducted to identify studies using indirect methods to incorporate or evaluate the impact of test measurement uncertainty on downstream outcomes (including clinical accuracy, clinical utility, and/or costs). Eighty-two studies were identified, most of which evaluated the impact of imprecision and/or bias on clinical accuracy. A common analytical framework underpinning the various methods was identified, consisting of 3 key steps: (a) calculation of “true” test values; (b) calculation of measured test values (incorporating uncertainty); and (c) calculation of the impact of discrepancies between (a) and (b) on specified outcomes. A summary of the methods adopted is provided, and key considerations are discussed.

CONCLUSIONS

Various approaches are available for conducting indirect assessments to inform outcome-based performance specifications. This study provides an overview of methods and key considerations to inform future studies and research in this area.

Although systematic and random variation around measured test values (henceforth, measurement uncertainty) is now routinely documented within the clinical laboratory, the potential impact of this uncertainty on downstream clinical, operational, and economic outcomes is rarely quantified. Meanwhile, evaluation of the impact of measurement uncertainty on clinical outcomes has become a recurring recommendation in protocols for determining analytical performance specifications. In its recently updated guidance, for example, the European Federation of Clinical Chemistry and Laboratory Medicine stipulates that for medical tests that “have a central role in the decision-making of a specific disease or clinical situation and where cutoff/decision limits are established,” specifications should be based on the effect of analytical performance on the clinical outcome (termed Model 1), as opposed to basing specifications on biological variation (Model 2) or state-of-the-art measurements (Model 3) (1).

Two types of studies are suggested to inform specifications under Model 1: (a) direct outcome studies (i.e., analyses based solely on empirical data, such as randomized controlled trials evaluating the impact of varying analytical procedures on outcomes); or (b) indirect outcome studies (i.e., analyses using nonempirical approaches, such as decision analytic modeling, to determine the impact of varying procedures on outcomes) (2). Because (a) is often unfeasible or impractical owing to ethical, financial, and time constraints associated with robust end-to-end test outcome studies, the indirect methods of (b) are expected to play the dominant role in this context (3).

Despite general agreement that outcome-based specifications provide the best mechanism to ensure tests best serve patients' needs, studies in this area remain uncommon. A primary reason often cited for this concerns the inherent difficulties in conducting direct outcomes studies (1, 3). It is likely, however, that a lack of awareness and specific guidance concerning alternative indirect methods that may be used is also a key limiting factor. Therefore, the aim of this study was to review methodological approaches used in previous indirect assessments and outline an analytical framework to inform future outcome-based performance specifications.

Materials and Methods

A literature search was conducted in November 2017 across 4 databases [Ovid Medline^®, Embase, Web of Science (core collection), and BIOSIS Citation Index] and covering a 10-year publication period (2008 to November 2017). The search was subsequently updated in 2019 (covering the period 2008 to March 2019). The search strategy (provided in the Appendix in the Data Supplement that accompanies the online version of this article at http://www.clinchem.org/content/vol65/issue11) combined key terms relating to (a) tests, (b) measurement uncertainty, and (c) simulation/methodology. From those studies identified via the database searches, subsequent citation tracking (including extensive backward and forward tracking) was conducted to identify additional studies published on any date (i.e., including studies published before 2008).

Studies were included if they met the inclusion criteria shown in Table 1. Studies were required to include an assessment of downstream outcomes, including clinical accuracy (the ability of a test to distinguish between patients with and without a specified condition, or identify a change in condition), clinical utility (the ability of a test to affect healthcare management decisions or patient health outcomes), and/or cost-effectiveness (the ability of a test to produce an efficient impact on health outcomes in relation to cost). Note that studies using indirect methods at any stage of the analysis were eligible for inclusion; this means, for example, that several method comparison studies (an essentially empirical study design) were nevertheless included in cases when an indirect method was subsequently used to assess the impact of identified measurement discrepancies on outcomes.

Table 1.

Review inclusion criteria.

Population	Any human population with any indication
Intervention	In vitro test (excluding imaging) or any kind of medical device used for the purpose of screening, diagnosis, prognosis, monitoring, or predicting treatment response
Comparator	Any
Outcomes	(a) Clinical accuracy, e.g.,
	Diagnostic sensitivity and/or specificity
	Positive/negative predictive values
	ROC curve/AUCa analysis
	Relative risks
	Likelihood ratios
	(b) Clinical utility
	Impact on treatment management decisions
	Impact on patient health outcomes
	(c) Costs
	(d) Cost-effectiveness
Method	Analysis includes indirect methods (i.e., excluding purely empirical analyses) to incorporate or assess the impact of ≥1 components of measurement uncertainty (below) on ≥1 outcomes (above):
	Bias (e.g., calibration or method bias)
	Imprecision (e.g., repeatability, within-laboratory or between-laboratory imprecision)
	Preanalytical or analytical effects
	Summary metrics [e.g., total error (TE) or uncertainty of measurement (UM)]
Study type	Full paper relating to an original study
Language	Full text in English
Year of publication	Database search: January 2008 to March 2019 Citation tracking: any date

^aAUC, area under the curve.

All screening (including initial title/abstract screening, full text screening, and citation tracking) was conducted by the primary reviewer (AS). A data extraction form was developed (including items on key study, test, and method details) and piloted on the first 10% of included studies. Subsequent full data extraction of included studies was conducted by the primary reviewer and double checked by 1 of 4 secondary reviewers (BS, MM, CH, and PH). Regular meetings with all authors were conducted to review the ongoing study findings and resolve (via group consensus) any inclusion and/or extraction uncertainties.

Results

STUDY CHARACTERISTICS

A total of 82 studies were identified (see Fig. 1). Regarding data extraction checking, 35 papers (43%) were checked by BS, 16 (20%) by CH, 16 (20%) by MM, and 15 (18%) by PH. Agreement between reviewers across extraction items was >99%.

Fig. 1. PRISMA flow diagram of included studies.

Study characteristics are summarized in Table 2, and details of measurement uncertainty components and test outcomes evaluated are provided in Table 3. Most studies focused on evaluating tests or devices used for the purposes of monitoring, diagnosis, and/or screening across 4 key disease areas: diabetes or glycemic control, cardiovascular diseases, cancer, and metabolic or endocrine disorders. Imprecision was most commonly addressed, followed by bias and total error, and studies primarily evaluated clinical accuracy outcomes.

Table 2.

Study characteristics.

	Number	Percentage
Year of publication
Pre-2008 (identified via citation tracking alone)	25	30
2008–2009	3	4
2010–2011	7	9
2012–2013	9	11
2014–2015	18	22
2016–2017	13	16
2018–2019	7	9
Clinical areaa
Diabetes and glycemic control	43	52
Cardiovascular diseases	17	21
Cancer	10	12
Metabolic and endocrine disorders	8	10
Kidney disorders	3	4
Prenatal screening	3	4
Noise-induced hearing loss	2	2
Role of testa
Monitoring	44	54
Diagnosis	24	29
Screening	11	13
Prognosis	7	9

^aSeveral studies included a test or tests used in multiple clinical areas or roles (hence, total percentages under these categories sum to >100%).

Table 3.

Components of measurement uncertainty included and test outcomes assessed.

	Number	Percentage
Component(s) of measurement uncertainty includeda
Imprecision
Analytical	31	38
Preanalytical/combined preanalytical and analytical	8	10
Nonspecific	11	13
Total	50	61
Bias
Analytical	18	22
Calibration bias	9	11
Nonspecific	9	11
Preanalytical/combined preanalytical and analytical	2	2
Between-method bias	1	1
Total	39	48
Total error
Method comparison study	18	22
EQA study	2	2
Other	6	7
Total	26	32
Biological variation included?
Yes: included as a separate element	13	16
Yes: combined with imprecision	5	6
Total	18	22
Primary test outcome assesseda
Clinical accuracy	45	55
Clinical utility
Impact on treatment management	23	28
Impact on health outcomes	13	16
Costs	7	9
Cost-effectiveness	2	2

^aSeveral studies included multiple components of measurement uncertainty or assessed multiple test outcomes (hence, total percentages under these categories sum to >100%).

AIM OF ANALYSES

Most studies were conducted with the objective of: (a) determining/informing analytical performance specifications (4–22), (b) exploring the impact of uncertainty allowed by current performance specifications (23–34), or (c) evaluating the potential impact of measurement uncertainty on outcomes (without explicitly defining specifications) (35–78). A final group of studies consisted of “incidental” analyses, in which the impact of measurement uncertainty on outcomes was incorporated within the analysis but was not part of the primary study aim (79–85).

METHODOLOGY FRAMEWORK

Based on the included studies, a common analytical framework underpinning the various approaches to evaluating the impact of measurement uncertainty on outcomes was identified. This framework consists of 3 key steps: (a) calculation of “true” test values; (b) calculation of measured test values (i.e., incorporating measurement uncertainty); and (c) calculation of the impact of discrepancies between (a) and (b) on the outcome(s) under consideration. An outline of the various methods adopted within this framework is provided below and summarized in Fig. 2. A summary table detailing the methods used in each individual study is provided in Table 1 of the online Data Supplement.

Fig. 2. Summary box outlining the 3-step analytical framework, primary methods identified for each step in the framework, and key questions for consideration in future analyses.

Step 1: calculation of “true” test values.

Calculation of “true” test values was based either on empirical data values (5, 7, 9–11, 18, 21, 26, 30–32, 34–37, 39–42, 45, 49–53, 56–58, 60, 61, 64, 66–69, 71, 74, 77, 78, 85) and/or simulated values (4–6, 8, 12–17, 19, 20, 22–25, 27–29, 33, 36, 38, 43, 44, 46–48, 54, 55, 59, 62, 63, 65, 70, 72–76, 79–84).

Studies using empirical data here included (a) method comparison and external quality assessment (EQA) studies, which used indirect methods to determine the impact of discrepancies between empirical reference (i.e., “true”) test measurements and index (i.e., uncertain) test measurements on specified outcomes (e.g., using the “error grid” approach outlined in Step 3) (35, 37, 41, 42, 51, 53, 56–58, 60, 64, 66–69, 71, 75, 78); and (b) studies that derived uncertain measurements from “true” empirical data values using various (nonempirical) approaches outlined in Step 2 (5, 7, 9–11, 18, 21, 26, 30–32, 34, 36, 39, 40, 45, 48–50, 52, 61, 77, 85).

Studies using simulation methods here used a range of approaches—the simplest of which was to assume a fixed set of individual “true” values specified across the measurement range and simulate uncertainty around these values (see Step 2) (12, 16, 27, 33, 36, 38, 79, 83, 84). Although this approach does not require any simulation for the “true” measurements per se, the values here are nevertheless generated rather than using real-world data directly. An extension of this approach is to assume a uniform distribution to describe the “true” frequency distribution(s): That is, assume a constant probability of occurrence for each test value along a specified measurement range, and draw from this distribution within the simulation (14, 17, 19, 44, 55). Alternatively, the expected likelihood of test values was often modeled using gaussian (i.e., normal) or log-gaussian frequency distributions, specified using published or empirical data on the expected mean and variance of test values (4–6, 8, 13–15, 20, 46, 47, 59, 63, 65). Other infrequently adopted parameterizations included mixed gaussian distributions (54, 62), multivariate gaussian distributions (when correlations between tests are known (43), and the exponential distribution (82). Nonparametric simulation approaches were also used, based on sampling with replacement from an empirical data set (18, 30). Finally, several studies used simulation techniques (22, 23, 70, 74, 75) or used findings from previously published simulation studies (24, 25, 73, 76), but did not clearly report details regarding the calculation of “true” baseline values.

An important issue with respect to the estimation of “true” test values concerns how well the underlying data may be considered a reliable proxy for the truth. A handful of studies attempted to directly address this issue by “stripping” known measurement uncertainty from baseline “true” test values via statistical adjustment: Imprecision, for example, can be removed from the variance term of a specified gaussian/log-gaussian distribution using a reverse form of the “sum of squares rule,” whereas bias can be removed from the mean term (7–10, 13, 15, 31). In general, however, the likelihood that the adopted “true” test values would, in fact, be representative of the truth was either implicitly assumed or not discussed.

Step 2: calculation of measured test values (incorporating measurement uncertainty).

Approaches to the calculation of measured test values predominantly fell into 4 broad categories: (a) empirical assessment (35, 37, 41, 42, 51, 53, 56–58, 60, 64, 66–69, 71, 74, 78), (b) graphical assessment (5, 7, 9–11, 36), (c) computer simulation (4–6, 8, 12, 14–25, 27–31, 34, 38, 39, 44, 46, 49, 50, 52, 54, 55, 59, 61–63, 65, 70, 72–77, 79–85), or (d) regression analysis (26, 32, 43, 47).

Studies using empirical assessment here included method comparison studies (35, 37, 41, 42, 53, 56–58, 60, 64, 66–69, 71, 75, 78) and an EQA study (51), which based “true” test values on the specified reference test and measured values on the index test measurements.

An alternative method, first appearing in 1980, is based on applying hypothetical measurement uncertainty to “true” values via graphical manipulation (5, 7, 9–11, 36). This approach centers on plotting the cumulative percentage frequency of “true” values on the probit scale (x axis) as a function of “true” values on the logarithmic scale (y axis); assuming that the log-transformed data are gaussian, then in the bimodal case (for which healthy and diseased populations are modeled separately), cumulating the healthy (diseased) population from high (low) values results in 2 straight lines sloping in opposite directions for each population (i.e., forming an X on the plot). The addition of negative (positive) bias is then explored by shifting the straight lines to the left (right) on the x axis, whereas the addition of imprecision is explored by rotating each line around their mean value (i.e., broadening the 95% CI of the values on the probit scale). Given a specified cutoff threshold, the proportion of false-positive and -negative findings at a particular level of bias and imprecision can be read off directly from this plot by observing the point at which healthy/diseased populations cross the threshold line.

In response to modern computational capabilities, the graphical method has been superseded by computer simulation approaches that can accommodate more complex specifications of the measurand distribution and measurement uncertainty. The most flexible and widely adopted approach in the identified studies was based on iterative simulation, with uncertainty added on to “true” test values according to a specified error model—a function relating measured test values to baseline “true” values plus specified components of measurement uncertainty (14, 17–19, 28–30, 34, 54, 62, 79, 82–84). This method is largely attributed to the seminal 2001 study by Boyd and Bruns (14)—the first study of this kind to clearly specify the error model as a mathematical function [as opposed to earlier (4–6) and later (21–25, 44, 49, 52, 70, 72, 73, 76, 77, 80, 81, 85) studies limited to textual descriptions or indirect referencing]. An example of a typical error model is as follows:

$${\text{Test}}_{measured}={\text{Test}}_{true}+\left[{\text{Test}}_{true}*\text{N}\left(0,1\right)*\text{CV}\right]+\text{Bias}$$ (1)

where Test_true is the “true” measurement value; Test_measured is the observed test value measured with imprecision (CV%) and absolute bias (Bias); and N(0,1) is a normal distribution (mean = 0, SD = 1) applied with the CV% value to produce a spread of gaussian-distributed results around Test_true.

The error model iterative simulation approach works as follows: (a) a random draw is taken from the distribution of “true” values to generate a value for Test_true; (b) components of measurement uncertainty are applied to Test_true according to the error model formula to simulate a value for Test_measured [this may require random number draws—for example, in Eq. 1, a random draw from N(0,1) is required for the application of imprecision]; (c) points (a) and (b) are repeated (e.g., 10000 times to simulate 10000 Test_true and Test_measured values) for a given level of measurement uncertainty (e.g., CV% = 5% and Bias = 5%); and (d) points (a) to (c) are repeated for varying levels of measurement uncertainty (e.g., CV% ranging from 0% to 20% and Bias ranging from ±10% in 1% increments). This iterative process can be efficiently implemented using standard statistical software, such as Excel or R.

Rather than iteratively adding on uncertainty via error model simulation, an alternative approach is to incorporate uncertainty directly within a specified probability distribution (e.g., incorporating bias within the mean term, and imprecision within the variance term of a gaussian or log-gaussian distribution). This distribution can be applied iteratively around individual “true” values (12, 16, 18, 27, 30, 38, 46, 59, 61) or at a population level by adjusting a specified “true” population distribution to include additional uncertainty (8, 15, 31, 63, 65).

The remaining studies used regression analysis (26, 32, 43, 47), other one-off methods (12, 13, 33, 40, 45, 48), or reported insufficient details regarding simulation techniques to determine the exact method used (74, 75). Within the identified regression analyses, bias or total error was applied as a multiplicative factor to baseline measurements within a specified regression model, with the resulting impact on the regression output (e.g., likelihood ratio) explored. Details of studies using other one-off/indeterminate methods can be found in Table 1 of the online Data Supplement.

Step 3: calculation of the impact on test outcomes.

The final step is to assess the impact of deviations between “true” and measured values on the outcome(s) of interest.

Most studies focused on evaluating clinical accuracy (4–13, 15, 16, 20, 26–29, 31–33, 38, 39, 43, 45–52, 55, 59, 61–63, 65, 79–85). In this case, the calculation is generally straightforward: The rate of change in miscategorizations (e.g., false-positive/negative diagnoses) is determined according to the change in the proportion of measured values pushed above or below the given test cutoff threshold(s) used to define disease status or inform treatment decisions, compared with the “true” value classifications. This was the typical approach taken in studies using the graphical and simulation approaches outlined in Step 2, for example.

Several studies evaluated the impact of measurement uncertainty on treatment management decisions (14, 18, 21, 30, 35, 37, 41, 42, 51, 53, 56–58, 60, 64, 66–69, 71, 74, 75, 78). Most of these were method comparison studies that determined the impact of measurement deviations on treatment decisions using error grid analysis (35, 37, 41, 42, 53, 56–58, 60, 64, 66–69, 71, 74, 78). Two studies similarly used the error grid approach but used simulated (rather than empirical) reference and index test measurements (74, 75). First developed in the 1980s, the original error grid aimed to evaluate the potential impact of measurement discrepancies between self-monitoring blood glucose devices and laboratory reference measurements in terms of insulin dosing errors (35). Using a scatter plot of reference vs index test measurements, the plot was divided into 5 error grid “zones” according to assumed severity of associated dosing errors (from zone A = clinically accurate results, to zone E = erroneous results leading to dangerous failure to detect and treat). More recently, studies have attempted to build on this approach, for example, by expanding on the small sample of experts used to define the initial error grid (37, 74, 75), accounting for temporal aspects of measurement (41), or applying the same methodology to alternative clinical settings (64).

Others have attempted to incorporate the impact of measurement uncertainty on patient health outcomes (17, 19, 22, 23, 44, 54, 70, 72). All of these studies related to evaluations of monitoring devices for glycemic control, in which health outcomes such as hypoglycemia and hyperglycemia were determined using decision analytic models based around sequential glucose measurements (incorporating measurement uncertainty via the error model simulation approach, for example). Combined with data on insulin dose administrations (resulting from measured values) and additional factors such as patient insulin sensitivity and gluconeogenesis, these models were used to track patients' response to administered doses and resulting health outcomes.

Nine final studies included an assessment of costs or cost-effectiveness (7, 8, 11, 24, 25, 40, 73, 76, 77). Four were based on a simple assignment of expected costs of misdiagnoses to rates of false-positive/negative results (7, 8, 11) or expected costs of adverse events applied to simulated health outcomes data (77). One study included a more comprehensive costing analysis, in which the potential financial implications of calibration bias in serum calcium testing were explored (40). The remaining 4 studies all used the previous work of Breton and Kovatchev (23), in which the impact of reduced glucose meter imprecision on glycemic events was simulated using a published simulation platform. Two studies constructed simple cost-consequence decision models, combining the Breton and Kovatchev (23) findings with data on patient population numbers, glucose meter costs, and the rate of myocardial infarctions resulting from glycemic outcomes, to estimate annual cost savings associated with improved meter precision (73, 76). Two more recent studies conducted full cost-effectiveness analyses using cohort Markov (i.e., state-transition) models to link the data on improved glycemic control and reduced glycemic event rates with data on diabetes complication rates, patient health-related quality of life, and health service costs (24, 25). Using these models, the authors were able to estimate the incremental cost per additional quality-adjusted life-year associated with reduced device error.

Discussion

REVIEW FINDINGS

Based on our methodology review findings, a 3-step analytical framework underpinning the various approaches to determining the impact of measurement uncertainty on outcomes was identified (see Fig. 2). Key points for consideration within this framework are discussed below.

With regard to Step 1 (calculation of “true” test values), the primary advantage of using either empirical data or informed parametric distributions is that, by accounting for the expected frequency of values, population-level conclusions (such as analytical performance specifications) may be derived. In contrast, the primary drawback of the fixed-values approach, and by extension the uniform distribution approach (assuming this is not a realistic parameterization), is that population-level conclusions cannot be derived. Nevertheless, such approaches may be useful for exploring the impact of measurement uncertainty in specific scenarios—for example, to explore the impact of uncertainty on test values close to the test cutoff threshold.

A question that must be considered when using either empirical or parametric distributions is how well the underlying data may be considered to represent the truth. If values used to inform the “true” distributions are themselves subject to measurement uncertainty (even if this uncertainty is expected to be small), then all subsequent analyses may be affected by this confounding factor, and care should be taken when asserting absolute maximum bounds for imprecision and bias. A handful of studies did attempt to address this issue using statistical adjustment methods; however, this approach depends on having reliable information on the expected measurement uncertainty contained in the baseline “true” measurement values and can be used only when modelling test values as parametric distributions (7–10, 13, 15, 31).

A second consideration in the adoption of parametric distributions concerns the appropriateness of the assumed parametric form. Although a few studies provided some form of justification for the parametric choice (e.g., using the Kolmogorov–Smirnov test for normality), a common implicit assumption was that data would be likely to be gaussian or log-gaussian distributed. The validity of this assumption is not always clear, however.

Within Step 2 (calculation of measured test values), computer simulation methods offer the most flexible approach for exploring alternative specifications and levels of measurement uncertainty. In the context of setting performance goals, studies based on method comparison analyses are of limited use given the fact that alternative levels of measurement uncertainty cannot be efficiently explored, and analyses using the graphical method suffer from the issue that nongaussian parameterizations or nonconstant/nonlinear specifications of bias or imprecision cannot be accommodated. The error model approach is particularly useful in this respect. Although the example formula provided in Eq. 1 specifies 1 CV% element representing total imprecision, additional elements of imprecision (e.g., preanalytical, analytical, and biological) may be separately specified. Alternative characterizations of imprecision may also be defined: for example, using (a) a fixed SD, (b) different SD/CV values for different sections of the measurement range, or (c) imprecision defined as a linear/nonlinear function of Test_true. Similarly, bias may also be characterized in alternative ways.

With regard to Step 3 (calculation of the impact on outcomes), a further advantage of the simulation approach is that, by sampling over a range of bias and imprecision values, the joint impact of these components on outcomes can be clearly explored. In particular, several studies used contour plots to present their findings (14–19, 21, 30, 34, 62): An example, provided in Fig. 3, represents a hypothetical case in which bias and imprecision have been applied (according to Eq. 1) to normally distributed healthy [N (30,5)] and diseased [N (60,10)] populations. The plotted lines indicate at which values of imprecision and bias a given value of clinical sensitivity/specificity is maintained. For example, in this case, at imprecision = 0, increasing positive bias decreases clinical specificity and increases clinical sensitivity, whereas negative bias has the opposite effect. Based on this plot, we expand on the typical contour plot to show how maximum allowable bounds for imprecision and bias can be identified according to specified minimum requirements for clinical accuracy. Suppose, for example, that we require sensitivity to remain >90% and specificity to remain >80% to maintain expected health utility gains. The region of acceptable analytical bias and imprecision values for this specification of clinical accuracy is illustrated by the shaded region of the contour plot—from this we can see that if bias is zero, we can tolerate up to 20% imprecision, whereas if imprecision is zero, we can tolerate −8 to +6 U of absolute bias. Plots such as this one offer an effective means of highlighting acceptable bounds for measurement uncertainty.

Fig. 3. Example contour plot based on simulations using the error model approach (adding increasing magnitudes of bias and imprecision onto assumed “true” measurand values).

The contour lines indicate what level of clinical accuracy is achieved across the range of bias and imprecision inputs explored: Varying sensitivity levels as a function of bias and imprecision are represented by the solid contour lines, whereas varying specificity levels are represented by the dashed contour lines. The gray region represents an “acceptability region” for bias and imprecision, which maintains sensitivity ≥90% and specificity ≥80%.

Although most studies focused on the intermediate outcome of clinical accuracy, ideally technologies should be evaluated in terms of their influence on “end point” outcomes, i.e., health (clinical utility), operational, and/or cost-effectiveness outcomes. Several of the identified studies used analytic decision-modeling techniques to determine the impact of measurement uncertainty on health outcomes: Although these all related to the context of glycemic control devices, decision models can feasibly be used to explore any clinical pathway of interest, subject to data availability. Within the field of health technology assessment, for example, decision models are routinely used to evaluate the expected clinical utility and cost-effectiveness of novel tests by linking data on disease prevalence and test clinical accuracy (e.g., the proportion of correct and incorrect diagnoses) with downstream data on the expected change in patient treatment, patient compliance to treatment, and treatment effectiveness (often referred to as the “linked-evidence approach”) (86–88). Although this approach is more resource- and data-intensive, and care must be taken to ensure that the model structure appropriately reflects key aspects of the clinical pathway, it nevertheless has the advantage of explicitly capturing the impact of additional parameters (e.g., treatment effectiveness) on end point outcomes (which may not always produce expected or intuitive results), and uncertainty around the exact values of these parameters can be quantitatively characterized in the model framework (89). We identified 2 recent studies that used health-economic models to estimate the cost-effectiveness of improved analytical performance (24, 25). These studies explored a limited set of fixed imprecision levels relating to preexisting performance specifications: Future studies could extend this methodology to explore a broader range of measurement uncertainty values (e.g., by linking error-model simulations with the downstream health-economic modeling) and derive de novo performance specification based on maintaining or optimizing cost-utility and cost-effectiveness outcomes.

STRENGTHS AND LIMITATIONS

In light of the sustained international focus on outcome-based analytical performance specifications, it is expected that the indirect approaches outlined in this study will become increasingly important. The analytical framework presented in this study provides a useful starting point to inform future studies in this area by clearly outlining available methods in sufficient detail to enable practical implementation and highlighting possible advantages and limitations to consider under each approach. Whereas previous studies have provided commentaries and general reviews of various approaches to setting analytical performance specifications (3, 90, 91), this is the first methodology review to focus specifically on indirect methods for setting outcome-based performance specifications.

As a methodology review, the aim of this study was not to systematically identify all evidence, but rather to ensure that key examples of relevant methods were identified. Although we attempted to make the database search as sensitive as possible, because of the vast volume of literature in this area we necessarily had to focus the search strategy by (a) concentrating on terms related to in vitro biomarkers, (b) including a filter for simulation and methodology terms, and (c) restricting the initial database search period to 10 years. Extensive citation tracking was additionally conducted, extending into preceding years, to ensure that seminal papers informing modern practices would be identified in addition to current state-of-the-art methodology. Although we believe that this 2-stage strategy will have captured key methodologies, not all relevant material relating to each method will have been identified; therefore, we cannot draw definitive conclusions regarding the frequency that each method has been used. Nevertheless, we believe our findings provide a valuable overview of indirect study methods and an informative starting point for future studies in this area.