Simulation Analysis and Comparison of Point of Care Testing and Central Laboratory Testing

Abstract

Background. In response to demand for fast and efficient clinical testing, the use of point-of-care testing (POCT) has become increasingly common in the United States. However, studies of POCT implementation have found that adopting POCT may not always be advantageous relative to centralized laboratory testing. Methods. We construct a simulation model of patient flow in an outpatient care setting to evaluate tradeoffs involved in POCT implementation across multiple dimensions, comparing measures of patient outcomes in varying clinical scenarios, testing regimes, and patient conditions. Results. We find that POCT can significantly reduce clinical time for patients, as compared to traditional testing regimes, in settings where clinic and central testing areas are far apart. However, as distance from clinic to central testing area decreased, POCT advantage over central laboratory testing also decreased, in terms of time in the clinical system and estimated subsequent productivity loss. For example, testing for pneumonia resulted in an estimated average of 27.80 (central lab) versus 15.50 (POCT) total lost productive hours in a rural scenario, and an average of 14.92 (central lab) versus 15.50 (POCT) hours in a hospital-based scenario. Conclusions. Our results show that POCT can effectively reduce the average time a patient spends in the system for varying condition profiles and clinical scenarios. However, the number of total lost productive hours, a more holistic measure, is greatly affected by testing quality, where POCT often is at a disadvantage. Thus, it is important to consider factors such as clinical setting, target condition, testing costs, and test quality when selecting appropriate testing regime.

New technologies and processes are constantly being introduced into the health care delivery system. Point-of-care testing (POCT) refers to a set of medical diagnostic technologies that are applied and evaluated close to where the patient receives care.¹ This is in contrast to “traditional” testing strategies, in which test samples are sent to a central laboratory for analysis. In principle, POCT allows for on-site evaluation and faster clinical decision making. In the past decade, POCT has expanded substantially in the United States and Europe.² POCT tools now exist for evaluating conditions ranging from pregnancy to HIV, and from malaria to cancer.² While laboratory testing directly accounts for a relatively low percentage of the average hospital budget (approximately 4%), it is estimated to influence nearly two thirds of hospital activities and their associated costs.³ This is through testing’s role in making diagnoses, influencing patient flow, and affecting patient outcomes.³ Thus, improving testing procedures may have significant downstream effects on the health care delivery system as a whole. However, at present, POCT may not necessarily be advantageous in all clinical settings. A recent study involving six hospitals in the United Kingdom found that the effects of POCT on patient outcomes and test costs related to myocardial infarction varied significantly between hospitals, in some increasing the number of successful discharges and in some the opposite.⁴ The costs and benefits of POCT may vary significantly depending on the specifics of the clinical setting and target condition. Therefore, there exists a need for frameworks to quantify and evaluate the tradeoffs involved in adopting different testing regimes in varying clinical settings. In any given scenario, the costs and benefits of adopting POCT should be compared to the major alternative options, both involving processing of samples at a specialized laboratory facility: central lab testing and point-of-care sample acquisition (POCA). In central lab testing, patients physically travel to a central lab to take tests, where the samples can be directly collected and analyzed. After analysis, the test results are sent back to the clinic. In POCA, samples are acquired at the point of care and sent to the central lab for processing. This regime does not require the patient to physically travel to the central lab. Past studies on POCT have tried to identify and capture one or more dimensions of POCT that affect the health care delivery system, including test cost, test efficiency, test accuracy, and success of treatment. Compared to testing regimes involving the use of a central lab, the main advantage of POCT is shortening or eliminating steps in the testing process.⁵ Specifically, patients or samples do not need to be transported to an external laboratory for testing. Additionally, patients or test results do not need to be returned from the external laboratory. Thus, total turnaround time for test results can be significantly reduced. This may lead to further operational benefits; for example, in a hospital-based randomized trial, Renaud et al. found that POCT was associated with an approximately 45 minute decrease in time to anti-ischemic therapy for acute coronary syndrome.⁶

On the other hand, some researchers note that the POCT process might be particularly prone to errors throughout the testing process.⁷ Nichols et al. argued that systematic changes in patient management may be required for implementation of POCT to be beneficial in a cardiology/radiology setting.⁸ Errors in POCT may be due to limited resources or lack of domain knowledge of POCT among clinical practitioners, who may be less experienced in quality control and quality assurance practices than laboratory personnel.¹ O’Kane et al. suggested that quality error rates associated with POCT are also considerably higher than those associated with central lab testing.⁹ For example, in blood gas analysis, the most common error in POCT occurs as a consequence of onsite operators being unwilling to perform minor maintenance. This is usually not an issue in central lab settings with robust quality control systems.⁹ Even with adequate training, the short turnaround time in POCT can increase the risk of misdiagnosis if the results of testing contain errors. In contrast, the relatively longer time interval between result generation and report release in central lab or POCA testing may allow more opportunities for error detection. In addition, as central lab/POCA testing is often able to take advantage of economies of scale, cost per test is often higher in POCT.^3,10 For example, Lee-Lewandrowski et al. estimated an average POCT unit test cost for creatinine testing to be $10.06 in their clinic, compared to $5.32 for the estimated unit cost of a central laboratory test.¹¹

Computer simulation is one method of modeling strategic health care delivery decisions¹² and predicting their outcomes. Previous studies on health care simulation have focused on patient scheduling,^13,14 staff scheduling,¹⁵ and resource allocation.^16,17 Computer simulation focused on evaluating testing practices is more limited in the literature. Storrow et al. simulated an emergency department and quantified improvements in patient throughput and outcomes as lab turnaround time decreased from 120 to 10 minutes (modeling, e.g., the introduction of point of care testing).¹⁸ Powell et al. also simulated emergency department patient flow and found that employing POCT in some proportion of the patients significantly decreased average length of stay in the emergency department.¹⁹ However, neither of these studies attempted to quantify the tradeoffs between POCT turnaround time, test cost, and test quality characteristics; nor did they explore such tradeoffs in varying clinical settings. In this article, we attempt to address this gap using a simulation framework, building on these prior simulation studies and the empirical literature on POCT.

POCT has great potential to reduce the total amount of testing time required in a clinical setting. However, the potential test quality and cost characteristics of POCT mean that its introduction must be evaluated based on the particulars of the health care setting in which it will be used. As Nichols et al⁸ argue, POCT alone cannot guarantee improved patient outcomes. Because of this, it is important to explore and understand the impact of different testing regimes on patient throughput and outcomes in different clinical scenarios. To our knowledge, this is the first study to use a simulation-based framework to evaluate the clinical tradeoffs between POCT, POCA, and central lab testing in terms of both turnaround time and test quality, in settings that range from rural clinics to urban hospitals.

Methods

We developed a simulation model of a primary care clinic, adapted from Stahl et al.,²⁰ that models the activities of patients from their first arrival at the clinic to discharge from the clinical system, and monitors subsequent health outcomes of each patient resulting from the diagnosis and treatment received within the clinic. The portion of the model that mimics the testing process simulates five subprocesses: patient transportation, sample collection, sample transportation, sample processing, and result delivery. The model aims to identify and assess the costs and benefits of different testing regimes as perceived by providers, by patients, and by society. The health care delivery system costs include the time patients spend in the clinic and the resources they use. Costs to patients and society additionally must take into account the effectiveness of any diagnoses and treatment received at the clinic as a result of the testing. We measure this as the number of productive hours lost by patients subsequent to their discharge from the clinic, modeled as a (generally) stochastic function of their condition and treatment received, which in turn is a function of test accuracy. The more accurate the test, the more likely an effective match of treatment to condition is. We also examine the monetary considerations associated with both the tests themselves, treatments, and with the distribution of patient outcomes. Overall societal costs are then evaluated by a combined monetary measure of total productive hours lost per patient, including both time spent in the clinical system and any subsequent productive hours lost to the condition being tested for in the clinic, as well as treatment and testing costs.

Testing Strategies

We consider three different testing regimes: 1) central lab testing, where patients physically travel to a central lab for sample collection and processing; 2) POCA, where samples are acquired at the site of patient care and sent to the central lab; and 3) POCT, where diagnostic testing is performed and evaluated at the site of patient care so that neither patient travel time nor sample transportation time is involved.

Clinical Scenarios

We use community-acquired pneumonia as our primary exemplar condition where POCT might be deployed. In Appendix A, in order to examine the variations in system output in response to conditions of varying urgency, we subsequently run our model on three other conditions with differing characteristics: opiate addiction, chlamydia (a sexually transmitted disease), and cholesterol (a marker for cardiovascular risk). Conditions were chosen because of their relatively high volume in the outpatient setting.

Community-acquired pneumonia is a life-threatening disease²¹ with a 30-day mortality rate between 4% and 11%. In central lab testing, the time required to identify the causative pathogen is often relatively long (compared to POCT), due to the longer transportation times between the clinic and the laboratory, and the rigorous nucleic acid amplification and culture techniques used in central labs. Typically, this process may take several days.²² In order to examine the potential of different testing regimes to improve this performance, we model the use of POCT in a patient population exhibiting varying indicators that may be related to the target condition. We examine tradeoffs between testing regimes in three different primary care clinical scenarios: Rural, Community, and Hospital-based. The Community and Rural scenarios refer to freestanding health care clinics at some distance from the hospital where the central lab is assumed to be located. Compared to the Community scenario, Rural clinics are often far away from the central lab. At the other extreme, the Hospital-based scenario models the case where the place patients receive treatment is very close to the location of central laboratory sample analysis, as in the case where the primary care clinic is located within a hospital. Community clinics thus represent an intermediate in terms of distance from the central laboratory.

Overview of the Model

Due to the complexity of a clinic’s operations, we adopted the discrete event simulation approach and developed a primary care patient flow model using Arena (version 15.0)²³ simulation software (developed by Rockwell Automation). The primary care processes from patient arrival at the clinic to discharge from the clinical system, as well as any lingering effects of the target condition, are simulated for a stream of generated patients. Limits on available providers and testing resources mean that delays can be generated or propagated by the buildup of queues at various points in the system. A visual overview of this simulation model is shown in Figure 1. Our simulation model separates patient time in the clinic into the following six modules:

Figure 1.

Framework of the simulation model.

• Patient Arrival Module: Patients arrive at the clinic and check in.

• Waiting Room Module: Based on their indicator severity levels, patients are allocated to different rooms and wait for a nurse.

• Triage Module: Patients are checked to determine whether they need to be transferred to emergency department.

• Physical Exam Module: Patients take a physical exam and prepare to be tested if it is determined that testing is necessary. This determination is made based on the severity of indicators: more severe indicators correspond to a higher rate of testing.

• Lab Work Module: This module models the details of each testing regime being employed: POCA, POCT, or central lab testing. Depending on the sensitivity and specificity of the test and the target condition in question, each patient tests either positively or negatively for the target condition.

• Patient Departure Module: Patients receive treatment if they have tested positive for the target condition. They are then discharged from the system.

Within the Lab Work Module, patients may follow different pathways depending on the testing regime. Figure 2 shows the details of the workflows of these three testing regimes.

Figure 2.

The workflow of the three testing regimes under consideration.

Patient Types

In each scenario, patients arrive showing one of three levels of condition indicator severity. Patients in each level of indicator severity are divided into two groups: patients carrying the target condition and patients not carrying the target condition. Patients with higher indicator severity are more likely to be carrying the target condition than patients with lower indicator severity. In addition, patients with higher indicator severity are more likely to be tested for that target condition by clinicians. In other words, indicator severity is meant to be an abstract representation of characteristics of a patient associated with having a certain condition, some of which are detectable by clinical examination and judgement. These might include physical indicators available prior to testing, patient history, or family background, depending on the condition.

According to standard classification for pneumonia patients’ severity, we identify three distinct indicator severity classes of patients based on the potential mortality risk should those indicators be associated with a true case of the target condition. Thus, in the case of pneumonia, severity levels 1, 2, and 3 correspond to increasing severity of indicators (irrespective of whether the patient has the underlying target condition). As noted above, patients of different severity levels have differing probabilities of having the underlying target condition and of being tested for that condition. New patients are assigned slightly longer expected exam times than established patients, reflecting increased efficiency associated with familiarity.

Outcome Measures

We chose several patient outcome measures to evaluate testing regimes: average time in the clinical system, average number of subsequent sick days, average total lost productive hours, and average test cost per patient. Averages are computed on a per patient basis from 120 replications of an entire year of a clinic’s operations (see Appendix B.2 for details).

• Average time in the clinical system: This metric captures the average time (in hours) a patient spends within the clinical system, from arrival at the clinic to discharge from the clinical system. It does not include transportation to the clinic before arrival but does include any transportation time from the clinic to laboratory locations and back. Thus, it is meant to capture average patient time expended seeking care.

• Average number of subsequent sick days: This metric measures the number of days (subsequent to discharge from the clinical system) that a patient is not able to be productive as a result of condition associated with their clinic visit(s). Each patient is assigned sick days according to the distributions in Table 6. This metric is thus meant to capture productivity costs as a result of patient health outcomes.

• Average total lost productive hours: This metric measures the estimated total number of productive hours a patient loses as a result of their visit(s) to the clinic and any sick days subsequent to discharge from the clinical system as a result of the condition associated with their clinic visit(s). It is calculated as the average number of hours the patient spends in the clinical system plus 8 hours for every subsequent day lost due to their condition.

• Average test cost per patient: This metric measures the average cost per patient of all tests that the patient receives. Depending on indicator severity and condition, a patient can receive a combination of different tests, so average test cost can vary from patient to patient, even within the same testing regime.

Table 6.

Distributions of Subsequent Sick Days for Varying Indicator Severities, Target Condition Presence/Absence, and Treatment/No Treatment

Target Condition	Treated?	Severity	Pneumonia Test	Opiate Test	Chlamydia Test	Cholesterol Test
Yes	Yes	1	DISC(0.5,0,0.5,1)	DISC(0.9,0,0.1,1)	0	0
Yes	Yes	2	Tri(0,2,5)	Tri(0,0.5,3)	Tri(0,0.2,1)	Tri(0,0.2,0.5)
Yes	Yes	3	Tri(0,5,10)	Tri(0,1,5)	Tri(0,0.5,1)	Tri(0,0.2,1)
Yes	No	1	Tri(0,3,6)	Tri(0,2,10)	Tri(0,1,2)	Tri(0,0.2,0.5)
Yes	No	2	Tri(5,10,15)	Tri(0,3,20)	Tri(0,1.5,2.5)	Tri(0,0.5,1.5)
Yes	No	3	Tri(10,14,18)	Tri(0,5,30)	Tri(0,2,3)	Tri(0,1,5)
No	Yes or No	1	DISC(0.5,0,0.5,1)	DISC(0.95,0,0.05,1)	0	0
No	Yes or No	2	Tri(0,1,2)	Tri(0,0.5,2)	Tri(0,0.2,0.5)	0
No	Yes or No	3	Tri(0,3,6)	Tri(0,2,5)	Tri(0,0.5,0.75)	0

Simulation Model Default Inputs

In the context of our model, we refer to Rural, Community, and Hospital-based settings as clinical scenarios, and POCA, POCT, and central lab testing as testing regimes. Time costs and monetary costs are evaluated for each of these clinical scenarios and for each of these testing regimes by simulating clinic operations over the course of a year. The 120-year-long (1 work year = 240 eight-hour work-days) replications of the simulation model, each simulating clinic operations over the course of the year, were run for each combination of clinical scenario and testing regime. Then, averages of the outcomes across the 120 replications are calculated and reported. Our simulation runs require a number of input parameters, including the time spent on each subprocess, patient arrival rates, probabilities of events in the clinic (e.g., referral of a patient to the emergency department), and the time costs of various operations (e.g., the time costs of filling out the paperwork), and so on. Many of these parameters (shown in Tables 1 to 8) are specified by the distributions for these clinic characteristics and based, wherever possible, on direct observation, expert knowledge, or the health care delivery literature.

Table 1.

Distribution of Time Spent, by Testing Regime and Subprocess, in the Rural Scenario

Test Regime	Subprocess
	Patient Transportation (hr)	Sample Collection (min)	Sample Transportation (hr)	Sample Processing (min)	Result Delivery (hr)
POCT	0	Tri(20,30,40)	0	Tri(10,30,50)	0
POCA	0	Tri(20,30,40)	Tri(8,16,24)	Tri(5,15,25)	Tri(4,8,12)
Central lab	Tri(8,16,24)	Tri(20,30,40)	0	Tri(5,15,25)	Tri(2,4,6)

POCA, point-of-care sample acquisition; POCT, point-of-care testing.

Table 8.

Probability of Receiving a Particular Procedure or Procedure Combination as Part of a Pneumonia Test, by Indicator Severity (Severity Level 3 Automatically Treated)

Indicator Severity	P(Blood Test Only)	P(X-ray Test Only)	P(All Three Tests)
1	0.30	0.60	0.10
2	0.60	0.10	0.30

The times spent on each combination of testing regime (POCA, POCT, or central lab) and clinical scenario (Rural, Community, or Hospital-based) are documented in Tables 1, 2, and 3. In these tables, 1 work-day is assumed to be 8 hours long. In Tables 1, 2, 3, 4, and 6, N(µ,σ) refers to values drawn from a normal distribution with mean µ and standard deviation σ. Tri(α,β,γ) refers to values drawn from a triangular distribution with minimum at α, mode at β, and maximum at γ. DISC(α₁,β₁,α₂,β₂, . . .) refers to values drawn from a discrete distribution where outcome β₁ occurs with probability α₁, outcome β₂ occurs with probability α₂, and so on.

Table 2.

Distribution of Time Spent, by Testing Regime and Subprocess, in the Community Scenario

Test Regime	Subprocess
	Patient Transportation (hr)	Sample Collection (min)	Sample Transportation (hr)	Sample Processing (min)	Result Delivery (hr)
POCT	0	Tri(20,30,40)	0	Tri(10,30,50)	0
POCA	0	Tri(20,30,40)	Tri(2,4,6)	Tri(5,15,25)	Tri(4,8,12)
Central lab	Tri(2,4,6)	Tri(20,30,40)	0	Tri(5,15,25)	Tri(2,4,6)

POCA, point-of-care sample acquisition; POCT, point-of-care testing.

Table 3.

Distribution of Time Spent, by Testing Regime and Subprocess, in the Hospital-Based Scenario

Test Regime	Subprocess
	Patient Transportation (hr)	Sample Collection (min)	Sample Transportation (hr)	Sample Processing (min)	Result Delivery (hr)
POCT	0	Tri(20,30,40)	0	Tri(10,30,50)	0
POCA	0	Tri(20,30,40)	Tri(0.5,1,1.5)	Tri(5,15,25)	Tri(0.2,1,1.8)
Central lab	Tri(0.2,1,1.8)	Tri(20,30,40)	0	Tri(5,15,25)	Tri(0.2,1,1.8)

POCA, point-of-care sample acquisition; POCT, point-of-care testing.

Table 4.

Default Input Parameter Values That Are Identical Across the Four Different Target Conditions

Variable Name	Severity	Value
Established Patient Inter-arrival Time	1	N(129,60) (min)
Established Patient Inter-arrival Time	2	N(61,60) (min)
Established Patient Inter-arrival Time	3	N(144,60) (min)
New Patient Inter-arrival Time	1	N(596,60) (min)
New Patient Inter-arrival Time	2	N(303,60) (min)
New Patient Inter-arrival Time	3	N(722,60) (min)
New Patient Exam Time	1	N(15,9) (min)
New Patient Exam Time	2	N(17,8) (min)
New Patient Exam Time	3	N(42,16) (min)
Established Patient Exam Time	1	N(13,6) (min)
Established Patient Exam Time	2	N(15,5) (min)
Established Patient Exam Time	3	N(42,16) (min)
Time to Fill Out Paper Work	All	Tri(10,15,20) (min)
Probability of a Patient Having Target Condition	1	0.2
Probability of a Patient Having Target Condition	2	0.4
Probability of a Patient Having Target Condition	3	0.75
Probability of Testing a Patient	1	0.5
Probability of Testing a Patient	2	0.7
Probability of Treating a Patient	3	0.9

Table 4 enumerates the most important default input parameter values that are identical across the four target conditions that we test in this article. Table 4 shows the arrival rates and characteristics of various population groups of simulated incoming patients. Parametrically fitted arrival and exam time Normal distributions are drawn from a national survey of the Society of General Internal Medicine,²⁰ the National Ambulatory Medical Care Survey,²⁴ and a series of studies on real-time location systems (RTLS) in health care delivery.^25–27 Further discussion of arrival rate modeling considerations is provided in Appendix B.1. In general, triangular distributions are used for stochastic model inputs for which data for parametric fitting of distributions was not available, an approach commonly used in simulation studies,²⁸ allowing for both ease of elicitation from experts and intuitive presentation. In Appendix B.3, we replace triangular distributions with (arguably more realistic) PERT distributions, which do not substantially alter our conclusions. As shown in Table 4, indicator severity positively correlates with both the probability of having the target condition and the probability of being tested. Patients of Severity 3 are automatically treated (following testing) with a probability given in Table 4, because it is assumed that a combination of clinical judgement and the urgency of their indicators necessitates treatment.

Tables 5 and 6 enumerate test characteristics and target condition characteristics, respectively, both for pneumonia and for the additional target conditions used in the sensitivity analysis in Appendix A. Test sensitivities and specificities shown in Table 5 characterize the differences in test quality between testing regimes. We analyze the robustness of our simulation results to these input parameters in our simulation of pneumonia testing. Quantitative estimates of sensitivity and specificity in various scenarios were derived from expert knowledge and assessment. In general, POCT tests are assigned worse values of default sensitivity and specificity than those assigned to central lab and POCA testing regimes. Patients who test positive for the target condition receive treatment for that condition and those who test negative do not receive treatment for that condition (or any treatment that affects the number of days they are unable to be productive after discharge from the clinical system). The number of subsequent sick days varies based on both presence of the target condition and treatment, as shown in Table 6. Therefore, test characteristics (sensitivity and specificity) affect average patient outcomes (in particular, subsequent sick days and thus total lost productive hours). Note that treatment is assumed to have no effect on the number of subsequent sick days if the patient does not have the target condition. We also did not include any potential harm resulting from treating a patient in the well state. Discrete distributions are used for situations where there is a significant probability of no subsequent work days being lost.

Table 5.

Default Values of Test Sensitivities/Specificities and Treatment Costs

Variable Name	Pneumonia Test	Opiate Test	Chlamydia Test	Cholesterol Test
POCT Sensitivity (%)	80	75	80	90
POCT Specificity (%)	80	70	80	90
POCA Sensitivity (%)	95	95	95	97
POCA Specificity (%)	95	95	95	97
Central Lab Test Sensitivity (%)	95	95	95	97
Central Lab Test Specificity (%)	95	95	95	97
Per-patient Treatment Cost ($)	61	5,980	66	68

POCA, point-of-care sample acquisition; POCT, point-of-care testing.

Default test costs follow a specific structure as outlined in Tables 7 and 8. Default costs for urine tests, blood tests,²⁹ and X-rays are given in Table 7, for each testing regime. Consistent with observations in the literature,^3,10,11 individual POCT tests are given higher costs than other testing regimes. In addition, Table 8 displays the probabilities of receiving blood tests, X-ray tests, or a combination of these three tests, with respect to indicator severity, in the case of pneumonia, because different testing mechanisms, or combinations thereof, can be used in the detection of this condition.³⁰ In this case, sensitivity and specificity represent aggregate measures of diagnostic quality. Per-patient treatment costs for pneumonia and alternative conditions used in sensitivity analysis in Appendix A,^31–34 estimated as annual costs for prescribed medications, are given in Table 5.

Table 7.

Test Costs ($) by Test Procedure and Testing Regime

Regime	Procedure
	Blood Test	X-Ray Test	Urine Test
POCT	323	138	108
POCA	308	123	25
Central lab	303	118	88

POCA, point-of-care sample acquisition; POCT, point-of-care testing.

The funding sources for this study provided support for research personnel and simulation software.

Results

Pneumonia Simulation Outcomes With Default Parameters

In this section, we present default simulation results when testing for pneumonia, summarized in Table 9.

Table 9.

Outcome Measure Averages (95% CI Half-Widths in Parentheses)

	Rural	Community	Hospital-Based
A: Average time in the clinical system per patient (hr)
POCT	1.69 (0.01)	1.69 (0.01)	1.69 (0.01)
POCA	17.17 (0.57)	10.28 (0.81)	2.74 (0.01)
Central lab	15.62 (0.86)	6.78 (0.20)	2.74 (0.01)
B: Average subsequent sick days per patient
POCT	1.73 (0.01)	1.73 (0.01)	1.73 (0.01)
POCA	1.51 (0.01)	1.53 (0.01)	1.53 (0.01)
Central lab	1.52 (0.01)	1.52 (0.01)	1.52 (0.01)
C: Average total lost productive hours per patient
POCT	15.50 (0.08)	15.50 (0.08)	15.50 (0.08)
POCA	29.29 (0.57)	22.49 (0.81)	14.97 (0.08)
Central lab	27.80 (0.87)	18.93 (0.21)	14.92 (0.08)
D: Average cost per tested patient ($)
POCT	361.12 (0.58)	361.12 (0.58)	361.12 (0.58)
POCA	315.26 (0.45)	315.18 (0.43)	314.94 (0.41)
Central lab	328.09 (0.50)	328.9 (0.50)	328.01 (0.56)
E: Total societal costs per patient ($)
POCT	842.97 (3.24)	842.97 (3.24)	842.97 (3.24)
POCA	1244.61 (21.51)	1024.14 (29.36)	778.13 (3.22)
Central lab	1203.92 (30.99)	916.63 (9.42)	784.39 (3.23)

CI, confidence interval; POCA, point-of-care sample acquisition; POCT, point-of-care testing.

As shown in Table 9, section A, average time in the system per patient is similar under the POCT regime across clinical scenarios, as is expected, because neither patients nor samples leave the clinic. For POCA and central lab testing, time spent in the system in the Hospital-based scenario is much smaller than that in the Community scenario, and time spent in the system in the Community scenario is in turn much smaller than that in the Rural scenario. This is because of the corresponding variation in transportation time to the central lab necessary in each of these respective scenarios. On the dimension of time spent in the system alone, patients do much better with POCT in Rural and Community scenarios. For Hospital-based scenarios, times are comparable across regimes. As default sick day settings were identical between Rural, Community, and Hospital-based scenarios, average number of subsequent sick days per patient displayed in Table 9, section B, are the same across all three clinical scenarios. When we compare testing regimes, we can see that due to higher test quality, POCA and central lab have fewer subsequent sick days than POCT.

Table 9, section C, displays the total lost productive hours per patient, which is a (weighted) combination of the values in sections A and B, that is, Time in Clinical System+8*(Subsequent Sick Days). For Rural and Community clinical scenarios, POCT is the best testing regime on this dimension. In the Hospital-based scenario, the results of the three testing regimes are similar to each other on this dimension. In summary, we can see that POCT has a large advantage in total lost productive hours in the Rural and Community scenarios because of the large transportation time to a central lab. In these two clinical scenarios, gains from reduced transportation time outweigh losses as a result of lower test quality. However, these advantages are not clear in the Hospital-based scenario, where advantages in test quality compensate for the deficiencies in result turnaround times for the POCA and central lab testing regimes.

Table 9, section D, displays the average cost per tested patient, in terms of the usage of the direct testing resources. As expected, per-patient costs for POCT are higher than the per-patient costs for other testing regimes across the board. As noted by Lee-Lewandrowski and Lewandrowski,³ assessing the full costs of testing regime decisions for various stakeholders is much more challenging than simply assessing the direct unit costs of administering different types of tests. One approach to moving beyond these unit costs is to look at total lost productive hours in monetary terms, such that they may be compared and added to the direct unit costs of testing to estimate patients’ total actual costs. As a simple but crude estimate, we can take the pre-tax average hourly earnings of American workers plus fringe benefits (approximately $36 per hour in 2018³⁵) as a typical societal cost of a lost hour of work, and the value of a lost hour of leisure as post-tax average hourly earnings of American workers plus fringe benefits, as recommended by the Second Panel on Cost-Effectiveness in Health and Medicine.³⁶ Time spent seeking and receiving care is typically assumed to come from leisure time,³⁶ so we apply a value of $36 minus 9.2% (the estimated effective individual tax rate in the United States in 2014)³⁷ per hour to average time spent in the system, or $32.69. We can thus estimate total “societal” costs (sans treatment costs) by multiplying the values in Table 9, section A, by $32.69, the values in section B (multiplied by 8) by $36, and then adding these both of these values to the test costs per tested patient in section D scaled by the ratio of tested patients to total patients discharged from the clinic (this ratio ranged from 0.75 to 0.76).

Finally, we consider costs of treatment. These arise in two testing situations: true positive and false positive testing results. These can be estimated on a per-patient basis by calculating the proportions of true and false positives among all patients checking out, multiplying these proportions by the unit cost of treatment ($61 for pneumonia, as noted in Table 5), and finally adding the per-patient true positive and false positive treatment costs. The resulting average total treatment costs per patient, as well as true and false positive calculations, are shown in Table 10 for the Rural scenario (unsurprisingly, numbers for other scenarios are very similar). As expected, POCT has a higher proportion of, and thus higher cost due to, false positives relative to central lab testing and POCA because of its lower specificity—these costs can be considered unnecessary expenditures owing to test quality. On the other hand, lower POCT sensitivity means that true positive treatment costs are lower as well. The resulting total treatment costs are similar between the three testing regimes, and the costs of treatment relative to other expenses in the case of pneumonia are relatively minor. These total treatment costs (shown in the final column of Table 10 for the Rural scenario) are added to the societal costs sans treatment costs calculated above, and the results of this combination are presented as total societal costs in Table 9, section E.

Table 10.

Average Per-Patient Treatment Costs for Pneumonia (Rural)

Testing Regime	False Positive Treatments (% of Total)	False Positive Costs ($)	True Positive Treatments (% of Total)	True Positive Costs ($)	Total Costs ($)
POCT	12.24	7.47	16.38	10.00	17.47
POCA	9.57	5.84	18.95	11.56	17.40
Central lab	9.69	5.91	18.83	11.49	17.40

POCA, point-of-care sample acquisition; POCT, point-of-care testing.

While this total societal cost estimation does not change the ordering found in the costs of Table 9, section C, by much (with the exception of POCA and Central lab costs in the Hospital-based scenario), the relative advantages of POCT decrease across the board. This is particularly apparent when examining the relative advantages of POCT in the Hospital-based scenario—it is easy to imagine that with different assumptions about the costs of a lost hour of productivity (work or leisure), expected number of sick days after discharge, and direct costs of testing, total costs evaluated in this way might give POCA or central lab testing an edge in some scenarios. Figure 3 illustrates an example of this point by varying the estimated hourly wage rate + fringe benefits: increasing cost of labor can significantly increase advantages of the testing regime that minimizes hours lost.

Figure 3.

Total societal cost per patient ($) as hourly wage rate + fringe benefits is varied for Rural (left panel) and Hospital-based (right panel) scenarios (with pneumonia as the target condition). Increasing hourly wage rate increases the advantages of lower lost productive hours: in the Rural scenarios, the advantage of POCT becomes significantly greater, while in Hospital-based scenarios, the disadvantages of POCT become more obvious.

Sensitivity Analysis on Pneumonia Input Parameters

The simulation results for pneumonia presented thus far are based on the default values for sensitivities and specificities as listed in Table 5, most of which were estimated based on expert knowledge. POCT’s sensitivity and specificity are important factors that affect its utility relative to other testing regimes. For this reason, we conducted sensitivity analyses to analyze our simulation output under various combinations of sensitivity and specificity parameter values. Simulations were run for 30 replications each (see Appendix B.2).

We vary the sensitivity and specificity for each testing regime in the neighborhood of their default values. This allows us to check if, and to what extent, outcomes vary in the expected direction. It also allows us to understand the tradeoffs in costs and outcomes with respect to sensitivity and specificity. For example, one might imagine the introduction of a new POCT technology with better test characteristics but similar costs, which can significantly change POCT utility in a given clinical scenario if patient outcomes are greatly improved due to the better test characteristics.

Figure 4 plots total lost productive hours, as sensitivity and specificity are varied, for different clinical scenarios and testing regimes. From Figure 4a, we find that the total number of lost productive hours per patient for all three testing regimes in the Rural scenario decrease as sensitivity and/or specificity increases for that scenario. Total lost productive hours under the POCT regime are much smaller than those under the other two testing regimes, even when POCT sensitivity and specificity values are much lower than those for the other two testing regimes (less than 0.80/0.80, for example). This is explained by the fact that in the Rural scenario, patient transportation and sample transportation times are much higher than the other two clinical scenarios.

Figure 4.

Total number of lost productive hours, for the Rural (a), Community (b), and Hospital-based (c) scenarios, for varied sensitivity and specificity values on the x- and y-axes, respectively. Green = POCT; Red = POCA; Blue = Central Lab.

Figure 4b shows the total number of lost productive hours in all three testing regimes for the Community scenario. Because of shorter transportation times, the time difference between POCT and other testing regimes is smaller. If POCT’s sensitivity and specificity are much lower than those of central lab testing (lower than 0.70/0.70, for example), central lab testing could gain an edge over POCT in this metric. Otherwise, POCT still has an advantage over the other two testing regimes in this scenario. Figure 4c shows the total number of lost productive hours in the Hospital-based scenario. Because transportation times for central lab testing and POCA are relatively small in this scenario (approximately 1 hour each), the total time spent for all three testing regimes is similar in overlapping sensitivity and specificity ranges. Because POCT usually has a lower sensitivity and specificity among all three testing regimes, POCA and central lab testing have a slight advantage over POCT in the Hospital-based scenario. As expected, for the same sensitivity and specificity levels, POCT always has an advantage over central lab testing and POCA in terms of lost productive hours.

Another revealing sensitivity analysis can be performed on the cost of tests. New, cheaper POCT technology may alter the calculus of when different testing regimes are most desirable. We frame this as an operational question: at what cost would a new POCT technology have to be for POCT to be preferable in all scenarios, Rural, Community, and Hospital-based? In order to explore this question concretely, we vary the cost of the POCT blood test administered to potential pneumonia patients from its default value of $323. At this default value, based on estimated total societal costs in Table 9, section E, POCT is the most preferable among the three testing regimes in Rural and Community scenarios, but not in the Hospital-based scenario. We adjust the blood test cost, running 120 replications for each cost scenario, until the estimated total societal costs per patient under the POCT testing regime dip below those of POCA and central lab testing regimes. As shown in Figure 5, somewhere between a cost of $280 and $260 per blood test, POCT becomes cheaper (from a societal perspective) than both the POCA and central lab regimes even in the Hospital-based scenario.

Figure 5.

Estimated total societal costs per patient in the Hospital-based scenario (with pneumonia as target condition), as POCT blood test cost is varied. POCA and Central lab test costs are held at their default values.

Discussion

Driven by changes in the health care system and advances in medical technology, the use of POCT is growing quickly in US health care. At present, however, understanding of where and when POCT can be optimally applied is not completely clear. There is a great opportunity to significantly improve health care delivery by analyzing the performance of different testing regimes in different clinical scenarios.

In this study, we used a simulation-based approach to model a primary care clinic workflow and explore three different clinical scenarios (Rural, Community, and Hospital-based) to evaluate POCT performance. Under default parameter values, derived from expert knowledge, we analyzed the performance of three different testing regimes in each of these three clinical scenarios along several dimensions. Our results show that POCT can reduce the average time in the clinical system for each condition and clinical scenario. However, the total number of lost productive hours per patient can be greatly affected by diagnostic quality, where POCT often has a disadvantage (and can vary greatly depending on the condition in question, as shown in Appendix A). When the total lost productive hours are used to estimate a total societal monetary cost in conjunction with direct per-test costs and treatment costs, these advantages are not obvious in the Hospital-based scenario. If costs of lost productive time are increased (e.g., in the case of higher salaried workers or essential service workers such as doctors or air traffic controllers), the advantages of POCT are greater in Rural scenarios, and POCT becomes a significantly worse choice than POCA or central lab testing in Hospital-based scenarios. Thus, costs of lost productivity per hour in the patient population may be an important consideration when evaluating testing regimes.

Our study has a few limitations. First, our evaluation only considers patients’ time spent/lost and the costs of treatment and testing resources used, but neglects other potential outcome measures such as staff utilization or annual clinic revenue. Central lab testing may allow for economies of scale that could have advantages in these dimensions. Additionally, many simulation parameters, including sensitivity and specificity values, costs, and time spent in each clinical process are based on expert knowledge. These parameters and other aspects of clinic operation may vary significantly in different settings and with different patient populations. For example, estimated process times and patient demographic information would significantly benefit from further data collection, and integration of fitted distributions into the model could greatly improve its utility. Longer term modeling of patient health outcomes beyond short-term lost days of productivity could allow for deeper insights about testing tradeoffs and the consequences of false negatives and false positives (e.g., mortality risk as a result of untreated pneumonia). With this article, we have developed a simulation-based framework for evaluating tradeoffs of different testing regimes in different clinical scenarios. It is hoped that future work will expand the flexibility, scope, and accessibility of models of this type, such that decision makers in clinical situations can effectively evaluate and compare different testing regimes in their particular settings of interest.