Abstract
Background
Registry-based trials have the potential to reduce randomized clinical trial (RCT) costs. However, observed cost differences also may be achieved through pragmatic trial designs. A systematic comparison of trial costs across different designs has not been previously performed.
Methods
We conducted a study to compare the current Steroids to Reduce Systemic inflammation after infant heart surgery (STRESS) registry-based RCT vs. two established designs: pragmatic RCT and explanatory RCT. The primary outcome was total RCT design costs. Secondary outcomes included: RCT duration and personnel hours. Costs were estimated using the Duke Clinical Research Institute's pricing model.
Results
The Registry-Based RCT estimated duration was 31.9 weeks greater than the other designs (259.5 vs. 227.6 weeks). This delay was caused by the Registry-Based design's periodic data harvesting that delayed site closing and statistical reporting. Total personnel hours were greatest for the Explanatory design followed by the Pragmatic design and the Registry-Based design (52,488 vs 29,763 vs. 24,480 h, respectively). Total costs were greatest for the Explanatory design followed by the Pragmatic design and the Registry-Based design ($10,140,263 vs. $4,164,863 vs. $3,268,504, respectively). Thus, Registry-Based total costs were 32 % of the Explanatory and 78 % of the Pragmatic design.
Conclusion
Total costs for the STRESS RCT with a registry-based design were less than those for a pragmatic design and much less than an explanatory design. Cost savings reflect design elements and leveraging of registry resources to improve cost efficiency, but delays to trial completion should be considered.
Keywords: Cost analysis, Data collection, Registry, Clinical trial
Highlights
-
•
Registry-based trial observed cost savings may be achieved through pragmatic trial designs.
-
•
We compared costs for the STRESS registry-based RCT vs. two established designs: pragmatic RCT and explanatory RCT.
-
•
STRESS registry-based design total costs were 32 % of the explanatory and 78 % of the pragmatic design.
1. Background
Conducting randomized clinical trials (RCT) in pediatric cardiac populations is challenging due to patient/disease heterogeneity, the infrequency of ‘hard’ outcomes, and logistical barriers that increase study costs [1,2]. Several initiatives have accelerated therapeutic discovery for pediatric patients with congenital and acquired heart disease including the Pediatric Heart Network (PHN) and the Society of Thoracic Surgeons Congenital Heart Surgery Database (STS-CHSD) [[3], [4], [5]]. STS-CHSD collects data from >96 % of US congenital heart surgery centers that perform 98 % of US congenital heart operations [3]. Ten percent of STS-CHSD participating institutions undergo onsite audits each year [6,7]. Recently, registry-based trials have been proposed as a means to build upon these initiatives and improve pediatric cardiac RCT efficiency [8,9].
Three studies are largely responsible for the present interest in registry-based RCT designs [[10], [11], [12]]. These are the 7244 patient Thrombus Aspiration during ST-Segment Elevation Myocardial Infarction (TASTE) trial that used the Swedish Coronary Angiography and Angioplasty Registry (SCAAR); the 496 patient Study of Access site For Enhancement of Percutaneous Coronary Intervention for women (SAFE-PCI for Women) trial that used the American College of Cardiology Foundation's CathPCI Registry; and the 74,256 patient Randomized Evaluation of Decolonization versus Universal Clearance to Eliminate MRSA (REDUCE MRSA) trial that used Hospital Corporation of America (HCA) data warehouses. Some studies indicate that registry-based RCTs can dramatically reduce clinical trial costs [13,14]. However, other studies note that these cost savings may have resulted from their incorporation of pragmatic trial design elements and not specifically their use of registries [15]. The Clinical Trials Transformation Initiative's (CTTI) Recommendations for Registry Trials concluded that embedding RCTs within registries may be associated with improvements in [1]: data collection [2], patient identification and recruitment [3], database lock time [4], time to critical decision-making, and [5] RCT costs [16]. However, achieving these benefits depends upon a registry's ability to support a specific RCT.
Clinical trial designs can be categorized along an explanatory to pragmatic spectrum where explanatory trials are designed to determine an intervention's effects under “ideal” conditions and pragmatic trials determine the intervention's effects under “real-world” conditions without increased research support [17]. Registry-based trial designs create a randomized trial on an existing registry platform and can be either explanatory or pragmatic in design [14]. When a clinical trial is conducted in an inpatient setting and involves specialized medical care, there may be few differences between explanatory vs. pragmatic interventions and care settings. This leads to questions as to whether differences in trial designs will be associated with meaningful differences in resource use and total trial costs.
The Steroids to Reduce Systemic inflammation after infant heart surgery (STRESS) trial was a 1200 patient, 24 site, inpatient randomized clinical trial that utilized STS-CHSD registry infrastructure (ClinicalTrials.gov number, NCT03229538) and was conducted under IND 129,266 from the US Food and Drug Administration's Division of Cardiovascular and Renal Products [18,19]. Site personnel consented patients for the STS-CHSD registry and then used a separate consent for the STRESS trial. STRESS outcomes occurred during postsurgical hospitalization and included up to 30 days of out-patient follow up if discharged. We conducted a study to compare the economic attractiveness of the STRESS trial's registry-based RCT design vs. two alternative designs: pragmatic RCT and explanatory RCT. We also sought to identify design elements that were responsible for differences in total registry-based RCT costs.
2. Methods
The Second Panel on Cost-Effectiveness in Health and Medicine (Second Panel) proposed guidelines for conducting health economic analyses [20]. The STRESS Economic Analysis used the Second Panel's five-phase process with registry-based, pragmatic, and explanatory RCT designs as alternative strategies [21]. The five processes were [1]: Design the economic analysis [2], Identify consequences of each strategy [3], Measure each strategy's consequences [4], Assign values to each strategy's consequences, and [5] Summarize consequence values by strategy. We also conducted sensitivity analyses to evaluate the impact of key STRESS design elements upon this study's economic results.
2.1. Economic analysis design
We conducted a cost-minimization analysis to calculate, report, and compare total costs for the STRESS trial conducted as a registry-based RCT vs. STRESS conducted as a pragmatic RCT and as an explanatory RCT [22]. The objective was to inform key decision makers (e.g., research sponsors, trial designers, and other funders) as to the potential financial benefits that might accrue by selecting among alternative STRESS trial designs. Potential non-financial benefits associated with RCT design differences (e.g., differences in inclusiveness of enrollment, endpoint identification and data quality) were not considered. This economic analysis was conducted from the clinical trial perspective with secondary analyses conducted by clinical trial service groups [23]. The study period began with initial RCT planning and continued through primary results dissemination. The economic analysis did not consider potential financial consequences beyond the study period (e.g., changes in long-term patient health care costs).
2.2. Study and subjects
For each of the three RCT designs, we assumed 24 sites, 1289 subjects enrolled, 1250 subjects randomized, and 1201 subjects completing the study. The STRESS investigators’ proposal included a sample size of 1250 subjects enrolled with 1200 completing the study. The sample size estimate using STS-CHSD data with Monte Carlo simulations improved the STRESS sample size estimate [24]. However, because there was no change in the STRESS sample size, we assumed the actual number of subjects enrolled would be the same in all three RCT designs.
2.3. Intervention and comparator strategy consequences
We began the analysis by [1]: identifying consequences relevant to each RCT design and [2] describing their data sources and the measurements we would use [25]. In this analysis, differences in RCT designs would be evident in the clinical trial services (e.g., on-site vs. remote (phone, web meeting) monitoring visits) and their levels of intensity (e.g., 4 vs. 2 remote monitoring visits) and in the study timelines required by the three RCT designs. Each RCT-design's services and study timeline were used to estimate that design's personnel hour requirement, personnel costs, and contracted service costs.
The Duke Clinical Research Institute (DCRI) clinical trial pricing model was the primary source for study measurements. This model provides a common method for estimating clinical trial resource use and costs [26,27]. The STRESS trial's pricing model was used to estimate registry-based RCT measurements. Measurement for the pragmatic and the explanatory RCT designs were in part based upon service assumptions for contemporary National Institutes of Health funded studies. The budget for the registry-based design is the actual STRESS trial budget as submitted and approved for NIH funding. The budgets for the pragmatic and explanatory designs were completed after the STRESS trial started enrollment. This study's budgets were constructed assuming National Institutes of Health funding. These only included direct costs as indirect costs would vary by institution and would not be related to the work in a specific RCT. Site payments were estimated as a fixed per patient direct cost plus 50 % average indirect costs.
2.4. Strategy consequences measurements, valuations and summaries
Strategy consequences were measured as elapsed time for each RCT design (total and by phase) and as service hours (total and by functional activity). Service hours were assigned costs in each RCT's pricing model. This meant that similar services could have different costs depending upon their levels of intensity (i.e., number of times performed, service hours assigned per occurrence, and the hourly rate for the role performing that service). Service hours and costs were summarized by functional group. Table 1 shows the 21 service activities that were summarized into 6 service groups. Personnel hours and costs are presented by RCT design (registry-based, pragmatic, and explanatory) with differences for the explanatory vs. pragmatic designs and for the pragmatic vs. registry-based designs. All cost estimates are reported in 2020 US dollars.
Table 1.
Study activity assumptions.
| Activities | Registry-Based Trial | Pragmatic Trial | Explanatory Trial |
|---|---|---|---|
| Leadership | |||
| Steering Committee | X | X | X |
| Trial Leadership | X | X | X |
| Project Management | X | X | X |
| Site Management & Monitoring | |||
| Site Coordination | X | X | X |
| Monitoring | X | X | X |
| Data Management & Statistics | |||
| Participant Safety and Data Monitoring | X | X | X |
| Clinical Events Committee | X | ||
| Clinical Data Management | X | X | X |
| Information Technology | X | X | X |
| Registry Access | X | ||
| Statistics | X | X | X |
| Other Services | |||
| Safety Surveillance | X | X | X |
| Quality Assurance | X | ||
| Contracts/Payments | X | X | X |
| Helpline | X | ||
| Communications | X | ||
| Regulatory Services | X | X | X |
| Miscellaneous Activities | |||
| Meetings, Travel and Supplies | X | X | X |
| Randomization and Support | X | X | X |
| Study Site Activities | |||
| Site Payments | X | X | X |
| Site Data Collection | X | X | X |
Costs for site data collection were not estimated by the DCRI pricing model. These costs were estimated by determining the STRESS trial's required data elements and their data sources for each RCT design, and then calculating each design's site data collection costs. Two STRESS trial reports contained the required data elements [1]: FDA Progress Report and [2] Final Statistical Report. In the STRESS trial, data for these reports comes from a site data entry system and from the STS-CHSD database. We assumed that data sources for the registry-based design would be the same as for the STRESS trial. We also assume that all data for the pragmatic and explanatory designs would come from an expanded site data entry system.
2.5. Sensitivity analyses
Sensitivity analyses were conducted to determine the extent to which the STRESS registry-based trial design might be adapted to reduce total trial costs. The scenarios tested were in part based upon recommendations in three expert consensus publications [[28], [29], [30]]. These included [1]: reducing sample size requirements [2], accelerating patient enrollment, and [3] automating data collection. Reducing sample size requirements is hypothesized to occur when registry data are used to estimate the sample size for an RCT that will be conducted in the registry's population. Similarly, accelerated enrollment may occur when the registry is used to identify potential participants. Lastly, automated data collection occurs when all of the RCT data are available from the registry allowing study data to be copied from the registry to the study database [31]. Besides eliminating site data collection this also would reduce the need for other data management and onsite monitoring activities.
3. Results
3.1. Services provided
Table 1 describes the service activities provided in each RCT design. The Explanatory RCT design included Clinical Events Committee (CEC) endpoint adjudication, Quality assurance, a Helpline, and Communications Department support activities that were not provided in the other RCT designs. Similarly, the Registry-Based RCT was the only design that had Registry Access that provided access to an established data infrastructure. There also were significant differences between RCT designs in the intensity of service activities required for Leadership, Site Management & Monitoring and for Data Management & Statistics.
Leadership differences were driven by the number of faculty members and the intensity of estimated Faculty and Project Management interaction with study sites. Site Management & Monitoring strategies differed by RCT design. The Registry-Based and Pragmatic designs used combination evaluation and initiation calls, a single onsite interim monitoring visit, and closeout calls. In contrast, the Explanatory design used separate Evaluation and Closeout calls, but divided Initiation visits between 5 onsite and 19 phone, with 4 onsite monitoring visits planned per site. Data Management service differences largely were attributable to differences in the use of registry data. The Registry-Based RCT design had fewer electronic data capture screens, but it also had periodic data transfers from the STS-CHSD registry and paid a fee for the use of this registry's data. In contrast, the Pragmatic and Explanatory designs had no STS-CHSD data transfers or registry use fees.
3.2. Study timeline
Table 2 describes the timeline assumptions for each RCT design. The Registry-Based RCT estimated duration was 259.5 weeks; whereas, the estimated durations for the Pragmatic and Explanatory RCTs were 227.6 weeks (31.9 weeks less than for the Registry-Based RCT). This difference was due to the Registry-Based design's required use of periodic data harvesting from the central registry data warehouse that delayed Site Closing and the time at which Statistical Reporting could occur. We assumed the timeline for Statistical Reporting and Dissemination in the Pragmatic and Explanatory designs would reflect standard practice. In this context, Statistical reporting refers to the statistical analyses and report preparation that occurred after sites are closed and the study database is locked. Reports and Dissemination refers to the time to prepare and submit the final study reports and the primary study manuscript.
Table 2.
Clinical trial timeline assumptions in weeks.
| Study Phase | Registry-Based Trial | Pragmatic Trial | Explanatory Trial |
|---|---|---|---|
| Planning | 52 | 52 | 52 |
| Enrollment | 156 | 156 | 156 |
| Treatment | 4.3 | 4.3 | 4.3 |
| Follow-up | 4.3 | 4.3 | 4.3 |
| Database lock delay | 20.7 | 0.0 | 0.0 |
| Sites closed | 4.3 | 4.3 | 4.3 |
| Statistical Report | 4 | 4 | 4 |
| Reports and Dissemination | 14 | 2.7 | 2.7 |
| Total Weeks Duration | 259.5 | 227.6 | 227.6 |
3.3. Personnel hours by functional group
Table 3 describes the expected hours by functional group for each of the three RCT designs. Total personnel hours for the Explanatory design were 22,725 h greater than those for the Pragmatic design (52,488 vs. 29,763 h) and the Registry-Based design total hours were 5282 h less than for the Pragmatic Design (24,480 versus 29,763 h). Service groups with the greatest contribution to total hour differences were Leadership, Site Management & Monitoring, and Data Management & Statistics. Leadership hours for the Registry-Based (8456 h) and Pragmatic (7798 h) trial designs were much less than for the Explanatory design (13,926 h), reflecting the greater complexity associated with explanatory vs. pragmatic RCT designs. Similarly, Site Coordination and Monitoring hours were lowest for the Registry-Based design (8627 h), higher for the Pragmatic design (12,548 h) and much higher for the Explanatory design (19,397 h). Besides differences in explanatory vs. pragmatic designs, these differences also can be attributed to the STS-CHSD registry performing specific site management and monitoring functions. Data Management and Statistics time was lower for the Registry-Based (5802 h) than the Pragmatic design (7833 h) and higher for the Explanatory design (14,979). These differences reflect additional statistical analyses and reporting for the explanatory design beyond that required for the two pragmatic designs.
Table 3.
Personnel hours by functional group.
| Functional Group | RCT Design |
Personnel Hour Differences |
|||
|---|---|---|---|---|---|
| Registry-Based | Pragmatic | Explanatory | Explanatory vs. Pragmatic | Pragmatic vs. Registry-Based | |
| Leadership | |||||
| Steering Committee | |||||
| Trial Leadership | 5657 | 5048 | 9026 | 3978 | −609 |
| Project Management | 2799 | 2750 | 4900 | 2150 | −49 |
| Subtotal | 8456 | 7798 | 13,926 | 6128 | −658 |
| Site Management & Monitoring | |||||
| Site Coordination | 5056 | 8302 | 10,479 | 2176 | 3247 |
| Monitoring (Lead) | 3571 | 4246 | 8918 | 4672 | 675 |
| Subtotal | 8627 | 12,548 | 19,397 | 6849 | 3921 |
| Data Management & Statistics | |||||
| Data and Safety Monitoring Board | NA | NA | NA | NA | NA |
| Clinical Events Committee | 0 | 0 | 2698 | 2698 | 0 |
| Clinical Data Management | 1745 | 3772 | 4948 | 1176 | 2027 |
| Information Technology | 893 | 893 | 755 | −137 | 0 |
| Registry Access | |||||
| Statistics | 3164 | 3168 | 6578 | 3410 | 4 |
| Subtotal | 5802 | 7833 | 14,979 | 7146 | 2031 |
| Other Services | |||||
| Safety Surveillance | 1279 | 1299 | 1312 | 13 | 20 |
| Quality Assurance | 0 | 0 | 1312 | 1312 | 0 |
| Contracts/Payments | NA | NA | NA | NA | NA |
| Helpline | NA | NA | NA | NA | NA |
| Communications | 0 | 0 | 1277 | 1277 | 0 |
| Regulatory Services | 316 | 285 | 285 | 0 | −31 |
| Subtotal | 1595 | 1584 | 4186 | 2602 | −11 |
| Miscellaneous Activities | |||||
| Meetings, Travel, and Supplies | NA | NA | NA | NA | NA |
| Randomization and Support | NA | NA | NA | NA | NA |
| Subtotal | NA | NA | NA | NA | NA |
| Study Site Activities | |||||
| Site Payments | NA | NA | NA | NA | NA |
| Site Data Collection | NA | NA | NA | NA | NA |
| Subtotal | NA | NA | NA | NA | NA |
| Total Hours | 24,480 | 29,763 | 52,488 | 22,725 | 5282 |
NA = These services are contracted by the coordinating center or included in government supported trial indirect costs.
3.4. Site data collection
Table 4 describes site data collection costs in each of the STRESS RCT designs. The STRESS trial required 1020 unique variables of which 409 were captured by the STRESS site data entry system (for drug safety) and 611 came from the STS-CHSD registry (for trial outcomes). On average 94.45 variables per patient would be entered into the STRESS site data entry system and 153.45 variables per patient would come from the STS-CHSD registry. Assuming 10 unique variables per site data entry screen, the Registry-Based design would require 41 screens; whereas, the Pragmatic and Explanatory designs would require 102 screens. Assuming site clinical research coordinators would locate, abstract, and enter 60 data elements per hour, data collection time per patient was 1.6 h for the Registry-Based design and 2.6 h for the Pragmatic and Explanatory designs. Using query rates and clinical research coordinator costs from a previous study, we estimated STRESS per patient data collection costs as $191.73 for the Registry-Based design and $311.50 for the Pragmatic and Explanatory designs [32].
Table 4.
Site data collection costs.
| Elements | RCT Design |
Personnel Hour Differences |
|
|---|---|---|---|
| Registry-Based | Pragmatic and Explanatory | Explanatory and Pragmatic vs. Registry-Based | |
| Per Patient | |||
| Data entry | |||
| Screens | 41 | 102 | 61 |
| Variables entered | 94.45 | 153.45 | 59.00 |
| Hours elapsed | 1.598 | 2.596 | 0.998 |
| Costs | $191.73 | $311.50 | $119.77 |
| Adverse event search | |||
| Hours elapsed | 3.500 | 3.500 | |
| Costs | $420.00 | $420.00 | |
| Total Per Patient | |||
| Hours elapsed | 1.598 | 6.096 | 4.498 |
| Costs | $191.73 | $731.51 | $539.77 |
| Total Study | |||
| Hours elapsed | 1919.20 | 7321.30 | 5402.10 |
| Costs | $230,267.73 | $878,543.51 | $646,263.77 |
In STRESS, the STS-CHSD registry collected adverse events for the trial. Without the registry, site clinical research coordinators would expend an additional 10 min per day per patient capturing adverse events. Given the 21-day average length of stay for STRESS patient, this added an additional 3.5 h per patient ($420.00 per patient) to data collection costs for the Pragmatic and Explanatory designs. This meant that total per patient data collection costs were $191.73 for the Registry-Based design and $731.51 for the Pragmatic and Explanatory designs. We assumed the $191.73 data entry costs were included in the STRESS Registry-Based trial site payment and included a $539.78 increment for additional data entry in the Pragmatic and Explanatory design site payment amounts. For the Registry-Based design trial (1201 patients), this represents a 5402.10 h reduction (1919.20 vs. 7321.30) and a $646,263.77 cost reduction ($230,267.73 vs. $878,543.51) from the Pragmatic and Explanatory designs.
3.5. Total costs by functional group
Table 5 describes the estimated costs by functional group for each of the three RCT designs. Total costs for the Explanatory design are $5,975,401 greater than those for the Pragmatic design ($10,140,263. vs. $4,164,863) and the Registry-Based design's total cost ($3,268,504) is $896,358 less than for the Pragmatic design. Thus, the Registry-Based design's total costs were 32 % of the Explanatory and 78 % of the Pragmatic design.
Table 5.
Total costs by functional group.
| Functional Group | RCT Design |
Total Cost Differences |
|||
|---|---|---|---|---|---|
| Registry-Based | Pragmatic | Explanatory | Explanatory vs. Pragmatic | Pragmatic vs. Registry-Based | |
| Leadership | |||||
| Steering Committee | 20,000 | 20,000 | 20,000 | 0 | 0 |
| Trial Leadership | 530,819 | 533,342 | 942,986 | 409,644 | 2523 |
| Project Management | 194,858 | 206,227 | 304,803 | 98,576 | 11,369 |
| Subtotal | 745,677 | 759,569 | 1,267,900 | 508,219 | 13,893 |
| Site Mgmt & Monitoring | |||||
| Site Coordination | 230,328 | 376,928 | 408,283 | 31,355 | 146,600 |
| Monitoring | 196,578 | 293,192 | 616,582 | 323,391 | 96,614 |
| Subtotal | 426,905 | 670,119 | 1,024,865 | 354,745 | 243,214 |
| Data Mgmt & Statistics | |||||
| Data and Safety Monitoring Board | 23,172 | 23,172 | 56,250 | 33,078 | 0 |
| Clinical Events Committee | 0 | 0 | 331,262 | 331,262 | |
| Clinical Data Mgmt | 191,872 | 313,039 | 370,317 | 57,278 | 121,167 |
| Registry Access Fee | 158,531 | −158,531 | |||
| Statistics | 185,918 | 187,083 | 585,544 | 398,461 | 1165 |
| Subtotal | 559,493 | 523,294 | 1,343,373 | 820,079 | −36,199 |
| Other Services | |||||
| Safety Surveillance | 77,867 | 79,397 | 79,397 | 0 | 1530 |
| Quality Assurance | 0 | 0 | 3105 | 3105 | 0 |
| Contracts/Payments | NA | NA | NA | NA | NA |
| Communications | 0 | 0 | 119,701 | 119,701 | 0 |
| Regulatory Services | 18,932 | 17,788 | 17,788 | 0 | −1144 |
| Subtotal | 96,799 | 97,185 | 219,990 | 122,806 | 386 |
| Miscellaneous Activities | |||||
| Meetings, Travel, Suppls | 9624 | 9989 | 207,401 | 24,389 | 365 |
| Randomization and Support | 148,613 | 148,613 | 0 | ||
| Subtotal | 9624 | 9989 | 356,014 | 346,026 | 365 |
| Site Payments | 1,430,006 | 2,104,706 | 5,928,232 | 3,823,526 | 674,700 |
| Total Costs | 3,268,504 | 4,164,863 | 10,140,263 | 5,975,401 | 896,358 |
Mgmt = Management.
Suppls = Supplies.
NA = These services are included in government supported trial indirect costs.
Service groups with the greatest contribution to total cost differences are Leadership, Site Management & Monitoring, Data Management & Statistics, and Study Site Activities. The primary factor accounting for total cost differences between designs was the Explanatory design's $5,928,232 site payments vs. $2,104,706 for the Pragmatic and $1,430,006 for the Registry-Based design. While the Explanatory design's site payment was based upon the actual site payment amount for similar trials, a lower site payment might be appropriate for the STRESS design. Nonetheless, the Explanatory design's site payment would be expected to be significantly greater than those for either the Pragmatic or Registry-Based designs.
Leadership cost differences largely were driven by differences in Trial Leadership costs (Explanatory RCT, $942,986; Pragmatic RCT, $533,342; Registry-Based RCT, $530,819). Site Management & Monitoring cost differences were in part caused by increased Site Coordination costs for Explanatory and Pragmatic designs ($408,283 and $376,928 respectively) vs. the Registry-Based design ($230,328); whereas, Monitoring costs increased from the Registry-Based design ($196,578) to the Pragmatic design ($293,192), with the Explanatory design being highest ($616,582).
Many factors accounted for the observed differences between-RCT design in Data Management & Statistics costs. Importantly, the Explanatory design alone had a Clinical Events Committee ($331,262), the Registry-Based design had Registry Access fees ($158,531), and the Explanatory design's Statistics costs ($585,544) were higher than those for either the Pragmatic or Registry-Based designs ($187,083 and $185,918, respectively). Interestingly, the total costs for this Service Group were similar for the Pragmatic and Registry-Based designs ($523,294 vs. $559,493, respectively). The Explanatory design also had $148,613 for contracted randomization support costs; whereas, the other two design managed patient randomization through their electronic data capture systems.
3.6. Sensitivity analyses
Table 6 evaluates three methods that have been proposed for reducing RCT costs. Reducing the number of subjects by 10 % reduced Registry-Based design total trial costs by $129,052. This reduction is driven by a $125,001 reduction in site activities (site payments and data collection). Reducing enrollment time by 10 % only reduced total trial costs by $36,399. This reduction is driven by a $26,620 reduction in site management & monitoring costs. The greatest reduction in total trial costs ($176,239) was seen when all study data were collected in the registry and transferred into the study database. This reduction was largely driven by a reduction in data management costs ($102,465). If site payments were reduced due to a reduction in site data collection effort, the savings would be greater.
Table 6.
Registry-based RCT sensitivity analysis.
| Functional Group | Registry-Based | 10 % Fewer Subjects |
10 % Less Enrollment Time |
Automated Data Collection |
|||
|---|---|---|---|---|---|---|---|
| Cost Estimates | Cost Differences | Cost Estimates | Cost Differences | Cost Estimates | Cost Differences | ||
| Leadership | 745,677 | 745,677 | 0 | 743,695 | −1982 | 720,931 | −24,746 |
| Site Mgmt & Monitoring | 426,905 | 422,854 | −4052 | 400,286 | −26,620 | 381,189 | −45,716 |
| Data Mgmt & Statistics | 559,493 | 559,493 | 0 | 556,912 | −2580 | 457,028 | −102,465 |
| Other Services | 96,799 | 96,799 | 0.00 | 91,582 | −5217 | 93,486 | −3312 |
| Miscellaneous Activities | 9624 | 9624 | 0 | 9624 | 0 | 9624 | 0 |
| Site Activities | 1,430,006 | 1,305,006 | −125,001 | 1,430,006 | 0 | 1,430,006 | 0 |
| Total Costs | 3,268,504 | 3,139,452 | −129,052 | 3,232,105 | −36,399 | 3,092,265 | −176,239 |
MGMT = management.
4. Discussion
In this study, the STRESS Registry-Based and Pragmatic designs had much lower costs than the Explanatory design ($10,140,263 vs. $4,164,863 vs. $3,268,504, respectively). While this finding supports the position that it is the pragmatic design that accounts for a significant portion of the reduction in total costs previously reported for registry-based RCTs [[13], [14], [15]], it does not account for the further cost reduction associated with the Registry-Based versus Pragmatic design.
Traditionally, registries have been associated with non-interventional studies [33]. Thrombus Aspiration in Myocardial Infarction (TASTE) was one of the first RCTs to highlight the potential benefits of a registry-based design [10]. However, TASTE was conducted entirely with registry data. While this approach has many benefits, the logistics of adding data elements to an existing registry can be a major barrier. Moreover, issues may arise when there is a lag between the time critical events occur, data are recorded in the registry database, and the registry data are transferred to the RCT's database. These issues can occur either when study data are required to monitor and adjust the study's treatment or when reportable events occur. Each of these may require that the registry-based RCT create a separate database to permit better patient management and event reporting or automate data transfer from study sites to the registry and then to the study database. In the STRESS trial, the 6-month data harvesting lag was problematic. However, with shorter time lags many of these issues may minimize the data lock penalty incurred when using registry data.
Although there is little research on potential cost savings associated with registry-based designs, previous researchers have noted that RCT data collection costs typically are not significant and that the major RCT cost drivers are procedures, site monitoring, site retention, and central laboratories [34,35]. Site data collection costs were $230,268 for the registry-based design and $878,544 for the pragmatic and explanatory designs. These amounts were 8.7 % of total trial costs for the explanatory design, 21.1 % for the pragmatic design, and 7.0 % for the registry-based design. Similarly, site data collection and clinical data management costs $422,140 for the registry-based design, $1,191,583 for the pragmatic design and $1,248,861 for the explanatory design. These amounts were 12.3 % for the explanatory design, 28.6 % for the pragmatic design, and 12.9 % for the registry-based design. Hence, although site and coordinating center data management costs might be relatively less important in an explanatory design, they remain important cost drivers and the primary source of cost savings when comparing registry-based and pragmatic designs where methods to reduce data collection costs have a greater relatively impact due to their lower overall trial costs.
An additional consideration is the 6-month lag caused by the use of a registry that had periodic data harvesting. The STRESS trial was conducted under an FDA-issued IND and the registry-based, pragmatic trial design required a modified approach to regulatory interactions, most specifically related to AE reporting. An ancillary database was required for real-time AE reporting and this cost was included in our analyses. Had the registry allowed continuous data harvesting, the ancillary AE database would not have been required. Other trial designs may necessitate other approaches to regulatory interactions and changes in data collection strategies.
The major service difference between the three RCT designs was that the Explanatory design included CEC endpoint adjudication; whereas, both pragmatic designs used actual practice data collected from sites without adjudication. In the Pragmatic design, study coordinators directly collected study data. In the Registry-Based design, data collected from study sites were made available to the study team. Previous studies have shown that a CEC and site investigators may identify different events [36,37]. This is in part because explanatory and pragmatic trials are designed to answer different questions: efficacy (Can it work?) vs. effectiveness (Does it work in practice?) [38]. RCT designers should specify which question they are addressing and design their endpoint assessment strategy accordingly. Although the present study's Explanatory design offered onsite monitoring and 10 % source document verification (SDV), researchers have long questioned the value of these services (both their costs and their effectiveness) [[39], [40], [41], [42], [43]]. Essentially, 10 % SDV means that 90 % of endpoints are not subject to SDV. While this approach may be able to identify some systematic data collection errors, most data errors will not be identified. Further, previous studies have demonstrated that remote monitoring can equal the event detection rate with SDV and that this rate will apply to all participant data and not only to the cases with SDV evaluation [42].
There are a number of limitations to the present study. First, services for the Pragmatic and Explanatory RCT designs in our models were imputed from previous studies without distinguishing between the explanatory trial's ‘need to have’ vs. ‘nice to have’ elements. Hence, our simulation results test scenarios and are dependent upon that degree to which those scenarios could be applied to the STRESS trial. Nonetheless, we believe that these services are generally representative of explanatory and pragmatic RCT designs and can be used to illustrate their potential differences with registry-based trials in the types of services required as well as their associated personnel hours and costs. Second, the registry-based design's registry access fee was substantial and contributed to that design's data management and statistics costs being higher than those for the pragmatic design. For studies with simpler site data collection requirements or with more complex registry data requirements and greater access fees, this could mean that a pragmatic design would be less costly than the registry-based design. Third, each RCT protocol has different design requirements and different research centers may be able to take advantage of existing registries to a greater or lesser extent. The Clinical Trials Transformation Initiative's Recommendations: Registry Trials report provides criteria for determining whether a registry (existing or proposed) is appropriate for embedding clinical trials [16]. Unfortunately, there is a dearth of research on the economics of clinical trials and the cost-benefit tradeoffs associated with different design strategies [44]. Hence, we do not have sufficient evidence to compare results with our study and to clearly describe how different RCT services may contribute to or detract from the economic attractiveness (cost and benefit) of different design elements.
5. Conclusions
We have demonstrated that total costs for the STRESS RCT with a registry-based design are less than those for a pragmatic design and much less than an explanatory design.
CRediT authorship contribution statement
Eric L. Eisenstein: Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Writing – original draft. Kevin D. Hill: Conceptualization, Investigation, Writing – review & editing. Nancy Wood: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Resources, Software, Writing – review & editing. Jerry L. Kirchner: Conceptualization, Investigation, Writing – review & editing. Kevin J. Anstrom: Writing – review & editing. Christopher B. Granger: Writing – review & editing. Sunil V. Rao: Writing – review & editing. H. Scott Baldwin: Writing – review & editing. Jeffrey P. Jacobs: Writing – review & editing. Marshall L. Jacobs: Writing – review & editing. Prince J. Kannankeril: Writing – review & editing. Eric M. Graham: Writing – review & editing. Sean M. O'Brien: Writing – review & editing. Jennifer S. Li: Conceptualization, Funding acquisition, Investigation, Supervision, Writing – review & editing.
Declaration of competing interest
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:
Eric L. Eisenstein: Nothing to disclose.
Kevin D. Hill, MD, MSCI: Nothing to disclose.
Nancy Wood, BS: Nothing to disclose.
Jerry L. Kirchner, BS: Nothing to disclose.
Kevin J. Anstrom, PhD, MS: Merck, Bayer, the National Institutes of Health, and the.
Patient-Centered Outcomes Research Institute.
Christopher B. Granger, MD: Consulting fees for the following companies: Abbvie, Abiomed, Alnylam Pharmaceuticals, Anthos, Bayer Corporation, Boehringer Ingelheim, Boston Scientific Corporation, Bristol Myers Squibb, Cardionomics, CeleCor Therapeutics, Cadrenal, Janssen Pharmaceutical, Medscape, LLC, Medtronic, Inc., Merck, Nation Institutes of Health, Novo Nordisk, Novartis Pharmaceutical Company, Pfizer, Philips, REATA, NephroSynergy. He also has salary funded by Duke grants sponsored by: Boehinger Ingelheim, Bristol Myers Squibb, Food and Drug Administration, Janssen Pharmaceuticals, Lilly Pharmaceutical, Novartis Pharmaceutical Company, Pfizer, and Philips. He has Equity in Tenac. io.
Sunil V. Rao, MD: Grants from National Heart, Lung, and Blood Institute, American College of Cardiology.
H. Scott Baldwin, MD: Nothing to disclose.
Jeffrey P. Jacobs, MD: Consultant to American Academy of Dermatology.
Marshall L. Jacobs, MD: Nothing to disclose.
Prince J. Kannankeril, MD, MSCI: Nothing to disclose.
Eric M. Graham, MD: Multiple NIH grants to perform research involving congenital heart disease.
Sean M. O'Brien, PhD: Nothing to disclose.
Jennifer S. Li, MD MHS: Nothing to disclose.
Funding Acknowledgement
The STRESS RCT was funded by the U01 grant to Duke University Titled: Leveraging existing registry resources to facilitate clinical trials (5U01TR001803). The present work also was funded in part by the Duke University – Vanderbilt University Medical Center Trial Innovation Center (U24TR001608), and supported by 1U24TR001579, U24TR001597, UL1TR002553, and U24TR001608 (NCATS, Trial Innovation Network). This work is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Data availability
Data will be made available on request.
References
- 1.Ashkanase J., Nama N., Sandarage R.V., et al. Identification and evaluation of controlled trials in pediatric cardiology: crowdsourced scoping review and creation of accessible searchable database. Can. J. Cardiol. 2020;36:1795–1804. doi: 10.1016/j.cjca.2020.01.028. [DOI] [PubMed] [Google Scholar]
- 2.Harris K.C., Mackie A.S., Dallaire F., et al. Unique challenges of randomized controlled trials in pediatric cardiology. Can. J. Cardiol. 2021 doi: 10.1016/j.cjca.2021.06.013. [DOI] [PubMed] [Google Scholar]
- 3.Mahony L., Sleeper L.A., Anderson P.A., et al. The Pediatric Heart Network: a primer for the conduct of multicenter studies in children with congenital and acquired heart disease. Pediatr. Cardiol. 2006;27:191–198. doi: 10.1007/s00246-005-1151-9. [DOI] [PubMed] [Google Scholar]
- 4.Morales D.L., Khan M.S., Turek J.W., et al. Report of the 2015 society of Thoracic Surgeons congenital heart surgery practice survey. Ann. Thorac. Surg. 2017;103:622–628. doi: 10.1016/j.athoracsur.2016.05.108. [DOI] [PubMed] [Google Scholar]
- 5.Jacobs J.P., Pasquali S.K., Austin E., et al. Linking the congenital heart surgery databases of the society of thoracic Surgeons and the congenital heart Surgeons' society: part 1--rationale and methodology. World J Pediatr Congenit Heart Surg. 2014;5(2):256–271. doi: 10.1177/2150135113519454. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Kumar S.R., Mayer J.E., Jr., Overman D.M., Shashidharan S., Wellnitz C., Jacobs J.P. The society of thoracic Surgeons congenital heart surgery database: 2021 update on outcomes and research. Ann. Thorac. Surg. 2021;112(6):1753–1762. doi: 10.1016/j.athoracsur.2021.10.002. [DOI] [PubMed] [Google Scholar]
- 7.Overman D.M., Jacobs M.L., O’ Brien J.E., et al. Ten years of data verification: the society of thoracic Surgeons congenital heart surgery database audits. World J Pediatr Congenit Heart Surg. 2019;10(4):454–463. doi: 10.1177/2150135119845256. [DOI] [PubMed] [Google Scholar]
- 8.Burns K.M., Pemberton V.L., Pearson G.D. The pediatric heart network: meeting the challenges to multicenter studies in pediatric heart disease. Curr. Opin. Pediatr. 2015 Oct;27(5):548–554. doi: 10.1097/MOP.0000000000000260. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Torok R.D., Li J.S., Kannankeril P.J., et al. Recommendations to enhance pediatric cardiovascular drug development: report of a multi-stakeholder think tank. J. Am. Heart Assoc. 2018;7(4) doi: 10.1161/JAHA.117.007283. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Frobert O., Lagerqvist B., Olivecrona G.K., et al. Thrombus aspiration during ST-segment elevation myocardial infarction. N. Engl. J. Med. 2013;369:158–197. doi: 10.1056/NEJMoa1308789. [DOI] [PubMed] [Google Scholar]
- 11.Hess C.N., Rao S.V., Kong D.F., et al. Embedding a randomized clinical trial into an ongoing registry infrastructure: unique opportunities for efficiency in design of the Study of Access site for Enhancement of Percutaneous Coronary Intervention for Women (SAFE-PCI for Women) Am. Heart J. 2013;166:421–428. doi: 10.1016/j.ahj.2013.06.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Huang S.S., Septimus E., Kleinman K., et al. Targeted versus universal decolonization to prevent ICU infection. N. Engl. J. Med. 2013;368:2255–2256. doi: 10.1056/NEJMoa1207290. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Lauer M.S., d'Agostino R.B., Sr. The randomized registry trial-the next disruptive technology in clinical research? N. Engl. J. Med. 2013;369:1579–1581. doi: 10.1056/NEJMp1310102. [DOI] [PubMed] [Google Scholar]
- 14.James S., Rao S.V., Granger C.B. Registry-based randomized clinical trials – a new clinical trial paradigm. Nat. Rev. Cardiol. 2015;12:312–316. doi: 10.1038/nrcardio.2015.33. [DOI] [PubMed] [Google Scholar]
- 15.Anderson B.R., Gotlieb E.G., Hill K., et al. Registry-based trials: a potential model for cost savings? Cardiol. Young. 2020;30(6):807–817. doi: 10.1017/S1047951120001018. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Clinical Trials Transformation Initiative CTTI recommendations: registry trials. https://ctti-clinicaltrials.org/wp-content/uploads/2021/06/CTTI_Registry_Trials_Recs.pdf.AccessedJun2023 Available at:
- 17.Margolis K.L., Crain A.L., Green B.B., et al. Comparison of explanatory and pragmatic design choices in a cluster-randomized hypertension trial: effect on enrollment, participant characteristics and adherence. Trials. 2022;23:673. doi: 10.1186/s13063-022-06611-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Hill K.D., Kannankeril P.J., Jacobs J.P., et al. Methylprednisolone for heart surgery in infants – a randomized, controlled trial. N. Engl. J. Med. 2022;387(23):2138–2149. doi: 10.1056/NEJMoa2212667. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Hill K.D., Kannankeril P.J., Jacobs J.P., et al. Methylprednisolone for heart surgery in infants – a randomized, controlled trial. N. Engl. J. Med. 2022;387(23):2138–2149. doi: 10.1056/NEJMoa2212667. Supplementary Material, Protocol. In. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Neumann P.F., Sanders G.D., Russell L.B., Siegel J.E., Ganiats T.G., editors. Cost-Effectiveness in Health and Medicine. Oxford University Press; New York: 2017. [Google Scholar]
- 21.Salomon J.A., Trikalinos T.A., Sanders G.D., Mandelblatt J.S. In: Cost-Effectiveness in Health and Medicine. Neumann P.F., Sanders G.D., Russell L.B., Siegel J.E., Ganiats T.G., editors. Oxford University Press; New York: 2017. Identifying and quantifying the consequences of interventions; p. 137. [Google Scholar]
- 22.Owens D.K., Siegel J.E., Sculpher M.J., Salomon J.A. In: Cost-Effectiveness in Health and Medicine. Neumann P.F., Sanders G.D., Russell L.B., Siegel J.E., Ganiats T.G., editors. Oxford University Press; New York: 2017. Designing a cost-effectiveness analysis; pp. 75–104. [Google Scholar]
- 23.Neumann P.J., Sanders G.D., Basu A., et al. In: Cost-Effectiveness in Health and Medicine. Neumann P.F., Sanders G.D., Russell L.B., Siegel J.E., Ganiats T.G., editors. Oxford University Press; New York: 2017. Recommendations on perspective for the reference case; pp. 67–74. [Google Scholar]
- 24.Hill K.D., Baldwin H.S., Bichel D.P., et al. Overcoming underpowering: trial simulations and a global rank endpoint to optimize clinical trials in children with heart disease. Am. Heart J. 2020;226:188–197. doi: 10.1016/j.ahj.2020.05.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Salomon J.A., Trikalinos T.A., Sanders G.D., Mandelblatt J.S. In: Cost-Effectiveness in Health and Medicine. Neumann P.F., Sanders G.D., Russell L.B., Siegel J.E., Ganiats T.G., editors. Oxford University Press; New York: 2017. Identifying and quantifying the consequences of interventions; pp. 137–166. [Google Scholar]
- 26.Li J.S., Eisenstein E.L., Grabowski H.G., et al. Economic return of clinical trials performed under the pediatric exclusivity program. JAMA. 2007 Feb 7;297(5):480–488. doi: 10.1001/jama.297.5.480. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Eisenstein E.L., Collins R., Cracknell B.S., et al. Sensible approaches for reducing clinical trial costs. Clin. Trials. 2008;5(1):75–84. doi: 10.1177/1740774507087551. [DOI] [PubMed] [Google Scholar]
- 28.Marquis-Gravel G., Roe M.T., Turakhia M.P., et al. Technology-enabled clinical trials: transforming medical evidence generation. Circulation. 2019;140(17):1426–1436. doi: 10.1161/CIRCULATIONAHA.119.040798. [DOI] [PubMed] [Google Scholar]
- 29.Inan O.T., Tenaerts P., Prindiville S.A., et al. Digitizing clinical trials. NPJ Digit Med. 2020;3:101. doi: 10.1038/s41746-020-0302-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Usman M.S., Van Spall H.G.C., Greene S.J., et al. The need for increased pragmatism in cardiovascular clinical trials. Nat. Rev. Cardiol. 2022;19(11):737–750. doi: 10.1038/s41569-022-00705-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Garza M., Myneni S., Nordo A., Eisenstein E.L., et al. eSource for standardized health information exchange in clinical research: a systematic review. Stud. Health Technol. Inf. 2019;257:115–124. [PubMed] [Google Scholar]
- 32.Eisenstein E.L., Garza M.Y., Rocca M., et al. eSource-enabled vs. traditional clinical trial data collection methods: a site-level economic analysis. Stud. Health Technol. Inf. 2020;270:961–965. doi: 10.3233/SHTI200304. [DOI] [PubMed] [Google Scholar]
- 33.Concato J., Stein P., Dal pan G.J., et al. Randomized, observational, interventional, and real-world-What’s in a name? Pharmacoepidemiol. Drug Saf. 2020 Nov;29(11):1514–1517. doi: 10.1002/pds.5123. [DOI] [PubMed] [Google Scholar]
- 34.Sertkaya A., Wong H.-H., Jessup A., Beleche T. Key cost drivers of pharmaceutical clinical trials in the United States. Clin. Trials. 2016;13(2):117–126. doi: 10.1177/1740774515625964. [DOI] [PubMed] [Google Scholar]
- 35.Sertkaya A., Birkenbach A., Berlind A., Eyraud J. U.S. Department of Health and Human Services; Washington DC: July 25, 2014. Examination of Clinical Trial Costs and Barriers for Drug Development. [Google Scholar]
- 36.Olivier C.B., Bhatt D.L., Leonardi S., et al. Central adjudication identified additional and prognostically important myocardial infarctions in patients undergoing percutaneous coronary intervention: results from CHAMPION PHOENIX. Circ Cardiovasc Interv. 2019;12 doi: 10.1161/CIRCINTERVENTIONS.118.007342. [DOI] [PubMed] [Google Scholar]
- 37.Felker G.M., Butler J., Januzzi J.L., et al. Probabilistic readjudication of heart failure hospitalization events in the PARAGON-HF study. Circulation. 2021;143:2316–2318. doi: 10.1161/CIRCULATIONAHA.121.054496. [DOI] [PubMed] [Google Scholar]
- 38.Haynes B. Can it work? Does it work? Is it worth it? The testing of healthcare interventions is evolving. BMJ. 1999;319(7211):652–653. doi: 10.1136/bmj.319.7211.652. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Baigent C., Harrell F.E., Buyse M., et al. Ensuring trial validity by data quality assurance and diversification of monitoring methods. Clin. Trials. 2008;5:49–55. doi: 10.1177/1740774507087554. [DOI] [PubMed] [Google Scholar]
- 40.Olsen R., Bihlet A.R., Kalakou F., et al. The impact of clinical trial monitoring approaches on data integrity and cost--a review of current literature. Eur. J. Clin. Pharmacol. 2016 Apr;72(4):399–412. doi: 10.1007/s00228-015-2004-y. Epub 2016 Jan 4. [DOI] [PubMed] [Google Scholar]
- 41.Manasco G., Pallas M., Kimmel C., et al. Risk-based monitoring versus source data verification. Appl. Clin. Trials. 2018;27:11. [Google Scholar]
- 42.Yamada O., Chiu S.-W., Takata M., et al. Clinical trial monitoring effectiveness: remote risk-based monitoring versus on-site monitoring with 100% source data verification. Clin. Trials. 2021;18(2):158–167. doi: 10.1177/1740774520971254. Apr. Epub 2020 Dec 1. [DOI] [PubMed] [Google Scholar]
- 43.Manasco P., Bhatt D.L. Evaluating the evaluators – developing evidence of quality oversight effectiveness for clinical trial monitoring: source data verification, source data review, statistical monitoring, key risk indicators, and direct measure of high-risk errors. Contemp. Clin. Trials. 2022 Jun;117 doi: 10.1016/j.cct.2022.106764. Epub 2022 Apr 15. [DOI] [PubMed] [Google Scholar]
- 44.Speich B., von Niederhäusern B., Schur N., et al. Systematic review on costs and resource use of randomized clinical trials shows a lack of transparent and comprehensive data. J. Clin. Epidemiol. 2018 Apr;96:1–11. doi: 10.1016/j.jclinepi.2017.12.018. Epub 2017 Dec 27. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Data will be made available on request.