Assessing the cost burden of United States FDA-mandated post-approval studies for medical devices

Abstract

Approved medical devices frequently undergo FDA mandated post-approval studies (PAS). However, there is uncertainty as to the value of PAS in assessing the safety of medical devices and the cost of these studies to the healthcare system is unknown. Since PAS costs are funded through device manufacturers who do not share the costs with regulators, we sought to estimate the total PAS costs through interviews with a panel of experts in medical device clinical trial design in order to design a general cost model for PAS which was then applied to the FDA PAS. A total of 277 PAS were initiated between 3/1/05 through 6/30/13 and demonstrated a median cost of $2.16 million per study and an overall cost of $1.22 billion over the 8.25 years of study. While these costs are funded through manufacturers, the ultimate cost is borne by the healthcare system through the medical device costs. Given concerns regarding the informational value of PAS, the resources used to support mandated PAS may be better allocated to other approaches to assure safety.

Introduction

Failure of medical devices, particularly permanent implanted devices, carries substantial risk of serious harm. Post-marketing surveillance is critical to evaluating the safety and efficacy of these devices, which generally enter the U.S. market with less clinical data than pharmaceutical agents.^{1, 2} A recent evaluation demonstrated that Food and Drug Administration (FDA) mandated post-approval studies (PAS) often fail to provide useful clinical information for either regulators or treating physicians.³ While the high costs of premarket studies have been documented⁴, there has been limited exploration of the costs of mandated PAS which are financially supported by device manufacturers, but whose costs are ultimately borne by healthcare consumers. Given recent efforts to improve the postmarketing surveillance of medical devices^{5, 6}, we sought to better understand the costs to manufacturers of FDA-mandated PAS. We hypothesized that the aggregate costs of PAS, as currently performed, may significantly exceed their value in helping to inform regulators, healthcare providers and the public as to the safety and efficacy of recently approved medical devices. Because the actual costs of PAS borne by the manufacturers are considered confidential business information, there is currently no publicly available inventory of such costs. Therefore, we sought to develop cost estimates for PAS for medical devices by generating a cost model that could be used to predict the cost of a proposed or completed PAS.

Methods

Overall Study Design

We sought to estimate the cost of recently FDA-mandated PAS studies by developing a cost estimation model through iterative structured interviews with domain area experts from clinical trial design arena, using a modified Delphi approach to develop cost estimation consensus⁷. Participants were interviewed in two phases, with the development of a cost estimation model after Phase I and calibration of the cost model after Phase II. The final cost model was then applied to the existing FDA PAS database to estimate the costs to manufacturers of PAS studies ordered by FDA during the study period of 2005 through 2013.

Interviews of Expert Participants and Development of Initial Cost Estimation Model

We interviewed experts from the medical device manufacturing industry, academic research organizations/clinical research organizations, and experts who act as site level investigators at clinical investigative centers. These experts were identified by the investigators based on reputation and expertise in the field of medical device development and investigation. A total of 14 subjects were invited to participate, 10 accepted and were interviewed (4 from the medical device industry, 4 from academic research organizations, and 2 from clinical investigative centers).

Experts were interviewed in two phases in accordance with a modified Delphi approach to developing consensus from an expert panel⁷. During the Phase I interviews, participating experts were asked structured open-response questions to determine how the participant estimates the costs of post-marketing surveillance studies and what prior experience or knowledge was used in the estimation. The participants were then asked structured questions asking them to provide numerical total cost estimates for fifteen hypothetical post-approval study scenarios in order to isolate how different study design factors would affect the participants’ estimates of the cost of the PAS scenario.

The results of the phase I interviews was used to construct a preliminary cost model, which was then re-calibrated based on the results of a second set of expert interviews. For the purposes of this exploration, we focused our estimates on the costs to the medical device industry related to execution of the post-approval studies. We explicitly excluded the cost of the approved medical device itself, as well as all costs associated with “usual care” that included the clinical evaluation of patients receiving the devices. During the Phase II interviews, participants were re-contacted and asked to estimate the costs of three additional scenarios in order to calibrate the model generated from Phase I.

PAS Study Scenarios and Cost Estimation Model Development

The 15 hypothetical PAS scenarios represented a broad spectrum of study types which varied the organ system, device type, study design (prospective randomized vs. registry), presence or absence of a concurrent control population, number of study sites, duration of subject follow-up and geography of study-site locations. The study design features that were altered in the hypothetical scenarios were selected based on the general availability of those features within the FDA PAS database (see below). Each scenario was designed such that only one study feature varied from the previous scenario in order to isolate the estimated impact of the change in the study design feature on the overall cost of the study. The 15 PAS scenarios are summarized in Table 1, and a detailed description of each scenario, as provided to the study participants are provided in Appendix 1

Table 1.

Summary of design features of the Phase I interview Post-Approval study scenarios.

	Scenario 1	Scenario 2	Scenario 3	Scenario 4	Scenario 5	Scenario 6	Scenario 7
Study Design Feature Adjusted	Base Case	Recruitment	Extend 2yr	Decr 50 sites	50% OUS	No Angio	Incr 4000 sub
Device Type	Coronary Stent	Coronary Stent	Coronary Stent	Coronary Stent	Coronary Stent	Coronary Stent	Coronary Stent
Subjects	1000	1000	1000	1000	1000	1000	5000
Sites	100	100	100	50	50	50	50
OUS %	0%	0%	0%	0%	50%	50%	50%
Eval per Year	1	1	1	1	1	1	1
Eval Type	Phone	Phone	Phone	Phone	Phone	Phone	Phone
Duration	3	3	5	5	5	5	5
Procedures	Angio	Angio	Angio	Angio	Angio	None	None
Organ System	CV	CV	CV	CV	CV	CV	CV
Recruitment	No	Yes	Yes	Yes	Yes	Yes	Yes
Rand omized	No	No	No	No	No	No	No
Control Group	No	No	No	No	No	No	No

	Scenario 8	Scenario 9	Scenario 10	Scenario 11	Scenario 12	Scenario 13	Scenario 14	Scenario 15
Study Design Feature Adjusted	ICD vs Stent	IABP vs ICD	Mesh vs IABP	TKR vs Mesh	Billiary vs TKR	Incr dev cost $5K	5000 controls	Randomize
Device Type	ICD Lead	IABP	Surgical Mesh	TKR	ERCP Stent	ERCP Stent	ERCP Stent	ERCP Stent
Subjects	5000	5000	5000	5000	5000	5000	5000	5000
Sites	50	50	50	50	50	50	50	50
OUS %	50%	50%	50%	50%	50%	50%	50%	50%
Eval per Year	1	1	1	1	1	1	1	1
Eval Type	Phone	Phone	Phone	Phone	Phone	Phone	Phone	Phone
Duration	5	5	5	5	5	5	5	5
Procedures	None	None	None	None	None	None	None	None
Organ System	CV	CV	CV	CV	CV	CV	CV	CV
Recruitment	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Randomized	No	No	No	No	No	No	No	Yes
Control Group	No	No	No	No	No	No	Yes	Yes

Abbreviations: Decr – Decrease number of enrolling sites by 50 from base-case; No Angio – follow up evaluates include only non-invasive studies; Incr 4000 sub – total subject recruitment increased 4000 above base-case, OUS% - proportion of enrolling sites outside of U.S.; Eval Type – type of annual follow up assessment (phone or in-person).

The initial cost estimation model was developed from the participating expert estimates for each scenario by calculating the change in the median of the cost estimate for the scenario as compared to an alternative scenario in which there was only one study design factor difference. We then assumed that the difference in median study costs was fully attributable to the single changed study feature. The interview process identified three principle categories responsible for overall PAS cost: the infrastructure costs associated with any research study (per study costs or study overhead), the size of the study (per subject costs) and the number of sites (per site costs). We therefore allocated the incremental cost for the change in one study feature to one of three major components: per study costs, per subject costs, and per study site costs based on the general budgeting approaches described by the domain expert participants. The initial cost estimation model was then fit to the median scenario cost predictions by iteratively adjusting weights for each cost component for each study feature. The weight adjustment process was continued until the cost model was able to generate estimates that accurately predicted the median overall cost estimates for the scenarios.

In Phase II the initial cost estimation model was applied to three new hypothetical PAS scenarios, which were then reviewed, using structured interview techniques, with the expert panel participants.

FDA PAS database and Estimated Total PAS Costs

The final cost estimation model was applied to each of the post-approval studies in a dataset provided by FDA reflecting all FDA-mandated post-approval studies for medical devices from 3/1/05 through 6/30/13, based on the publicly available PAS database, supplemented by a second data file provided by FDA to address missing data in the original database. Studies were excluded from the cost estimating exercise if they were categorized as “bench” or “laboratory” studies.

The costs estimates were intended to reflect the budgeted costs that would be anticipated if the study was conducted as originally planned. Therefore, no cost savings are assumed for incomplete or slowly enrolling studies, nor are cost overruns assumed for studies that are extended or expanded beyond the PAS filing as detailed to the FDA.

Results

Twelve experts, with a median professional experience of 14.5 years in clinical trial design (combined 149 years of experience), participated in the Phase I structured interview process. The participants reported leadership roles in the design or implementation of 139 post approval studies (median of 9.5 PAS per participant, interquartile range (IQR): 5.0-18.75). The participants rated the total number of subjects enrolled in a PAS, the frequency (and type of clinical follow-up), the use of randomization, and the inclusion of a concurrent control group as the most important trial design features influencing overall cost of the PAS. The participants rated their confidence in estimating the costs of PAS as 8.0 on a 10 point scale (IQR: 7.0 – 8.0).

PAS scenario 1 served as the base case on which all other scenarios were based, and was to include 1000 patients followed for three years following the implant of a coronary stent, the median estimate for the total cost of the study was $5.75 million (IQR: $3.35 - $9.25 million). An initial cost estimation model was developed and fit to the Phase I survey results, as described above, demonstrated excellent correlation (r² = 0.996) to the median expert estimated costs for the 15 hypothetical PAS scenarios (Figure 1). During the Phase II interviews, the cost estimation model was applied to three additional hypothetical PAS scenarios (see Appendix 2), and all 10 participants were re-interviewed to review the assumptions of the model and ask the participants to refine the model as applied to the new scenarios. The Phase II interviews confirmed consensus agreement of the cost estimation model by all 10 participants and there were no additional changes to the cost estimation model.

Figure 1.

Correlation of initial cost estimation model to median expert participant estimates

Table 2 provides details of the final cost estimation model which includes a simple “base-case” PAS that is then adjusted through assigning additional costs based on study features beyond those included in the base case. The base case study assumed two year extended follow-up of an existing cohort of 1,000 patients exposed to the device as part of premarket study of 100 enrolling centers, with phone surveys of the patients performed at 12 and 24 months. Additional study features included in the final model include: the need to recruit subjects, inclusion of control group, requiring randomization for treatment, type of clinical follow up (by phone, in person, with imaging or with an invasive study), extending the follow-up period beyond two years, adjustment of costs for very large studies as well as adjustments for the proportion of sites enrolling in the study that were located outside the U.S. and for those studies that were not of the cardiovascular system (which was universally identified by the participants as the highest cost medical device PAS to execute).

Table 2.

Final PAS cost estimation model.

		Per Subject Cost	Per Site Cost	Study Overhead	Notes:
Base Case:	Assumes a 2yr observational follow up of existing study cohort, with one phone evaluation per year	$600	$2,800	$400,000
Adjustments:	Need to Recruit Subjects	$2,600	$3,250	$325,000	De novo cohort required
	Adding a control group (1:1)	$325	$406	$40,625	Due to complexity of recruiting patients to two therapies
	Requiring Randomization	$1,300	$1,625	$162,500
	Additional Phone evaluations per year	$28	$129	$-	Per additional phone interview per year
	In-Person clinical evaluations	$138	$644	$-	Per additional in-person clinical evaluation per year
	Imaging study required	$1,000	$250	$50,000	Examples: Chest CT, renal ultrasound, echocardiogram
	Required invasive study	$2,000	$500	$100,000	Examples: coronary angiography, EGD, ERCP
	Extending additional year (w/ one eval per year)	$198	$924	$264,000	Per year costs
	Large Study Adjustments:
	Overhead Costs for >1000 subjects	$-	$-	$200,000	Per 1000 additional subjects
	Overhead Costs for >100 sites	$-	$-	$300,000	Per 100 additional sites
	Reduced base-case cost if OUS sites	$480	$2,240	$320,000	Based on proportion of study performed OUS
	Discount for non CV Organ system device study	$(120)	$(560)	$(80,000)

The FDA PAS dataset included 293 unique PAS approved between March 2005 and June 2013, of which 16 were excluded since they were categorized as “bench” or “laboratory” studies. Over the 8.25 years analyzed, a mean of 33 PAS were initiated in each year. As shown in Table 3, among the 277 PAS evaluated in this analysis, 203 (77%) were mandated in three organ systems: cardiovascular (138), orthopedics (39) and plastic surgery (36). Only 23 (8.3%) of studies included randomized control populations (the most expensive design, as estimated by expert participants), and 108 (39%) included no control group at all (the least expensive PAS design). To account for the very high predicted costs of three specific studies (1.1% of total PAS), a conservative assumption was used to cap the maximum budget for any individual PAS at the 99^th percentile of estimated costs of the PAS dataset; $35 million.

Table 3.

Results of final cost estimation analysis of the FDA PAS database

		Number of Studies	Percent of Total	Median Number of Subjects	Median Number of Sites	Median Duration (Years)	Median Cost per Subject		Median Cost per Site		Median Cost per Study		Total Cost - All Studies
All Studies		277	100.0%	320	20	5	$	3,243	$	5,891	$	2,151,792	$	1,224,578,176
Control Group	Active	128	46.2%	455	26	5	$	3,412	$	6,256	$	2,500,159	$	667,113,088
	Historical	41	14.8%	300	40	5	$	4,006	$	7,088	$	2,518,226	$	217,140,880
	None	108	39,0%	200	14	5	$	3,085	$	5,361	$	1,333,546	$	340,324,224
Follow-Up	In-Person	95	34.3%	350	25	5	$	3,686	$	6,929	$	2,547,148	$	529,545,024
	Phone Only	182	65.7%	320	20	5	$	3,080	$	5,366	$	1,870,425	$	695,033,152
Organ System	Cardiovascular	138	49.8%	350	40	5	$	3,410	$	6,365	$	2,393,066	$	652,936,128
	Opthamology	15	5.4%	360	20	5	$	3,243	$	5,510	$	2,605,352	$	46,522,336
	Orthopedic	39	14.1%	250	8	7	$	3,356	$	6,005	$	1,668,343	$	82,234,424
	Plastic Surgery	36	13.0%	902	14	5	$	3,083	$	5,361	$	1,752,964	$	247,068,128
	Other	49	17.7%	300	15	3	$	3,101	$	5,377	$	1,809,249	$	195,817,152
Outside US Sites	100% in US	267	96.4%	320	20	5	$	3,218	$	5,891	$	2,113,964	$	1,143,346,688
	<50% Outside US	4	1.4%	562	77	5	$	1,713	$	4,266	$	3,227,791	$	18,544,240
	>50% Outside US	6	2.2%	1778	150	5	$	5,379	$	10,191	$	12,387,804	$	62,687,228
Randomized	No	254	91.7%	325	20	5	$	3,237	$	5,743	$	2,131,993	$	1,112,402,560
	Yes	23	8.3%	290	26	5	$	3,243	$	7,301	$	2,547,148	$	112,175,656
Recruitment	No	138	49.8%	298	20	5	$	1,069	$	3,721	$	1,366,643	$	450,221,248
	Yes	139	50.2%	350	23	5	$	3,830	$	7,075	$	2,744,765	$	774,356,928

Application of the final cost estimation model to all 277 clinical PAS demonstrated an estimated median cost of $2.16 million per individual PAS with an overall budgeted cost of $1.22 billion during the 8.25 year time period. Therefore, the average “budgeted” cost of PAS was $146 million per year. The principal cost drivers of individual PAS were the number of subjects and study sites, as well as the inclusion of active control groups, the need for new subject recruitment, and randomization. Based on the distribution of these study features, there were substantial differences in the median cost per study as stratified by organ system (Figure 2).

Figure 2.

Median study costs ($) by organ system.

Conclusions

We conservatively estimate that more than $1.22 billion was planned to be spent on FDA-mandated PAS for medical devices approved between March 2005 and June 2013, representing more than $145 million per year anticipated to be spent by medical device manufacturers on these studies. This annual budgeted expenditure is intended to support the assessment of the safety and effectiveness of these recently approved devices. However, as recent investigators have reported, traditional PAS offer limited value from either a regulatory or clinical standpoint³. The sizeable annual expenses of PAS, borne by the public through the costs of medical devices themselves, may provide relatively modest informational value to the medical community, the public and industry regulators.

However, the FDA has proposed new strategies intended to improve the post-market safety assessment of medical devices^1,8. These include implementation of unique device identifiers, leveraging national clinical device registries, and further development of methodologies for “real time” monitoring of post-market information. Given the high costs and uncertain information value of traditional PAS, we advocate shifting resources from traditional PAS to support the building of a high-performance, continuous safety monitoring infrastructure for medical devices. Among the highest priority initiatives to consider, we would recommend focusing on cardiovascular devices (representing nearly 50% of PAS) and piloting the use of national registries to avoid the high costs of traditional patient recruitment. Collaboration between industry, FDA and other stakeholders to devise strategies to redirect current investment in PAS execution into higher value efforts would better serve the public interest and more efficiently invest the resources that are already being spent.

Abbreviations

PAS

Post-approval study

FDA

Food and Drug Administration

Appendix

Appendix 1

Phase I Interviews hypothetical post-approval study scenario descriptions.

Scenario 1

The proposed is a post-marketing surveillance study evaluating the efficacy of a new implantable cardiac device called Widget A. The device is a next generation coronary stent that is implanted in a cardiac catheterization laboratory. The proposed study will continue follow-up of a pre-market cohort of 1,000 patients against historic controls. Both men and women, ages 50-74 are included. The patients will be followed for 3 years, yearly by telephone, with mandated re-look angiography after 1 year. The study will be conducted at 100 US-only sites.

Scenario 2

Implanted cardiac device such as a coronary stent study as above, but the study population is a new cohort, not a pre-market cohort

The proposed is a post-marketing surveillance study evaluating the efficacy of a new implantable cardiac device called Widget A. The device is a next generation coronary stent that is implanted in a cardiac catheterization laboratory. The proposed study will evaluate a new cohort of 1,000 patients against historic controls. Both men and women, ages 50-74 are included. The patients will be followed for 3 years, yearly by telephone, with mandated re-look angiography after 1 year. The study will be conducted at 100 US-only sites.

Scenario 3

Implanted cardiac device such as a coronary stent study as above, but the follow-up period is extended by 2 years to 5 total

The proposed is a post-marketing surveillance study evaluating the efficacy of a new implantable cardiac device called Widget A. The device is a next generation coronary stent that is implanted in a cardiac catheterization laboratory. The proposed study will evaluate a new cohort of 1,000 patients against historic controls. Both men and women, ages 50-74 are included. The patients will be followed for 5 years, yearly by telephone, with mandated re-look angiography after 1 year. The study will be conducted at 100 US-only sites.

Scenario 4

Implanted cardiac device such as a coronary stent study as above, but the number of sites is halved from 100 to 50

The proposed is a post-marketing surveillance study evaluating the efficacy of a new implantable cardiac device called Widget A. The device is a next generation coronary stent that is implanted in a cardiac catheterization laboratory. The proposed study will evaluate a new cohort of 1,000 patients against historic controls. Both men and women, ages 50-74 are included. The patients will be followed for 5 years, yearly by telephone, with mandated re-look angiography after 1 year. The study will be conducted at 50 US-only sites.

Scenario 5

Implanted cardiac device such as a coronary stent study as above, but the sites are international as opposed to just US-only

The proposed is a post-marketing surveillance study evaluating the efficacy of a new implantable cardiac device called Widget A. The device is a next generation coronary stent that is implanted in a cardiac catheterization laboratory. The proposed study will evaluate a new cohort of 1,000 patients against historic controls. Both men and women, ages 50-74 are included. The patients will be followed for 5 years, yearly by telephone, with mandated re-look angiography after 1 year. The study will be conducted at 50 US and international sites.

Scenario 6

Implanted cardiac device such as a coronary stent study as above, but without mandatory re-look angiography. Instead, clinical follow-up with phone contact is performed for 3 years

The proposed is a post-marketing surveillance study evaluating the efficacy of a new implantable cardiac device called Widget A. The device is a next generation coronary stent that is implanted in a cardiac catheterization laboratory. The proposed study will evaluate a new cohort of 1,000 patients against historic controls. Both men and women, ages 50-74 are included. The patients will be followed for 5 years, with yearly phone contact only. The study will be conducted at 50 US and international sites.

Scenario 7

Implanted cardiac device such as a coronary stent study as above, but with a 5,000 patient cohort compared to a 1,000 patient cohort

The proposed is a post-marketing surveillance study evaluating the efficacy of a new implantable cardiac device called Widget A. The device is a next generation coronary stent that is implanted in a cardiac catheterization laboratory. The proposed study will evaluate a new cohort of 5,000 patients against historic controls. Both men and women, ages 50-74 are included. The patients will be followed for 5 years, with yearly phone contact only. The study will be conducted at 50 US and international sites.

Scenario 8

Implanted biliary device such as a biliary stent study as above, compared with a coronary stent

The proposed is a post-marketing surveillance study evaluating the efficacy of a new implantable biliary device called Widget B. The device is a next generation biliary stent that is implanted in an endoscopy suite during ERCP (endoscopic retrograde cholangiopancreatography). The proposed study will evaluate a new cohort of 5,000 patients against historic controls. Both men and women, ages 50-74 are included. The patients will be followed for 5 years, with yearly phone contact only. The study will be conducted at 50 US and international sites.

Appendix

Appendix 2.

An example of a Phase II Interview PAS cost estimation using initial cost model.

Scenario: The proposed study is a post-marketing surveillance study evaluating the efficacy of a new implantable cardiac pacemaker device called Pacer-A. The device is a next generation leadless pacemaker that is implanted in a cardiac electrophysiology laboratory. The proposed study will evaluate a new cohort of 250 patients who receive Pacer-A for bradycardia compared to 250 controls who will receive a traditional pacemaker for a similar indication. The subjects will not be randomized, but will be given the choice of either device. Both men and women, ages 50-74 are included. The patients will be followed for 3 years, with yearly telephone interviews as well as a required cardiac event monitor after Year 1. The study will be conducted at 30 US-only sites. Study endpoints will be independently adjudicated.
		Per Subject Costs		Per Site Costs		Study Overhead		Notes:
Base Case:	Assumes a 2yr observational follow up of existing study cohort, with one phone evaluation per year	$	600	$	2,800	$	400,000
Adjustments:	Need to Recruit Subjects	$	2,600	$	3,250	$	325,000	If de novo cohort required
	Adding a control group (1:1)	$	325	$	406	$	40,625	Due to complexity of recruiting patients to two therapies
	Requiring Randomization	$	1,300	$	1,625	$	162,500
	Additional Phone evaluations per year	$	28	$	129	$	-	Per additional phone interview per year
	In-Person clinical evaluations	$	138	$	644	$	-	Per additional in-person clinical evaluation per year
	Imaging study required	$	1,000	$	250	$	50,000	Examples: Chest CT, renal ultrasound, echocardiogram
	Required invasive study	$	2,000	$	500	$	100,000	Examples: coronary angiography, EGD, ERCP
	Extending additional year (w/ one eval per year)	$	198	$	924	$	264,000	per year costs
	Large Study Adjustments:
	Overhead Costs for >1000 subjects	$	-	$	-	$	200,000	per 1000 additional subjects
	Overhead Costs for >100 sites	$	-	$	-	$	300,000	per 100 additional sites
	Reduced base-case cost if OUS sites	$	480	$	2,240	$	320,000	Based on proportion of study performed OUS
	Discount for non CV Organ system device study	$	(120)	$	(560)	$	(80,000)

$2,500,000 $256,500 $1,343,700

Total cost: $4,100,000