Table 5.

Factors with positive and negative influence on challenges with CED schemes for devices

	Challenge	Factors with positive influence	Factors with negative influence
1	Deciding which medical devices are candidates for CED schemes	There is a structured process leading to the identification of potential candidates for CED schemes Prioritization and inclusion of technologies into a scheme is made according to explicit and shared criteria The suitability of the proposed study protocol is a pre-condition to inclusion of a technology in a scheme The request to provide additional data is applied to all technologies for which relevant evidence gaps are identified during an assessment and the main responsibility of data collection falls on the manufacturers/applicants	HTA processes for devices are less formalized, commissioning mainly occurs at the local level A high number of devices and lack of horizon scanning processes to inform candidates for CED schemes of medical devices Optimal allocation of the funds for CED schemes is hampered by the fact that proposals are evaluated at different times over the year It is not easy to establish whether the available evidence is sufficient to initiate CED scheme or whether it is too early for reimbursement
2	Obtaining stakeholder agreement on the scheme	There exists a well-defined and structured processes for stakeholder engagement All details of the scheme, including the roles and obligations of the stakeholders involved are defined in a contractual agreement before scheme initiation Relationships with clinicians and manufacturers are facilitated if CED schemes are perceived as the only means to use the technology The responsibility to collect the data (and coordinate with participating centres and other stakeholders) fall on manufacturers/applicants	The complexity of CED schemes and the different expectations of the stakeholders involved require a strong and time-consuming coordinating effort For devices, it is more difficult to find patients to participate in public consultations during the scheme (e.g., compared to pharmaceuticals) In countries with small markets manufacturers may have a high bargaining power when discussing the conditions for the schemes
3	Securing funding for the scheme	Fixed budgets for CED schemes are granted on a periodic basis The additional costs of running a scheme fall upon the manufacturers/applicants	Lack of ad hoc funds and/or human resources to run the schemes
4	Determining the appropriate study design for data collection	The health authority can explicitly or implicitly mandate the type of study to be conducted Study design is defined by a third-party research institution CED schemes are mostly relying on routinely collected data A registry on the disease/device is already in place and suitable to answer the research questions	Setting up the research governance is usually complex, with several organizations involved and many practical questions to answer There may be disagreement on study design between the government, the manufacturers and the providers Selecting the centres that will collect data for the schemes may be problematic and time consuming Original patients’ informed consent for registries may not allow subsequent analyses of data
5	Determining the relevant outcome measure(s) on which data are collected	The health authority defines the primary and secondary outcomes. Those responsible for carrying out the research must justify if they do not follow the indication Clinicians and experts are involved from the onset in the definition of the outcomes Previous evidence from the literature or international collaborations (e.g., EunetHTA reports) already outlined the most relevant outcomes	Relevant safety and effectiveness issues are more difficult to identify for devices compared to drugs at the time of the evaluation Patient Reported Outcomes data are generally difficult to collect A balance is required between what outcomes would be desirable and what can be pragmatically collected by the participating centres Different stakeholders may disagree on the relative importance of the outcomes to be collected (e.g. surrogate versus patient relevant outcomes)
6	Dealing with data collection and monitoring	Data collection is based on routinely collected data from electronic sources (e.g., electronic health records) Feasibility of the data collection burden is discussed and agreed among all actors involved at the beginning of the scheme There is interoperability of data across data sources and research centres/providers Continuous follow-up is done to check the quality and validity of data submitted and to ensure meaningful data is being collected	There is less availability of routinely collected outcomes data for devices compared to pharmaceuticals Uncertainties on devices may require the collection of long-term outcome data, incompatible with the length of the scheme Having to deal with many low-volume centres with different experience may affect data quality, and increase the collection effort Hospitals/participating centres may lack incentives to provide timely and high quality data if they do not receive specific funding for this task Recruitment may be slower than expected affecting the time when the scheme reports its results
7	Dealing with data analysis	An independent research body is appointed for data analysis, including quality and risk of bias assessment There is an established experience with data analysis	Difficult to find adequate controls with observational studies Getting the analysis done and timely delivered may be difficult if no additional funds are provided for this task
8	Ex-ante definition of decision rule, based on possible outcomes of the scheme	Schemes are only about collection of new data Decision rules, including stopping rules during the schemes, and management of specific cases at the end of the data collection (e.g., insufficient quality of data, technology not effective) are defined in a contract agreed among all parties involved	Fixed decision rules at the onset may be affected by unforeseen changes in the devices or market dynamics
9	Reaching an agreement on price, reimbursement or use of the device at the end of the scheme	At the end of the scheme, technologies are re-evaluated according to business as usual evaluation procedures	The scheme may not have collected the planned data by the time of the reassessment, or data may be un-conclusory Relevant differences in (cost) effectiveness less clear among similar devices compared to pharmaceuticals
10	Withdrawing a device from the market when evidence indicates the device is not (cost-) effective	An exit strategy in case the technology is not (cost) effective is defined at the onset in a contract agreed between all stakeholders involved Having a well-designed scheme which produces scientifically robust results	Patients and manufacturers may challenge the withdrawal decision and take actions against it The management of explants for implantable devices in case the study outlines safety issues is complex
11	Obtaining agreements about the duration of the scheme and the stopping rule	The duration of the scheme is agreed based on the time that is needed to collect the required data and the characteristics of the disease/technology Continuation of the scheme is linked to periodic monitoring on its progresses	Adopting the stopping rules defined at the onset of the scheme may be difficult when the scheme is ongoing There is a tension between the short life-cycle of devices and the need for long-term outcomes Different perspectives among involved stakeholders (e.g. clinicians, manufacturers, NHS and HTA bodies) Slow recruiting may impact on the time when the study reports its results
12	Adapting the scheme to account for product modifications or a learning curve	The time frame of the scheme is relatively short to avoid product modifications Considerations on the eligibility of a device to a scheme also consider if newer generations of the same devices are expected in the short-term The company shares in advance available information on potential evolutions of the device and these are considered when discussing the study protocol Data on the effect of the learning curve is publicly available	There is little policy experience with how to deal with product modifications and/or learning curves Interpretation of results are confounded by product modifications that occur during the time-frame of the study Existence of a learning curve may complicate the selection process of participating centres in the scheme
13	Dealing with the market entry of similar devices	Schemes evaluate the class of devices or the procedure, not individual devices A scheme can involve multiple devices from different manufacturers Schemes are not comparative in nature. Any similar product entering the market may be requested to provide additional data or not based on their level of evidence Manufacturers of similar devices entering the market after scheme is initiated may be required to provide data to the same nationally-wide registry	Identifying similar devices entering the market is hampered by the difficulty to do horizon scanning for devices More rapid changes in clinical practice with devices compared to pharmaceuticals Inclusion of a new device entering the market when the scheme is ongoing may be more difficult than including it from the beginning

CED coverage with evidence developmen