Diabetes Technology Society Report on the FDA Digital Health Software Precertification Program Meeting

Abstract

Diabetes Technology Society (DTS) convened a meeting about the US Food and Drug Administration (FDA) Digital Health Software Precertification Program on August 28, 2018. Forty-eight attendees participated from clinical and academic endocrinology (both adult and pediatric), nursing, behavioral health, engineering, and law, as well as representatives of FDA, National Institutes of Health (NIH), National Telecommunications and Information Administration (NTIA), and industry. The meeting was intended to provide ideas to FDA about their plan to launch a Digital Health Software Precertification Program. Attendees discussed the four components of the plan: (1) excellence appraisal and certification, (2) review pathway determination, (3) streamlined premarket review process, and (4) real-world performance. The format included (1) introductory remarks, (2) a program overview presentation from FDA, (3) roundtable working sessions focused on each of the Software Precertification Program’s four components, (4) presentations reflecting the discussions, (5) questions to and answers from FDA, and (6) concluding remarks. The meeting provided useful information to the diabetes technology community and thoughtful feedback to FDA.

Digital health is an emerging paradigm for medical devices to communicate. Digital health can include communication with other devices, with patients through software known mobile applications residing on smartphone, or with the Internet. FDA has announced efforts to develop new programs to ensure access to safe and effective digital health products. One of these programs will be the FDA Digital Health Software Precertification Program. FDA intends to focus on software as a medical device (SaMD), defined as software used for medical purposes without being part of a hardware medical device, for the initial phase of the Precertification Program. FDA is not defining software as being limited to SaMD. In the past few months, FDA has published multiple updates to a working model document, outlining the current thinking of the agency about this program.

A meeting was convened on August 28, 2018, in Herndon, Virginia, by DTS to foster two-way communication between the US Food and Drug Administration (FDA) and clinical, academic, and industry communities to provide input to the FDA Digital Health Software Precertification Program. DTS intended this meeting to be an opportunity to understand FDA’s needs and provide feedback as FDA continues planning for the next phase of the pilot effort.

Background of the FDA Digital Health Software Precertification Program

The FDA intends for a future regulatory model to provide more streamlined and efficient regulatory oversight of software-based medical devices. This program will be known as the FDA Digital Health Software Precertification Program and is intended to be a voluntary pathway with a regulatory model more tailored to the software development lifecycle than the current regulatory paradigm. The goals of the Precertification Program are ensuring the safety and effectiveness of software technologies while supporting patient access to these technologies. The program is further intended to provide more streamlined and efficient regulatory oversight of software-based medical devices from manufacturers who have (1) demonstrated a robust culture of quality and organizational excellence and (2) committed to monitoring real-world performance of their products after they are on the US market.^1,2 This proposed approach will first focus on the software developer and/or digital health technology developer, rather than on the product, which is the FDA’s current approach to medical devices.

Software can be quickly adapted to respond to adverse events and other safety concerns. FDA intends to establish a new regulatory framework that is equally responsive if issues should arise. This is because the FDA’s current regulatory framework for devices can be slow in some cases when a more agile regulatory paradigm may be needed to accommodate rapid development and innovation in software-based products compared to what is needed for hardware-based products. The intent of the FDA Software Precertification Program is for software products from precertified companies to meet the same safety and effectiveness standard that the agency expects for products that have followed the traditional regulatory path to market, while improving timely access to innovative products. A pilot program focused on development of a precertification framework was announced by FDA in July 2017, and the FDA has worked with nine selected industry organizations over the past year to inform development of the Software Precertification Program.³ The term “software as a medical device” (SaMD) is defined by the International Medical Device Regulator’s Forum (IMDRF)⁴ as software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device.¹

FDA has indicated that the Digital Health Software Precertification Program will consist of four key components. FDA is currently deciding on specific policies for each component.

The first key component is developing policies for excellence appraisal and determination of the precertification level. This component will identify objective criteria and methods for precertifying a company and deciding whether a company can retain its precertification status. At this time, FDA is considering basing their determination of excellence on five culture of quality and organizational excellence (CQOE) principles. These include (1) patient safety, (2) product quality, (3) clinical responsibility, (4) cybersecurity responsibility, and (5) proactive culture. The current plan is for software developers to be assessed by FDA or an accredited third party for the quality of their software design, testing, clinical practices, and real-world performance monitoring and for other appropriate capabilities to qualify for a more streamlined premarket review. The FDA is also currently considering two levels of precertification based on (1) how a company meets the excellence principles and (2) whether the company has demonstrated a track record of delivering products. Based on whether a company does not have or does have such a track record, then the company will be classified and regulated in this program respectively as a Level 1 or Level 2 software developer.

The second key component is determining the review pathway by way of a risk-based framework. This approach will determine the premarket review pathway for a software product. With this regulatory program in place, a precertified company could market a lower-risk device either without FDA premarket review or with only a streamlined premarket review based on both the company’s precertification level and the level of risk. FDA intends to leverage the IMDRF risk categorization framework in determining the risk level of a product, focusing on the purpose of the software and the medical condition that it is intended to treat.¹

The third key component is developing a streamlined review process. This component of the Software Precertification Program will require identifying the necessary information that must be submitted for software to be reviewed for determining safety and effectiveness.

The fourth component is deciding how to monitor real-world performance in the postmarket setting. FDA wants to use some real-world information about SaMD to verify ongoing excellence of precertified companies as well as to identify emerging risks. Various objective and subjective types of data, including outcome results, could be selected for incorporation into this part of the program’s evaluation process.

FDA has stated that the fourfold goal of the Software Precertification Program is to deliver nonburdensome regulatory oversight that (1) identifies precertified developers of SaMD that can be trusted to develop high-quality SaMD products, (2) monitors product performance across the entire lifecycle of SaMD, (3) delivers a streamlined premarket review, and (4) verifies the continued safety, effectiveness, and performance of SaMD in real-world use with postmarket data.

FDA’s five aims for this program are to (1) benefit a participating precertified software-developing organization by offering streamlined premarket review as well as structured collection and review of real-world postmarket data, (2) create a streamlined review process for SaMD that is no less safe and effective as the current review process, (3) enable a regulatory framework that can accommodate software iterations, (4) ensure high-quality software products throughout the lifetime of the products, and (5) adapt itself according to its effectiveness.

The Software Precertification Program is intended to demonstrate that participating organizations have documented the ability to build, test, monitor, and proactively maintain and improve the safety, efficacy, performance, and security of their medical device software products, so that they will meet or exceed existing FDA standards of safety and effectiveness. Anticipated benefits for various stakeholders from the FDA Digital Health Software Precertification Program can be found in Table 1 (adapted from the FDA Working Model²).

Table 1.

Anticipated Benefits for Various Stakeholders From the FDA Digital Health Software Precertification Program.

	End user	Business	FDA	Payor	Investor
	Patients, providers, caregivers	SaMD developer	Agency reviewer	Insurance provider	Venture capitalist
Enhanced confidence in organizations developing SaMD products	+		+	+	+
Improved quality/safety/proactivity to address known and emerging risks	+	+	+	+
Timely availability of solutions to patients	+	+	+	+	+
Enhanced regulatory simplicity and experience		+	+	+	+
Business simplicity—faster/timely market access	+	+	+		+

Source: Adapted from the FDA Working Model.²

FDA has been seeking input from the public on the four components of the program, which they have also referred to as four streams. These include (1) excellence appraisal and determining precertification level, (2) review pathway determination, (3) the streamlined premarket review process, and (4) real-world performance.

Meeting Format

Prior to the FDA Digital Health Software Precertification Program Meeting, FDA and DTS together developed and distributed four sets of challenge questions for meeting attendees to address. These are contained in Appendix A.

At the meeting on August 28, 2018, the agenda included introductory remarks, a Software Precertification Program overview presentation from FDA, working breakout sessions that focused on each of the four components of the Software Precertification Program (excellence appraisal, review determination, streamlined review, and real-world performance), and then four presentations reflecting the discussions and conclusions of the working sessions. All of the 48 meeting participants (Figure 1) were given the opportunity to contribute to each of the presentations. Following all these presentations, FDA officials addressed topics that they selected from a set of questions and a set of recommendations developed by the participants during the working sessions. The full sets of questions and recommendations are set forth in Appendix B and Appendix C of this article.

Figure 1.

Meeting participants at the FDA Digital Health Software Precertification Program Meeting.

Summaries of Presentations

Introductory remakes were provided by David Klonoff, Medical Director, Diabetes Research Institute at Mills-Peninsula Medical Center (Meeting Chair); Kong Chen, Acting Section Chief, Energy Metabolism Section, Diabetes, Endocrinology, and Obesity Branch of NIDDK (NIH); and Allan Friedman, Director of Cybersecurity Initiatives at the National Telecommunication and Information Administration (NTIA), US Department of Commerce. The keynote address was delivered by Bakul Patel, Associate Center Director for Digital Health at FDA, after which he answered questions from the audience. The participants’ comments in this report are all summarized from a meeting transcript.

Bakul Patel reviewed the current working model of the Precertification Program, many details of which are contained the background section of this report. He explained how the program was intended to create an ecosystem of trust, transparency, and verification. He described the four components of the program and the five excellence principles to qualify for precertification. Patel provided many details of the program according to FDA’s current thinking. He mentioned that the FDA Digital Health Software Precertification Program could be considered to be similar to the TSA Precheck program, where a participant shares information with TSA in exchange for a better airport experience.

Patel stated that most devices coming to FDA currently contain software that generates data. He also mentioned that approximately 1000 new health-related apps are introduced through the Apple Store monthly, and while many are general wellness products that do not meet the definition of a medical device, FDA expects increasing numbers of SaMD products in the near future.

Patel mentioned that there have been questions raised as to how this new process will differ from the 510(k), de novo, and Premarket Approval Application (PMA) pathways. Much of the content that the FDA will analyze is the same, but they will need to figure out how they can look at the content differently. The key factor may be reducing overlap. Something addressed in excellence appraisal may not need to be reviewed again in the review part of the process.

Finally, Patel stated that FDA is seeking input from industry about what would make the program valuable to them. For most software developers, the advantage will be streamlined review and the ability to make changes to a product once it is already on the market. The FDA wants to hear from clinicians as well. Precertification should be advantageous to them in that it may help clinicians recommend products to patients.

Allen Friedman commented on open source software. The US Department of Commerce feels very strongly that there should not be a separate solution for the energy sector, for the national security sector, and for the health care sector, and thinks that this is something that all sectors of the economy work on together. What we need to deal with is third party code that we do not know much about. Often we are not using the latest up to date products or else we are using an up to date product that does not use the latest components within it. Open code can help developers understand the risks of the software they are using and facilitate a secure development process.

Summaries of Conclusions About Four Streams From Breakout Groups

This section is a recap of the discussions in the breakout groups, rather than a review of the components of the program. Each of the 48 meeting attendees participated in a breakout group of six attendees for each of the four streams.

A pair of facilitators for each stream reviewed the conclusions of all the eight groups for that stream. These facilitators prepared a summary, which reflected all the groups’ ideas and then gave a presentation of the conclusions for all the attendees for that stream. Each of the four following sections is a summary of a presentation about one of the four streams delivered by the facilitators. The four summary statements reflect concerns and questions posed to FDA, but do not reflect FDA’s responses and clarifications.

Stream 1: Excellence Appraisal

The excellence appraisal portion of the program describes the criteria and standards that a company would be expected to demonstrate in order to obtain precertification.

During the meeting, when evaluating the excellence appraisal framework, many companies questioned how the FDA will give credit to existing recognized standards (eg, ISO 13485) in their review of company excellence. At this time, it is unclear which standards, if any, could serve as predictors of success in the program. Additionally, most standards address competence, or the minimum acceptable level of performance, whereas the FDA hopes that companies will use precertification as an opportunity to demonstrate excellence. The group noted that many standards do not relate to the specific intended uses of medical products and that this difference in intended use could lessen the relevance of meeting such standards. Participants also suggested that the FDA may want to consider using an advisory committee that reviews excellence appraisal applications. Alternatively, an advisory committee could advise on a one-time basis whether a standard was relevant and now it could be used.

Stakeholders also noted that given the public relations impact of FDA’s approval or clearance of SaMD under the existing regulatory framework, companies need to know how the FDA will similarly seek to instill, encourage, and inspire trust in the public about products that come to market through the Software Precertification Program. Trust must be earned over time. The public will need to hear that the program itself is promoting safety and effectiveness. FDA can consider two strategies. The first is to collect data on how products that have gone through precertification have performed against an objective measure of safety and effectiveness and then compare those products with other products that go either traditionally or concurrently through the 510(k) process. The expectation is that there would be either no difference or the precertified company’s products would turn out to be even safer or more effective than the ones that go through the 510(k) process. The second strategy is to collect data from multiple companies and synthesize/release that aggregated data with a similar comparison between outcomes of products as a group that went through different regulatory pathways. Many participants suggested that the FDA compile real-world data about the participants that are successful in the Software Precertification Program and identify factors that are correlated with success. This will help the FDA advise other companies that are seeking precertification.

Finally, participants asked if there are novel approaches to machine learning and artificial intelligence that, if incorporated into a product that is approved or cleared by the FDA, could become a precedent for other clearances or approvals. The consensus was that each algorithm is unique, making precedence unlikely, although most small groups did not express an opinion on that topic.

Stream 2: Review Pathway Determination

In this stream, stakeholder discussion focused on how FDA should distinguish between “major” and “minor” changes to software, how should “major” and “minor” be defined, and when and how manufacturers should provide the FDA with notice of such changes. With respect to the characterization of changes, stakeholders agreed that three types of “major” changes included (1) changes in intended use, (2) any change that is high risk (major impact on safety or effectiveness of the device), and (3) any change that the FDA and the manufacturer agree will be considered a “major” change during initial communications with the FDA. Any change that does not meet at least one of these three criteria for characterization as a “major” change should be classified as a “minor” change.

Consistent with the above, stakeholders recommended that FDA communicate with manufacturers during the precertification process to ensure agreement regarding what constitutes a “major” or “minor” change to the product. Predetermining the characterization of changes as major or minor will help both manufacturers and the FDA prepare for future regulatory submissions, where required. An emerging concept called semantic versioning might be used to distinguish major and minor changes.

Stakeholders also discussed the logistics of how companies will report changes to the FDA. Some stakeholders asked whether the FDA should be informed about new products and major changes to existing products from precertified companies if such products do not undergo premarket review. Moreover, when the FDA does need to be notified, how would manufacturers be expected to provide such notice? A framework that could be used by FDA is the Certified Health IT Product List, which lists software that is acceptable for Medicare claims. Another framework that FDA could use to determine whether a change is minor or major is OKR—objectives and key results—to assess whether the purpose of a software product is being accomplished.

Finally, some suggested that inclusion of universal device identifiers (UDIs) would be a useful addition to an updated registration and listing database for precertified software. Stakeholders noted that it can be difficult to locate product-specific information in FDA’s existing registration and listing database, and that inclusion of a product-specific identifier (such as the UDI) could help stakeholders more quickly identify the information applicable to a specific device.

Stream 3: Streamlined Review

To evaluate the benefits of participating in the Software Precertification Program, it is critical that industry understand how the normal review process compares to the elements that will be required in a “streamlined” review process. For example, while presubmission meetings can expedite the substantive review of a product application, it is unclear whether the number of presubmissions that would be required to support a streamlined review will make the process truly streamlined or more cumbersome, depending on what is required. Additionally, they would like to know the expected number of rounds of presubmission review for specific product types, and which products are unlikely to require any prior presubmission before entering the streamlined review. Companies need to be able to plan, and clarification of the specific timelines for the review pathway and the streamlined review pathway (through clearance of the product) will be helpful.

What elements should the FDA be considering during streamlined review to ensure safety and efficacy? Two themes emerged during discussion. The first approach was to look at what the FDA reviews as part of its “normal” review process and retain only those software elements that are highest risk per a hazard analysis. Then implement mitigations for those risks. The FDA would review only those high-risk elements of the software.

The second approach is that if FDA is satisfied with a company’s processes and approach to development and validation of products, then FDA’s focus during streamlined review should be exclusively on data used to support claims or usability of the product because the development of studies would have already been discussed with FDA in advance of the streamlined review. In that case, much of the information that FDA considered during the decision to precertify the company could be eliminated from the streamlined review process. The FDA should focus on the data that are needed to support claims and demonstrate the usability of the product. Human factors usability evaluation was a point of discussion on this theme. This is a product feature that is very relevant to the diabetes industry because many of the products are used by lay users. Current usability evaluations are subjective, but objective and systematic evaluations are needed.

Human factors evaluation is a point of tension for many companies because there is limited human factors staff at FDA, so that tends to be an area where reviews may slow down. It was suggested that FDA hire additional personnel familiar with human factors assessments to better enable streamlined review,

There was a request that there be efforts within FDA to align the streamlined review with expectations of post market compliance. This means good alignment between what is agreed to during the review determination and then the streamlined review and then how companies may be assessed during an inspection.

There was a specific proposal that came from one of the groups for developing decision trees for assessing whether, if a product happens to fall into a category, future streamlined review would be needed for modifications. Companies want greater certainty about when they might need to go back to the FDA and where they can work with the FDA to set the parameters around those changes.

Clinicians also mentioned that it would be helpful to know that the FDA had considered cybersecurity. Cyber vigilance is an element of the excellence appraisal, so there was not necessarily a consensus on what that would mean for streamlined review, but it is something that should be taken into account during the review. Finally, the streamlined review should be in line with the total product lifecycle approach.

Stream 4: Real-World Performance

Participants agreed that this area is much more concrete than the others. The key decisions that need to be made for the Software Precertification Program relate to the following two questions:

1. Which type of real-world performance data should be monitored by manufacturers and which should be monitored by the FDA?

2. Should a company that meets a higher level of certification have the same data requirements for real-world reporting and monitoring as a company that meets a lower level of precertification?

Real-world performance data that they know well and can explain should be easily collected so that they can be used to articulate how risk is managed in these products. Having access to adverse events data that are complete and accurate is a critical part of showing that devices are safe and effective. For review efforts to be most effective, the data should relay utilization and outcome detail of the product features. Therefore, it is essential to share this data with the FDA. Make sure it is complete and accurate and then have a great way of dealing with it.

For review efforts to be most effective, the data should relay utilization and outcome details of the product features. What is the software telling patients and providers and what is the impact of this software. How often are features being used, so regulators can know where to focus their review? This will determine the impact of a given feature or features and show which are used most often and which are most beneficial to the patient. Many lower-risk products will not be proactively assessing downstream clinical outcomes.

There is a cost to collecting real-world performance data. Companies will consider the cost when deciding whether to go through the precertification process or the 510(k) process. Real-world data collection should focus on quality, not quantity. We think companies should be rewarded for providing high-quality data. It will be easy for companies to provide a mountain of data, but it will help both the FDA and industry to focus less on volume and more on quality data. The data must be accurate and should relay utilization and outcome details of the product.

Participants agreed that real-world data requirements should depend on the risk level of the intended product claim. All companies, regardless of level of precertification, should expect the same rigor for postmarket review. The group also addressed the topic of whether organizations that meet a higher level of precertification should have the same requirements for real-world reporting as organizations with a lower level of precertification. The consensus was no they should not have the same requirements. Although Level 1 and Level 2 companies should expect the same rigor for real-world data reporting, these two types of companies might report different frameworks of results, or else results that best fit their identity and function as a software product or company. For precertification, Level 1 companies might tend to focus their real-world data on premarket related data because there they just don’t have the breadth of information in the marketplace yet, and Level 2 companies might tend to emphasize postmarket data because they will have a lot of history to report that demonstrates their qualities as a company and what type of data they can deliver.

While stakeholders acknowledged FDA’s desire to make real-world data readily accessible to the public, most participants agreed that performance data—particularly sensitive data—should be kept private (ie, confidential between the manufacturer and the FDA) until it is better understood how the public might interpret (or misinterpret) certain information collected by the FDA. In terms of the data to be collected, participants recommended that the FDA consider the intended use of the product, and based on such intended use, identify those results that best enable the agency to evaluate whether a product is performing as intended.

Industry expressed concern about whether certain types of data provided to FDA would become part of the public domain. Types of data that were concerning included raw data, information on a more extensive set of metrics than currently required, development processes long-term usage, complaints, the amount of technical support needed, and postmarket outcomes. Industry was not comfortable with these types of data (whether used to determine precertification or to verify ongoing excellence post-launch) possibly becoming public. Companies were concerned that consumers will not be able to understand the additional data that they will have access to for devices that are cleared through the Precertification Program compared to the lesser amount of information that will likely be available for devices cleared through other routes for device clearance.

Finally, participants were concerned about what should be an appropriate risk matrix for the FDA to use in determining which adverse outcomes should result in a loss of precertification status or other actions. Currently all of our history is about adverse events for products, but precertification is for a company. Reporting of adverse events for companies has to come with an appropriate context and review from the company when reporting to the FDA to explain the significance of the adverse event. (Participants recommended against structuring the program such that an adverse event in one product would trigger loss of precertification for all products. A precertified company with several products has to have a way of measuring the impact of an adverse event on all the company’s products and we do not want to penalize a beneficial product because of a product with a failure.

Questions for FDA About Precertification of Software as a Medical Device

At the conclusion of the meeting, FDA staff responded to several questions and comments posed by participants, including the following:

• 1. How will the four levels in the SaMD classification by IMDRF match up with the three FDA medical device classifications?

Response: In a future state the FDA will be thinking about international harmonization. The risk category framework proposed in the Software Precertification Program Working Model aligns with the framework described by the International Medical Device Regulators Forum. However, the FDA’s existing classifications remain relevant and somewhat controlling unless and until Congress changes the Food, Drug and Cosmetic Act. Higher risk categories could still require Premarket Approval (PMA) applications depending on the specific technology and intended use. Most SaMD are new innovative products that likely require De Novo reclassification requests.

• 2. How will review times be reduced?

Response: There are four ways that the FDA is considering reducing review times.

1. FDA is considering the list of items typically included in a premarket submission and will propose reducing to the most critical elements necessary for a streamlined review. Some items will be shifted into the excellence appraisal or used as real-world performance information.

2. FDA is also considering review processes that can be shifted earlier into the presubmission phase of review.

3. Reviews will be completed in a more interactive manner.

4. Some portions of review may be able to be automated.

Does precertification and the streamlined prereview process replace device classifications) and existing premarket review processes (eg, 510(k)s and PMAs)? Response: The Software Precertification Program does not replace device classification, as we know it today. It will be a voluntary program, and companies can still submit their products through the pathways that exist today.

• 3. With the design of the program, products made by precertification companies would have more postmarket requirements than devices that are not within the program. How will FDA work through this discrepancy?

Response: Precertified companies introducing SaMD products will have substantially reduced premarket requirements, and in exchange will be required to share postmarket performance metrics already collected by the organization.

• 4. Should software intended for a high-risk disease automatically be labeled as a high-risk product?

Response: It depends on the intended use and indications for use of the product. If the intended use is not high risk, then the product should not be considered high risk. The community should consider submitting comments to the docket about how risk should be considered.

• 5. How are recalls handled in the Software Precertification Program and can a modification of a software bug not lead to citation for a recall?

Response: The agency is aware that this needs to be addressed. Looking at the cybersecurity example, there are already similar concepts in place where the FDA provides criteria for determining when certain product modifications are not considered field corrections. The major issue is whether or not there are workflow changes and whether there are claims modifications. The broad-scale answer is that the current guidance already accounts for this. On a smaller scale, the answer may be more granular depending on what kind of software bug or software change is being considered. But there are existing mechanisms to be able to address this.

• 6. SaMD that is currently not regulated should remain unregulated.

Response: This program pertains to medical devices that are classified as medical devices currently and are not under enforcement discretion. The FDA is working on mapping the framework, not expanding the definition of a medical device for this program.

• 7. Can a company without a medical background be included in the program?

Response: Any company wishing to participate would need to demonstrate all five excellence principles. If a company meets all of the principles, including clinical responsibility, then the company could be placed into level 2 precertification.

• 8. Excellence means a high bar. Is FDA trying to get just the elite in this category? Or is better than average good enough? Or is it just for any company better than the bottom scum?

Response: Cisco Vicenty at FDA expressed the personal opinion that the current system works for companies to demonstrate that they are compliant, but this system does not allow them to bring the best value to the product. If we looked tomorrow, without a transition time for trust and mind-set shift, then it would be a very tough transition for the current device space. When companies gain perspective and get fear out of the equation in terms of how they will interact with the FDA, then most should be able to shift quickly into a much more value and quality driven, productive mind-set. This takes time to implement.

• 9. If only 5% of companies engage in the program, will that be enough?

Response: This would represent about 1200 companies that are currently in the medical device space. We just need to get the process going. Anything that relieves the burden on FDA is great. As a resource constrained organization, this would allow us to focus more on other things, especially if these are companies that are submitting a lot of 510(k)s and PMAs.

Conclusion

The FDA officials and the participants expressed concluding thoughts. The FDA noted that presubmissions and informational meetings are welcome. FDA asked participants to formally express their concerns during the upcoming public comment period for the program. FDA stated that while hearing about the issues raised at the meeting is helpful, there exists a formal process and public comment period for stakeholders to also submit written questions/comments via the official website to go on record at the FDA.

Finally, participants voiced a need to continue the conversation and continue to include diverse groups in the discussions of this program. The level of collaboration between industry and the FDA on this program was encouraging to all parties involved and it is clear that the Precertification Program has a bright future.

Most participants agreed that the Software Precertification Program will lead to better information for the users about the quality of products being cleared. Clinicians will likely gain a greater understanding of what to recommend for their patients to be better able to advocate for patient interests.

The FDA Digital Health Software Precertification Program shows foresight from FDA in three key areas. The FDA (1) recognizes that there is a need for streamlined processes, (2) will face a deluge of mobile apps and needs a way to process them, and (3) sees that clinicians need real-world performance data. The FDA is on the right track and we are excited to see how this program develops.

Appendix A

Challenge Questions Distributed to Attendees Before the Meeting

Excellence Appraisal

• 1. How do we give credits as part of the appraisal process for either accreditations or standards or other? (FDA has some docket comments that address this.) Is it enough to say that companies are doing this in clinical responsibility?

• Output: A proposal for how to approach tackle credits for the appraisal process

• 2. What is it you want to see, or like to see, from organizations when they decide to use a specific product, frame for clinical decision support type products and we do not really know why they have not gotten traction? How can we help industry and the community have more trust going through this EA process? What would inspire trust in you, through precertification, that you would be willing to support? Please explain your answers

• Output: Defined list of items that would lead to confidence in an organization developing SaMDs with an explanation of why for each.

• 3. Are there specific novel approaches to developing SaMD, such as machine learning and artificial intelligence, such that if one product of this type were precertified, then any product using the same technology would also be precertified?

Review Determination

Major/minor changes in SaMD (more focused on industry and engineers)

1. How should FDA think about major versus minor changes in SaMD, how should “major” and “minor” be defined, and how should these changes be handled?

2. Should FDA be informed about new products, major changes, and minor changes from precertified organizations that do not undergo premarket review, and if so, how?

• Outputs: Feedback on how to address major/minor changes for SaMD within review determination

• Feedback on six elements from precert companies for registration and listing (more focused on doctors and nurses)

• ✓ Significance of the information provided by the SaMD to the health care decision

• ✓ State of the health care situation or condition

• ✓ Core functionality of the SaMD

• ✓ Device description including key technological characteristics

• ✓ Organization’s precertification level and other relevant information related to organizational excellence

• ✓ Real-world performance information about the SaMD, which complies with all applicable privacy and disclosure laws, including user privacy and manufacturer intellectual property rights.

• 3. What would you like to see from a public health perspective?

• Feedback on review determination components for registration and listing

Streamlined Review

• 1. What are the safety issues that you want to be sure FDA considers when looking at SaMD?

• Output: list of safety issues with explanation of why it’s important

• 2. When a precertified company submits a SaMD for review, the streamline review team will know about aspects of their software development process based on the excellence appraisal the company went through for precertification.

• ○ In order for streamline review to determine that a device is effective, what outcomes from the software development process should they check and why? For example, what outcomes of the configuration management and requirements development processes for software or hazard analysis should we know for a device during streamlined review?

• Output: List of outcomes from specific excellence appraisal items that streamline review should check and a rationale for why.

• ○ In order for streamline review to determine that a device is safe, what outcomes from the software development process should they check and why? For example, what outcomes of the requirements development, hazard analysis, and traceability processes for software should we know for a device during streamline review?

• Output: List of outcomes from specific excellence appraisal items that streamline review should check and a rationale for why.

Assessing Real-World Performance

Performance assessment requires development of metrics and methodology to analyze data and define acceptable real-world performance.

1. Which critical real-world performance data (RWPD) elements should be monitored by SaMD manufacturers, and how can the FDA ensure the methods they use to review and evaluate RWPD for precertification are robust, applicable, and understandable across different types of organizations?

2. Should RWPD requirements depend on the risk level of the intended product claim, and should an organization that meets a higher level of precertification have the same requirements for RWPD monitoring as an organization at a lower level of precertification?

3. What would be an appropriate risk matrix for FDA to use in determining which adverse outcomes should result in a loss of precertification status or other actions?

Appendix B

Questions Developed by Participants During the Meeting

	Questions for FDA about precertification of software as a medical device from the Diabetes Technology Society meeting, August 28, 2018
1	How will the four levels in the SaMD classification match up with the 3 medical device classifications?
2	Is it possible to get the specifics on how to demonstrate five areas of excellence?
3	How will the four levels in the SaMD classification match up with the 3 medical device classifications?
4	How will review times be reduced?
5	If a company is using a third party developer to code their software, then does the third party also need to be precertified?
6	Will quality system regulation (QSR) requirements be reduced for precertified companies?
7	Does precertification and the streamlined prereview process replace device classifications (I, II, III), 510(k)s, and PMAs?
8	Will FDA clearly identify Design History File and other documentation requirements if they differ from 510(k) and PMA requirements?
9	How will algorithm updates be treated?
10	How will cloud-based software changes be treated?
11	How will FDA encourage smaller companies without an existing business line and a fixed QMS to develop breakthrough/new innovations through regulatory procedures?
12	Right now SaMD will have more postmarket requirements than most devices that have software within the device. How will FDA work through this discrepancy?
13	A software algorithm can be changed within a short time. Sensor devices greater than a nanoscale can be changed within hours. How can FDA speed up their decision-making procedure?
14	FDA has not defined the ramifications around not having precertification or losing precertification. Has FDA defined what would constitute an event where a manufacturer would lose certification?
15	Inspectors looking at postmarket data or device decisions do not have the skillset to determine if you have a business/quality culture. How is FDA going to educate the field investigators so industry is not caught in the middle?
16	Should software intended for a high-risk disease automatically be labeled as a high-risk product?
17	Can a small company without a quality management system be eligible for the Precertification Program?
18	How can a precertified company without a quality management system deal with complaints?
19	How are recalls handled in the Precertification Program and can a modification of a software bug not lead to citation for a recall?
20	What can a company do if FDA wants to remove it from the Precertification Program?
21	To what extent will FDA review records in areas other than product quality in the Precertification Program?
22	Will FDA hire new staff with experience in business excellence to assess a company’s worthiness to enter the Precertification Program?
23	Will there be a user fee for the new Precertification Program?
24	How often will a company have to be reapproved to remain in the Precertification Program?

Appendix C

Recommendations Developed by Participants During the Meeting

	Recommendations for FDA about precertification of software as a medical device from the Diabetes Technology Society meeting, August 28, 2018
1	A detailed process for review and management of machine learning products that are allowed to evolve in real time.
2	SaMD that is currently not regulated should remain unregulated. Don’t move it to a more regulated space through a Precertification Program.
3	Define postmarket data. Patient blogs, etc cannot be substantiated and a manufacturer could spend time chasing down data that means nothing. FDA should clearly define what is in and what is out of scope.
4	Clear commitments on timelines for streamline review.
5	Ensure that information, especially coming from outside quality management systems (QMS), real-world performance analytics (RWPA), and key performance indicators (KPI) will be kept confidential and not subject to FOIA.
6	FDA should sufficiently resource the enforcement side of any new policy (not the case currently). There is a competitive disadvantage for a company following the regs competing with a company not following the regs that suffers no consequences.
7	FDA should establish clear timeline goals for streamlined review that should be shorter than existing goals.
8	Provide specific examples of what are the expectations for clinical evidence.
9	A commitment for timely feedback within less than 6 weeks (similar to the expedited access pathways [EAP]/Breakthrough Devices Program.
10	FDA needs to provide more defined benefits of the Precertification Program.
11	FDA needs to provide the review time that they will be held to for initial review and subsequent review.
12	Transparency in general is good, but stuff like KPIs and software bills of materials can be considered proprietary.
13	The FDA should provide a means to keep this information only available to FDA and not to other companies or the public.
14	FDA should make provisions for smaller companies with demonstrated postmarket procedures but not yet data (ie, premarket).
15	Ensure consistency across company size, experience, software location (SaMD vs SiMD), software application, etc.
16	FDA should ensure that technology companies moving into regulated software must also have a quality system (like ISO 13485 or 21 CFR820) as do existing medical device companies.
17	FDA should consolidate the review of diabetes products to one branch, instead of the current state of some products being reviewed by ODE and some by OIR. These teams have different levels of expertise and engagement and in some cases, products appear to be arbitrarily assigned.
18	FDA should provide transparency. If a product is classified for enforcement discretion, then these decisions should be public.
19	FDA should clarify the terms for dismissal from the program and a path for a company to be able to legally substitute their products after leaving 510(k).
20	FDA should allow precertified companies to make certain modifications without subsequent submissions if they follow a preapproved protocol.
21	Assurance is wanted that the scope of FDA inspections will not increase for companies in the Precertification Program.
22	Precertification should replace FDA inspections for quality system regulations.
23	Real-world performance data need to be clarified in more detail.
24	Software classified as combination products should be included in the scheme.
25	The process for obtaining precertification should be defined.
26	FDA should significantly reduce the review time to get “buy-in” from to the system.
27	FDA needs to clarify if a third party company who codes for a company also needs to be certified.
28	FDA should include software classified as combination products into the scheme.
29	FDA should clarify if device classification and approvals routes (eg, 510(k) and PMA) are replaced by precertification.
30	FDA should ensure that the scheme does not place a greater burden on industry than the current schemes.
31	SaMD relies on source code, APIs, reading/writing another company’s data to feed an algorithm. How will a company’s algorithm be protected despite extraneous factors outside the company’s control?
32	There has to be a clear precertification roadmap for companies bundling their products to provide a better patient offering. Precertification has to accommodate these instances thoughtfully and clearly, so partnerships happen and don’t stop.
33	Real-world performance will need to be specified ahead of approval such that there is not as much burden on companies.
34	FDA should be specific about how the real-world performance data are fed back to evaluate precertification and review frequency.
35	KPIs and other company information should not be made publicly available.
36	SaMD should be included in the Precertification Program.
37	FDA should indicate exactly what kind of real-world evidence it will make publicly available.
38	FDA needs to ensure that NEST, the Office of Compliance, and the Review Branch have all been trained on the Precertification Program and the streamlined review pathway.
39	FDA should provide more context for how a major versus a minor software change is defined.
40	FDA should indicate the timeframe for streamlined review and if the timeline is different for each International Medical Device Regulators Forum (IMDRF) risk level.