Regulatory Considerations for Physiological Closed-Loop Controlled Medical Devices Used for Automated Critical Care: Food and Drug Administration Workshop Discussion Topics

Abstract

Part of the mission of the Center for Devices and Radiological Health (CDRH) at the US Food and Drug Administration is to facilitate medical device innovation. Therefore, CDRH plays an important role in helping its stakeholders such as manufacturers, health care professionals, patients, patient advocates, academia, and other government agencies navigate the regulatory landscape for medical devices. This is particularly important for innovative physiological closed-loop controlled (PCLC) devices used in critical care environments, such as intensive care units, emergency settings, and battlefield environments. CDRH’s current working definition of a PCLC medical device is a medical device that incorporates physiological sensor(s) for automatic manipulation of a physiological variable through actuation of therapy that is conventionally made by a clinician. These emerging devices enable automatic therapy delivery and may have the potential to revolutionize the standard of care by ensuring adequate and timely therapy delivery with improved performance in high workload and high-stress environments. For emergency response and military applications, automatic PCLC devices may play an important role in reducing cognitive overload, minimizing human error, and enhancing medical care during surge scenarios (ie, events that exceed the capability of the normal medical infrastructure). CDRH held an open public workshop on October 13 and 14, 2015 with the aim of fostering an open discussion on design, implementation, and evaluation considerations associated with PCLC devices used in critical care environments. CDRH is currently developing regulatory recommendations and guidelines that will facilitate innovation for PCLC devices. This article highlights the contents of the white paper that was central to the workshop and focuses on the ensuing discussions regarding the engineering, clinical, and human factors considerations. (Anesth Analg 2018;126:1916–25)

The traditional engineering domains such as aviation, automotive, and energy production have a persistent and formidable trend toward automating tasks and minimizing human intervention to increase efficiency, reduce costs, and prevent errors.^1,2 However, increased system automation in various fields has been shown to have its own pitfalls as a result of the new types of hazards, such as control system failure,³ automation bias (user’s tendency to accept computer recommendation without questioning its accuracy),^4,5 skill degradation,^4,6 lack of operational transparency, and increased risk arising from system complexity.^4,5,7 Therefore, designers and manufacturers of automated systems have developed rigorous mathematical models, controller synthesis techniques, and human factors engineering/usability methodologies to enable safe progress toward higher levels of automation (LOA). The automation systems used to control the commercial and military aircrafts are examples of a highly sophisticated system with a validated and acceptable safety profile.^1–3

There are also opportunities for medical devices that incorporate advanced LOA.^8–15 Automation applied to medical devices may produce analogous benefits and risks that have been observed in other industries but poses unique opportunities and challenges related to physiological closed-loop controlled (PCLC) devices. PCLC medical devices that automate therapy delivery have been emerging particularly for critical and emergency care environments due to extensive physiological monitoring and significant clinician cognitive overload and high-stress experiences in such environments. PCLC medical devices, referred to as PCLC devices in the remainder of this article,^a such as automatic anesthesia delivery, fluid resuscitation/vasopressor delivery, and mechanical ventilation (Table 1), are designed to deliver up-to-date therapy with improved and distraction-free performance. However, introducing automation and minimizing clinician involvement may incur new types of hazards that may necessitate mitigation. Algorithm flaws, automation bias, and lack of operational transparency are examples of potential automation-induced hazards.

Table 1.

Examples of PCLC Devices Discussed in the Workshop

PCLC Medical Device	Example of Sensed Physiological Variable	Control Objective	Actuated Therapy
Hemodynamic stability controller	Pulse pressure variability/mean arterial pressure	To maintain adequate cardiovascular performance	Titration of fluid volume infusion and/or vasopressors using
Mechanical ventilation/oxygenation controller	Blood oxygen saturation (Spo2)	To maintain peripheral oxygen saturation	infusion pump Fio2 adjusted by blender
Anesthesia delivery controller	End-tidal anesthetic concentration, EEG-based sensing, Spo2 respiratory rate	To maintain anesthesia. To maintain depth of hypnosis	Titration of anesthetics using infusion pumps or gas mixers

Abbreviations: EEG, electroencephalogram; Fio₂, fraction of inspired oxygen; PCLC, physiological closed-loop controlled; Spo₂, peripheral capillary oxygen saturation.

Design and evaluation of automatic safety critical systems in aerospace and automotive industries have significantly benefited from advances in computational modeling, control system analysis/synthesis, and improvements in human-machine interface design. However, application of such methods for design and evaluation of automatic critical care therapy delivery is nascent and less mature.⁸ This is in part due to the unique challenges posed by physiological systems, such as a lack of reliable physiological sensors, suboptimal characterization of physiological uncertainties, and lack of validated mathematical physiological models that can be used in model-based design and evaluation of PCLC devices. Additionally, uncertainties in evaluation methods and level of evidence necessary to demonstrate acceptable performance may complicate the regulatory process for innovative automated critical care devices. This article will outline the current regulatory science considerations for PCLC medical devices.

Motivation and Objective

It is part of Center for Devices and Radiological Health’s (CDRH) mission to facilitate medical device innovation and support stake-holders to improve patient care, particularly when alternative treatments are unavailable, ineffective, or associated with substantial risks to patient safety. CDRH believes that open discussion continues to help successfully advance this rapidly evolving product area. In an effort to engage stakeholders in a meaningful discussion, CDRH held a public workshop on October 13 to 14, 2015¹⁶ to discuss current challenges and opportunities with regard to PCLC devices used in automated critical care. Participants included a broad range of stakeholders involved in the design, testing, manufacturing, regulation, and use of PCLC devices. CDRH will use the information and feedback from the workshop to develop an overall strategy that will promote advances in innovation while maintaining appropriate patient protections. CDRH plans to build on advances in regulatory science and input provided from the workshop to develop guidance that provides recommendations for premarket submissions for PCLC devices.

Workshop Scope and Structure

The workshop focused on the design, development, and performance evaluation of PCLC devices intended for use in critical care environments. Such devices include closed-loop anesthetic delivery, closed-loop vasoactive drug and fluid delivery, and closed-loop mechanical ventilation (Table 1). The aims of the meeting were the following:

1. identify the challenges related to the design, development, and evaluation of critical care PCLC devices;

2. assess the unique benefits and risks introduced by PCLC devices;

3. understand the preclinical and clinical evidence needed to determine benefit/risk profile of PCLC devices;

4. initiate greater collaboration and interaction among stakeholders pursuing PCLC devices for critical care environments; and

5. promote innovation of safe and effective PCLC devices.

The white paper and discussion materials¹⁶ were developed to include topics of benefits and risks, as well as engineering, clinical, and human factors considerations for design and evaluation of PCLC devices. The purpose of this article is to inform a broader range of stakeholders of Food and Drug Administration’s proactive role toward advancing innovation in the field of automated critical care. This article will cover the discussion topics and content, along with the regulatory considerations that were discussed and emerged from the workshop. These considerations are listed in Table 2.

Table 2.

Regulatory Considerations for Design, Development, and Preclinical Evaluation of PCLC Medical Devices Used in Critical Care Environments

Sensor considerations Sensor validity and how accurately the sensed quantity represents the controlled variable. For example, if end-tidal anesthetic agent is being measured as feedback and surrogate quantity to adjust anesthetic gas concentration in the blood ultimately brain, consider providing a clinically robust justification for validity of this surrogate Characterizing sensor latency and potential detrimental effects on controller stability and inadequate therapy delivery Sensor robustness which may be thought of as the sensor’s continuous valid performance in face of patient variabilities/uncertainties Sensor reliability and resilience which may be defined as valid performance of the sensor in face of environmental uncertainties. For example in case of automated anesthesia delivery based on EEG-derived sensors, consider the effect of concomitant drugs (eg, neuromuscular blockers), motion artifact, and surgical disturbance (eg, electrocautery artifact) on the sensed depth of hypnosis

Controller design considerations Quantifying various types of uncertainties including but not limited to dynamic uncertainty imposed by inter and intra patient variability for the purpose of controller design Establishing a criteria for model selection and model optimality in design based on clinical context of use Pitfalls (eg, lack of established methods to determine stability and robustness) that may hamper system analysis and evaluation Potential additional validation activities that may ensue as a result of model-free design approach (eg, extensive simulation studies) Verifiable and structured methodologies for control system tuning during the design stage Designing the system with particular attention to nonlinearities in the whole system (eg, actuator saturation, plant, and sensor nonlinear behavior) which may result in system instability and failure Control algorithm conformity to the drug label

Clinical use considerations Relation between clinical outcomes and sensed/controlled variable(s) and the criteria that make a control variable appropriate for PCLC applications should be considered User interface design to provide operational transparency to the clinical users especially for device output is an important consideration Physical controls and display elements to improve user perception, cognition, and responses Information that needs to be conveyed to the user when reverting to an open-loop or completely manual control such that the state of the patient and previous delivered therapy is clear for the operator Specific environments of use (ie, hospital versus en route transport) may affect the interface design and how the PCLC medical device is used Design features to allow a clinician to recognize if a patient may not be responsive, or response changes, to therapy controlled by PCLC medical device The limitation of PCLC medical device in terms of delivering therapy based on a single sensor as compared with manual care where multiple factors are considered by the user System complexity and the training needed to ensure safe device use Including instruction for use language to enhance operational transparency and to create a mental model for the user of the PCLC medical device

Usability/human factors considerations Clinical setting, use scenarios, and potential for infrequent but critical events should be considered Simulator based training approaches may be leveraged to evaluate safety and assess human-automation interactions Depending on the level of automation, analysis techniques for identifying hazards such as overreliance and complacency may need to be considered Evaluating the effectiveness of any risk mitigation strategies for automation-induced hazards such as overreliance and/or complacency Language in the instruction for use should increase understanding and transparency of the controller operation

Implementation considerations Interoperability of system components such as pumps and sensors should be considered Implementing safety features and fallback modes for system fail-safe operation The need for data capture to allow forensic data and failure root cause analysis

Preclinical evaluation considerations Aiming to assess the performance of the whole system (controller, device components, patient, use environment) Analytical, bench computational (eg, in silico and hardware-in-the-loop), and animal testing or combination of these testing methods Analytical approaches for nonclinical assessment may depend on controller design (eg, knowledge-based; proportional, integral, and derivative; model-predictive control) Providing evidence of safe system performance (eg, stability) during inter and intra patient variability and disturbances that can be expected clinically Verifying functionality of fault detection and fallback modes Verification, validation, and uncertainty quantification of computational physiological models used for in silico and hardware-in-the-loop evaluation studies Animal testing needs to consider physiological and anatomical differences between animal model and human. Consider establishing clinical relevance of the animal model

Future challenges and considerations Development of standardized terminology in an effort to reduce the gap between expertise domains Development of recognized consensus standards on system performance metrics The relation between levels of autonomy and the benefit/risk profile design, risk mitigation strategy, and evaluation of a PCLC medical device Framework for establishing sensor validity criteria is an important consideration that may help in identifying safer sensors for PCLC medical devices Establishing a framework for identifying validation methods, extent of validation, and credibility of computational models used for design and evaluation of PCLC medical devices. This may be achieved by cross collaborative efforts between industry, academia and FDA

The content of the table are considerations discussed in FDA public workshop¹⁶ and should not be construed as regulatory guidance. Abbreviations: FDA, Food and Drug Administration; PCLC, physiological closed-loop controlled.

DISCUSSION

Benefit/Risk for PCLC Devices

PCLC devices have the potential to reduce the workload of clinicians and automatically deliver accurate,¹³ consistent,^9,11 and timely therapy.¹⁵ Furthermore, due to their programmable nature, PCLC devices can facilitate knowledge transfer¹¹ from clinical research to patient bedside to improve consistency of use, adherence to clinical protocols, and speed of adoption of clinical best practices. Another key advantage of PCLC devices is that, unlike manual therapy delivery, where the care provider may be prone to environmental distractions, PCLC devices are distraction-free because their sole function is to provide automated therapy to the patient.^12–14 This may reduce the incidence of human error by alleviating the workload of the caregiver and allowing him/her to focus on fewer more complex tasks in high-stress and workload environments.

While the types of risk to patients of PCLC devices are mainly unchanged as compared to manual care (eg, over-and underdelivery of therapy), automatic PCLC devices have the potential to introduce new hazards for the patient. These sources of risk can originate from (1) engineering aspects, such as lack of algorithm robustness^10,12,17,18 and necessary fail-safe mechanisms for hazardous scenarios previously corrected by clinicians^10,12; (2) from clinical aspects, such as sensor validity and reliability; and (3) from usability aspects, such as, complacency, loss of situational awareness (LSA), and skill degradation induced by automation in related applications.^4,19,20 Evaluation of the controller and algorithm interface design are central to assessing the risks associated with PCLC devices because PCLC devices can have additional complexities in their design and use as compared to manual care of similar therapies. Additional layers of automatic decision making, potentially further complicated by combining 2 or more PCLC devices (eg, automatic anesthesia and fluid resuscitation), are expected in critical care settings in the future, thus increasing the complexity of evaluating interactions and the difficulty of safety assessment.

One factor hampering an effective benefit-risk evaluation of such complex systems may be the absence of a systematic classification and framework for level of automation in critical care devices. While aviation and automobile industry domains have leveraged from LOA classification,^4,7,21 application of this concept to automated medical devices, particularly PCLC devices, remains limited. The utility of such a framework is that it introduces the concepts of automation as a continuum and assists in identification of the roles and responsibilities of operator and machine at each level of automation. A clear designation of such responsibilities and scenarios in which shifting of responsibilities may occur is essential for performance evaluation of the automatic device, as well as required training and operational transparency for the clinician. Table 3 provides an example of an LOA classification applied to a PCLCs device used in critical care environment.

Table 3.

Concept of Level of Automation Applied to an Example of a Physiological Closed-Loop Controlled Medical Device

Medical Device LOA	Task	Example Neonatal Oxygen Therapy
1	Manual therapy. All decisions pertaining to care of the patient as related to a specific therapy are made by the clinician. The device does not provide decisions nor does it provide recommendations	Clinician determines oxygen therapy is needed. Makes Fio2 adjustment for a hypoxemic patient based on vital signs, Spo2 monitor, and overall status of the patient. Clinician decides when to wean
2	Partial automation. Clinician determines the type of therapy that is needed, determines the input target/range. The device actuates automatically to keep the patient on target or within the prescribed range. All set points are determined by clinician	Clinician determines the patient needs oxygen therapy, inputs the prescribed range of Spo2 between 90% and 95%. Device senses Spo2 of 80% and increases Fio2 by 50% automatically to keep the patient Spo2 between 90% and 95% as determined by clinician. Clinician decides when to wean
3	High automation. The clinician determines the type of therapy that is needed and prescribes the first target range or set point. Device actuates to keep the patient within target prescribed range but determines and adjusts the subsequent set points automatically. Clinician will have override capabilities to assume manual care at any time	Clinician determines the patient needs oxygen therapy. Device senses Spo2 of 80% and increases Fio2 by 50% autonomously to keep the patient Spo2 between 90% and 95% as determined by the device algorithm. The clinician decides when and how to wean
4	Full automation. Device determines type of therapy. Determines the set points and course of oxygen delivery as well as initiation and rate of weaning. Clinician will have override capabilities to assume manual care at any time	Device determines oxygen therapy is needed and autonomously delivers oxygen to keep Spo2 within a range of 90%−95% as determined by device. The device determines when and how the patient is going to be weaned from oxygen. Clinician can always intervene

Abbreviations: Fio₂ fraction of inspired oxygen; LOA, levels of automation; Spo₂, peripheral capillary oxygen saturation.

Factors affecting LOA of a PCLC device may be environment dependent such as user cognitive workload and user demand/availability ratio, user related such as user level of expertise, or patient related such as clinical state of the patient and associated therapeutic plan. Increasing the LOA may not necessarily result in additional patient safety or a favorable benefit-risk profile because factors such as automation bias, loss of situation awareness, and skill degradation begin to weigh in as new sources of risk. Systematic mapping of different LOA to specific verification and validation activities such as bench, computational, animal, usability, and clinical testing while maintaining the optimal and least burdensome approach is not a trivial task and a work in progress.

PCLC Medical Device Design and Implementation Considerations

Open-Loop Versus Closed-Loop Design in Drug Delivery Systems.

Open-loop control of therapy delivery may be considered a precursor for closed-loop control.^b In fact, many closed-loop applications and design methodologies will first consider the open-loop response of the system before incorporating closed-loop control.^17,18 In open-loop control, a computational model is used to predict and target the response of the patient (Figure 1A). In closed-loop control, a feedback sensor is used to measure the patient response and use it to drive the control action (Figure 1B). Open-loop control does not rely on feedback from a sensor, and thus its safety and performance will depend on the accuracy, robustness, and predictiveness of the patient computational model (eg, pharmacokinetics pharmacodynamics [PK/PD] models for drug delivery pumps), as well as adequate characterization and handling of potential disturbances. The impetus for the addition of a sensor is that closed-loop feedback–controlled systems as compared to open-loop and manual-controlled systems by nature can attenuate disturbances and uncertainties by reducing the magnitude of an error signal^10,17,18.

Figure 1.

A, Block diagram of an open-loop controlled devices. The set point is determined by the clinician and the computational model (eg, pharmacokinetics pharmacodynamics model of the drug) determines a delivered dose which will be communicated to an actuator such as pump or ventilator blender. B, Block diagram of a physiological closed-loop controlled (PCLC) device. The set point is determined by the clinician and maintained by the PCLC device through sensing of the patient response and comparing it with the set point. The controller determines the level of therapy delivery (eg, drug infusion rate).

In both open- and closed-loop control of drug delivery, the quality and fidelity of the PK/PD model plays a central role in the device performance. For example, in open-loop infusion, the PK model may be used to estimate the drug target site concentration. The estimation of serum concentration using such a method may be considered a surrogate marker and will likely be validated before its acceptance as a potential end point.

The labeling for approved drugs includes dosing guidelines that have been demonstrated to be efficacious and safe in the patient populations for which the products are approved. The dosing guidelines take into consideration patient demographics (ie, race, age, sex), comorbidities (eg, renal insufficiency), and concomitant therapy that can affect safety or efficacy (eg, use of narcotics in conjunction with sedative agents). The degree to which a PK model, and the device that utilizes it, can take these factors into consideration is likely to impact device performance. Because PK models are studied in specific, targeted populations (eg, adults), it may not be possible to apply them in other patient populations (eg, pediatrics) without potential loss of accuracy. The same is likely to be true for each of the factors noted above.

PCLC Medical Device Design Methodologies.

Traditionally, control systems utilized in aviation and the auto industry use a mathematical description of the controlled plant. In the case of PCLC devices, this description is a mathematical model relating the physiological variable (eg, blood pressure, blood oxygen saturation) to the input variable (eg, fluid infusion, oxygen concentration).^3,17 Largely due to difficulty in modeling complex physiology, the design approach for PCLC devices has sometimes followed a model-free approach in which the system is tuned empirically based on trial and error or clinical knowledge. The choice of the PCLC device design can have significant impact on subsequent stages of product development including implementation, verification, and evaluation of the PCLC systems.

Model-Based Design.

Model-based design combines mathematical modeling of PCLC components with mathematical models of the patient to allow for understanding of the overall system response. Having this quantitative understanding is crucial to identifying the control approach and utilizing an iterative design approach to yield the optimal control strategy. Model-based design allows quantification of system dynamic response, system uncertainty, system sensitivity, and disturbance rejection properties based on control system engineering principles.^3,10,17,18 It allows the designer to find an acceptable tradeoff between stability and performance and analyze controller sensitivity and disturbance rejection properties.

Central to model-based design is the physiological model representing the patient. This model is typically developed using fundamental constitutive differential equations representing the target physiological behavior. In many cases, the parameters of the differential equations may be unknown, and system modelers resort to already established techniques, such as system identification, to estimate model parameters.^22,23

In the later stages of the product development, a model-based design approach can further enable performance evaluation of the controller through simulation of the system. Potential limitations of both the algorithm and hardware intended to be used with the controller can be explored using simulation and will allow refinement of algorithm design before initiation of clinical evaluation. A natural consequence of adopting a model-based design strategy is that it enhances verifiability of the PCLC algorithm, which might be helpful for regulatory purposes. Challenges of model-based design include the complexity of the model development and the lack of physiological relevance of some model parameters²⁴ making it difficult for the clinical community to adopt devices with such models. In addition, system identification techniques for physiological systems are challenging due to limitations in accurate measurement of input/output signals and limited excitability of the system.

Model-Free Design.

The PCLC design process can be independent of a mathematical model. For example, it can be based on clinical protocols with knowledge embedded in controller hardware. As a result, if the design of the controller is implemented without models of the patient or sensor dynamics, then quantification of closed-loop system metrics and responses may not be possible in a mathematical sense. Exhaustive simulation testing may be an option to assess performance of the already developed controller. However, a realistic patient model capable of simulating clinically challenging scenarios with an acceptable level of credibility will likely be needed for evaluation of the system.

Choice of Controller and Controller Design.

Advances in control system engineering have generated a plethora of controller types. Most industrial controllers use well-established control strategies, such as proportional, integral, and derivative. PCLC devices, in addition to proportional, integral, and derivative control, may include more advanced control algorithms including, but not limited to, adaptive, model-predictive, fuzzy control, and hybrid control system design are also used.^8,10,12 Identification of potential advantages and disadvantages of each algorithm type for its intended use remains a challenge. For example, the ability of the controller to address endogenous and exogenous disturbances and uncertainties arising from patient variabilities differs from 1 controller to another for a particular clinical application. Furthermore, controller type and choice of the control layers (eg, having supervisory control) dictates the versatility of control algorithm and how it handles events leading to system failure (eg, disconnected sensor, actuator saturation). The system designers determine the type of controller; it may be advantageous to select a controller to allow insight on the inner workings and decision making of the PCLC device. For example, the control logic, control variables, and parameters especially those that will likely be iteratively tuned may need to be reported for the evaluation process.

System Implementation.

Taking a PCLC device from the virtual design to the physical device involves hardware and software considerations. The designer of a control system will need to understand the physical limitations of the hardware (eg, sensor delay or actuator saturation) and considerations for digital control system implementation. An adequate implementation strategy can enhance controller safety and mitigate new sources of risk. These hazards may be related to, but not limited to, system (software and hardware) reliability, inadequate handling of fault conditions such as sensor degradation and failure, inadequate alarms, and lack of data collection mechanisms for forensic analysis. Verification activities are critical to ensuring the proper implementation of the system.

Preclinical Evaluation Considerations for PCLC Devices

The risks to the patient should be considered throughout the design and development of any medical device. This typically includes a formal risk analysis and evaluation to help identify the types of hazardous situations and potential risk control measures. Before use in a clinical setting, preclinical device testing may help to demonstrate and verify the effectiveness of risk control measures to support safety before conducting clinical studies used to support marketing clearance/approval. In addition to safety and performance criteria from existing standards (eg, biocompatibility, sterility, electrical safety) that should be considered across medical device types, PCLC devices may have unique testing considerations.

PCLCs adjust therapy based on an expected physiological response. Physiological responses can vary within an individual patient over time and from patient to patient, requiring a controller to perform over a wide range of uncertain conditions. PCLC devices could become unstable if the controller design is not sufficient for the expected range of physiological conditions. Additionally, the presence of physiological delays, improper inputs or external disturbances may induce instabilities in the response which could result in the incorrect or inadequate therapy being delivered to a patient.¹⁸

PCLC devices may combine sensors (eg, from a physiological measurement device) and actuators (eg, infusion therapy) with a controller with each subsystem requiring its own level of performance. New sources of risk can arise from the interactions between these subsystems, as well as with the patient and environment. PCLC devices that combine sensors and actuators will require consistent and unambiguous communication between all components. The PCLC device designer may need to understand detailed characteristics about the subsystems to ensure consistent performance, potential failure modes associated with each subsystem, and new risks that could emerge by the implementation of closed-loop control with existing sensors and actuators.

In “real world” use, sensors may be affected by external environmental or physiological disturbances resulting in short-term transients in the monitored physiological signals. In current clinical practice, the clinical staff may recognize and simply ignore these transients. However, in a closed-loop configuration, these transients will be passed into the controller and could directly affect the therapy being delivered. The characteristics and frequency of such artifacts may be increased during ambulatory or emergency use. Without being able to recognize and adjust to these artifacts, the PCLC device may enter into an unforeseen potentially hazardous state.

For some PCLC devices, the long-term performance may vary over time. For example, in drug infusion systems, due to the potential extended duration of infusion and possibility of variation in infused volume, an accurate report of past infusion profile and the total volume infused may need to be communicated to users to aid in adequate delivery. Drug or biological products that may be delivered to a patient with a PCLC device may eventually be depleted. The device should be able to recognize and report this information to the user in a timely manner.

It may be necessary for many of these aspects to be evaluated to safely proceed to clinical studies. Sufficient non-clinical testing may be needed to minimize the risk, number of end points, and total size of clinical studies. A number of nonclinical testing methods exist for evaluating closed-loop control systems including: analytical approaches to determine the stability and response of a controller, computer simulations (eg, in silico studies, Figure 2A) to estimate the controller performance over a wide range of physiological conditions,²⁵ real-time (hardware-in-the-loop) bench testing to verify the functionality of fail-safe mechanisms and expected device performance using the actual hardware of the closed-loop system (Figure 2B) with a computational patient model, and animal model testing to provide realistic physiological challenges with appropriate disease models. There is a large body of engineering knowledge that can be applied for formal analytical assessment of controllers. However, this type of evaluation for a system designed without a modeling approach remains challenging. Although hardware-in-the-loop testing has been used extensively in other industries for validation of control systems,^26,27 this type of testing has not been adopted widely for evaluation of PCLC devices. Because of the realistic nature of this type of testing, combined with the versatility given by a computational model, hardware-in-the-loop testing may be an example of a testing methodology for PCLC devices that can provide valuable safety evidence in preparation for clinical study initiation. Hardware-in-the-loop testing can cover a larger span of both physiological and nonphysiological (eg, physical disturbances) conditions than could be accomplished clinically. Using this method, unsafe conditions that a patient or trial subject should never be exposed to would be simulated and addressed early in device development.

Figure 2.

A, Complete in silico testing: the testing involves complete modeling of the entire system. Extreme functional scenarios of device components such as the pump and sensors may allow greater range of testing. The dashed lines indicate the link to and from a computer model. B, Hardware-in-the-loop testing: The actual physiological closed-loop controlled device is connected to a physiological patient model representing the patient response. The device will be evaluated in real time and may be stress tested under various physiological and patho-physiological conditions. The dashed lines indicate the link to and from a computer model connecting to an actual physical device.

The credibility of evidence obtained from computer simulations and hardware-in-the-loop testing depends on the computational physiological model used in the testing, and the verification, validation, and uncertainty quantification activities that have been performed to demonstrate the appropriateness of that model for the PCLC testing context of use. However, the extent of validation process to ensure that results from a computational modeling study are representative of PCLC device performance has not been sufficiently established. As the validity of computational models is established for evaluation of PCLC devices, the credibility of the safety and effectiveness evidence they provide will increase.

Human Factors/Usability Considerations

In critical care environments, such as intensive care units, surgical units, and emergency rooms, where high-stress situations can present usability and human factors challenge for clinicians, PCLC devices can manage the labor-intensive tasks, improve the timeliness, reliability, and consistency of therapy delivery, and help to reduce the incidence of human performance error. As such, PCLC medical devices present a unique opportunity to improve the current standard of care in critical care environments. Application and advancement of human-system interfaces to automatic PCLC medical devices is central toward realization of this vision.^28,29

Automatic PCLC devices may allow clinicians and other users to experience reduced interactions with the device. Depending on the extent of automation, the user of the PCLC device typically performs a supervisory or monitoring role with occasional interventional responses. It is noteworthy to mention that the crux of human-automation interaction hazards hinge on operators’ inappropriate trust in automation.^30–32 Consequently, the lack of dedicated interactions with the device and inadequate perception of trust might lead to human-automation interaction hazards, such as a LSA, complacency, and skill degradation. These are hazards identified in industry domains such as aviation that have incorporated automation^20,29,33 and will likely apply to PCLC devices as they progress toward advanced and higher LOA.³⁴

Loss of Situational Awareness.

The automation of a clinical decision-making function by PCLC medical devices may reduce the clinician’s awareness of a patient’s current condition and/or device status. As a PCLC device consistently and repeatedly selects and executes decision choices with potential minimal acknowledgement from the clinicians/users, the clinician may not be able to sustain a good “picture” of the patient condition because he or she is not actively engaged in patient evaluation. At high LOA, the role of the clinician primarily shifts to a supervisory role relative to continuous interaction with the device. Maintaining awareness of patient status and PCLC medical device states becomes a difficult challenge as a result of limited or no interaction. If the device fails or cannot respond appropriately to exceptional situations, the clinician/user may be unable to take over the control of the device clinical functionality or task to prevent harm to the patient. The clinician may not have an accurate understanding of what is happening or what course of therapy was followed by the device, and may therefore be unable to provide corrective response.

Complacency.

For high LOA characterized by perfectly reliable execution in decision choices, clinicians may not monitor the automation and its information sources, and fail to detect potential automation failure. Particularly, in a multitasking work environment, the effect of overtrust, complacency, or overdependence is greatest when the task of the user has been reduced to only monitoring the automated system.^19–21

Skill Degradation.

If the clinical decision-making choices are consistently executed by automation, there may come a time when the human operator will not be as skilled in performing any information seeking and processing, decision making, and executions. This may lead to forgetting and skill decay manifestation. Missed, delayed, or wrong diagnoses may be the result of the deterioration of cognitive diagnostic skills that are used rarely or not at all over a prolonged period of time due to automation. Deterioration of cognitive diagnostic skills can have a severe impact on patients, providers, and the entire health care system.³⁵

Collectively, these potential consequences of human interaction with automatic PCLC devices can be attributed to the phenomenon referred to as “out-of-the-loop” unfamiliarity for the clinician.²⁹ Standard clinical training may need to be augmented to address these hazards.

To eliminate potential hazards introduced by automation that will undoubtedly be incorporated in future PCLC devices, PCLC device’s user interfaces need to be designed to reduce incorrect responses from clinicians during exceptional situations. Current PCLC devices utilize interfaces and displays to provide feedbacks through alarms and alert clinicians of patient status and device state changes, and to elicit clinician/user responses and/or acknowledgments. The automation components of PCLC devices may need to be designed to reduce particularly “out-of-the-loop” unfamiliarity hazardous situations. PCLC developers/researchers may need to consider the user interface as an essential element for success of a PCLC device in the future. Future PCLC devices may need to be designed with intuitive decision-support user interface components to mitigate potential hazards associated with automation. Timely responses to infrequent critical events are vital for adequate delivery of care. An important consideration may be to design PCLC devices to support and sustain clinicians/users mental models so that patient/device states that can be easily detected and understood for the appropriate responses.

Another consideration is the environment where PCLC devices are used. The use environments for PCLC devices include a variety of conditions that could affect the user interface design and user interactions. While this is true for almost all devices, for PCLC devices, environment of use is central to the design of user interface. Different environments present different user interaction and, as such, would require unique user interface designs. For example, hemo-dynamic instability and mechanical ventilation crisis calls in critical care and interhospital vehicular transit require different clinical responses due to nature of the environment attributes.

There have been minimal human factor studies focusing on the role of the clinician or user of the PCLC device to mitigate hazardous situations related to situational awareness, complacency, and skill degradation. Infrequent or rare critical events associated with PCLC devices may be major life threatening events that require rapid and appropriate responses from clinicians. Medical simulation-based training characterized by realistic practice of rare but critical events to capture clinician’s LSA, complacency, and skill degradation can improve patient safety with the use of PCLC devices during actual settings.

Absence of deliberate studies to evaluate unique automation-induced potential hazards in clinical care environment of PCLC devices presents a challenge for evaluating risks of PCLC medical devices.

Clinical Considerations

Algorithm Complexity and Lack of Transparency.

As system level of automation increases, on 1 hand, the algorithms that guide PCLC therapy delivery grow in complexity. On the other hand, clinicians may not understand or be aware of the system due to algorithms not prioritizing transparency to the user. Combination of system complexity and lack of operational transparency may provide challenges for safety delivery of automated therapies. Absence of standardized terminology and lack of understanding of the medical device’s inner working and decision-making capabilities^10,12,36 may hamper the user’s ability to handle device failures promptly and effectively, potentially leading to patient harm.

PCLC Sensors.

While significant strides have been made in the area of physiological sensing and monitoring, sensors used as feedback for a PCLC device have inherent limitations. Current physiological monitors such as capnometers and electroencephalogram-based hypnosis depth monitors are used for monitoring and trending purposes in manual care. When used as a sensor for a PCLC device, these sensors will enable the system to take a therapeutic role by determining titration and adjustment of therapy. In most PCLC devices, these sensors function as the sole source of information that will be fed into the algorithm to determine therapy adjustment; while in manual care, the clinician determines the course of therapy based on multiple indicators obtained from various monitors and patient response assessment. As such, designing physiological sensors to drive automatic therapy may necessitate enhanced accuracy, reliability, robustness, and resilience against clinical (eg, patient variability) and environmentally challenging scenarios. Furthermore, developers of PCLC devices may consider combining multiple sensors to inform the control algorithm of varying patient conditions and therapeutic needs. Establishing clinical validity of sensors and the relation of the measured sensor parameter to the control objective remains a challenge in the development and evaluation of PCLC devices. There exist no formal guidelines as to what should be the validity and reliability criteria for sensors used in a particular PCLC application. The extent of scientific and clinical evidence to establish validity and reliability remains unclear.

Handling of Disturbances.

While many types of disturbances that occur during therapy delivery are the same for automatic and manual control, additional hazards may occur when disturbances induce an out-of-range response from the controller due to challenges previously mentioned. Thus, there may exist a significant chance of inadequate therapy delivery when taking into consideration the potential for overreliance and lack of sensor reliability and robustness. Identification, characterization, and handling of disturbances, including exogenous factors (motion artifact, patient movement, secondary drug infusion) and endogenous disturbances (change of physiological parameters) are potential steps toward successful control system design.^17,18

In many cases, such disturbances may happen infrequently and may not be present or captured in the course of a clinical validation study. Preclinical methods, such as mathematical modeling and simulation studies noted in section (Figure 2), may offer potential complementary evidence of safety, provided that high-fidelity validated models are used. Development and selection of such models remains a challenge and will require close collaboration of clinical and modeling experts.

Lack of Anticipatory Response.

In manual care therapy settings, clinicians can anticipate certain disturbances, such as surgical stimulation or resistance to drug therapy, and can plan accordingly to reduce the likelihood of over and/or under therapy delivery. PCLC devices may not be designed with anticipatory feedback. While closed-loop systems can respond faster than humans, the lack of forecasting of the patient’s state can introduce a new hazard when compared to manual care.

Knowledge Gaps and Lack of Standardization.

One of the main challenges in designing and evaluating a PCLC system is that it requires interdisciplinary interactions of clinicians and engineers to provide each other with a basic understanding of their perspective and expertise. This is an essential step before initiating and developing consensus on the least burdensome approaches to foster innovation in safe PCLC devices. While incorporating clinical expertise into control system design is essential, especially at the early stages of the design process, it is equally important for clinicians to understand fundamental principles, potential advantages, and limitations of PCLC devices.^8,10,36 Clinicians using PCLC devices that understand how the decision-making process has been designed into the device, the physiological relevance of algorithm parameters,²⁴ potential controller failure modes, and risk mitigation strategies can operate the device effectively. Moreover, clinician participation is essential for establishing control system performance metrics, and their clinical relevance, that may serve as design requirement specifications for engineers responsible for developing and evaluating the systems. Consensus standards such as IEC 60601-1-10³⁷ may be 1 option for ensuring broad acceptance, relevance, and standardization of performance metrics that may also help to clarify regulatory expectations. Developing standards that address performance evaluation of the PCLC devices may help advance the development of the field and bring products to market faster. Other areas that can significantly benefit from standardization include terminology, clinical best practice for some PCLC applications (eg, ventilator oxygen support), eligibility criteria for sensor technology used as feedback to drive a PCLC application, and worst-case scenario test cases used for stress testing of PCLC devices.

CONCLUSIONS

Automation has had a revolutionizing effect on the aviation and auto industry. The same technology is now being introduced to medical devices, particularly PCLC medical devices, intended for critical care settings. Automated PCLC devices have potential to enhance therapy by increasing accuracy and timeliness of therapy delivery. They may enable a more consistent therapy delivery and better adherence to protocols. Automation in critical care devices presents unique challenges for clinicians and system designers. While many advances have occurred in the areas of physiological sensing, computational modeling/simulation, and human factors usability of automated PCLC devices, further research is required to ensure that risks due to automation-induced clinical engineering and usability hazards have been properly addressed. CDRH recognizes the potential public health benefit of PCLC devices, particularly for low-resource environments, and, through a public workshop, gathered relevant stakeholders to engage in discussions regarding potential benefits, risks, and future challenges of PCLC devices. CDRH is currently synthesizing the discussion content from this workshop to draft regulatory recommendations.