Abstract
The dynamic nature of the medical domain is driving a need for continuous innovation and improvement in techniques for developing and assuring medical devices. Unfortunately, research in academia and communication between academics, industrial engineers, and regulatory authorities is hampered by the lack of realistic non-proprietary development artifacts for medical devices. In this paper, we give an overview of a detailed requirements document for a Patient-Controlled Analgesic (PCA) pump developed under the US NSF’s Food and Drug Administration (FDA) Scholar-in-Residence (SIR) program. This 60+ page document follows the methodology outlined in the US Federal Aviation Administrations (FAA) Requirements Engineering Management Handbook (REMH) and includes a domain overview, use cases, statements of safety & security requirements, and formal top-level system architectural description. Based on previous experience with release of a requirements document for a cardiac pacemaker that spawned a number of research and pedagogical activities, we believe that the described PCA requirements document can be an important research enabler within the formal methods and software engineering communities.
I. Introduction
In the ninetieth century, physicians sometimes practiced grave-robbing to obtain subjects for investigation. If only finding suitable subjects for application of software engineering and formal methods in the medical domain were so easy.
There are a number of desirable qualities of case study artifacts for facilitating research and pedagogy in the medical device domain:
the subject matter must be “real-world” enough to be relevant;
it must be supported by domain documentation including: appropriate background on device mechanics needed for the targeted physiological monitoring and actuation, relevant human physiology, information on typical clinical contexts including use cases and clinical workflows;
it must be “big” enough to show methods scale, yet not overwhelm small academic teams,
it should expose systems issues—both software and hardware functionality should be exposed to a degree of specificity needed to support work on techniques for risk management, hazard analysis, and system safety;
it should include or provide a pathway for execution on actual hardware or realistic simulation so as to enable realistic evaluation of testing, verification, and other quality assurance techniques;
it should include information sufficient for enabling academic teams to be aware of, and even develop, techniques for addressing regulatory and certification issues.
This paper describes a case-study artifact, a publicly-available requirements document [1], that possesses many of the qualities identified above and provides a pathway for realizing those remaining.
A. Previous Experience with Case Studies that Catalyze Research and Education
While working as an engineer at Boston Scientific, significant efforts by the first author resulted in the release into the public domain of a system specification for a previous generation implantable cardiac pacemaker [2]. The goal of this effort was to catalyze research and education on realistic applications of formal methods and evidence-based certification regimes. Larson advised students at the University of Minnesota and faculty at McMaster University in developing an inexpensive hardware platform for class projects on which pacemaker code could be executed/simulated and guidelines for evaluating solutions submitted in response to verification and certification challenge problems. McMaster University researchers and the Software Certification Consortium (SCC) developed and supported the “Pacemaker Formal Methods Challenge”1, which led to several special workshops and events that focused on highlighting formal methods. An upcoming Dagstuhl Seminar is dedicated to reporting on past work and facilitating future research related to the pacemaker artifacts. Up to this point, the pacemaker requirements document has been utilized in more than 30 publications and in class projects at a number of universities.
B. Goals of This Work
In this paper, we report on an effort that aims to have a similar catalyzing effect but this time for Patient-Controlled Analgesic (PCA) pumps. We give an overview of a detailed requirements document for a PCA pump developed in consultation with US Food and Drug Administration (FDA) engineers under the auspices of the US National Science Foundation (NSF) Food and Drug Administration’s (FDA) Scholar In Residence (SIR) program. There are a several important features of this document:
the 60+ page document follows the methodology outlined in the US Federal Aviation Administrations (FAA) Requirements Engineering Management Handbook (REMH) [3];
it includes a domain overview providing relevant clinical context;
it provides a collection of normal and exceptional use cases, as well as
a formal architecture description including both software and hardware components specified using the SAE standard Architecture and Analysis Definition Language (AADL) [4], [5],
the architecture is organized to provide a distinct safety architecture—a separate subsystem designed to monitor for system faults and take appropriate action to mitigate associated hazards and ensure patient safety.
The requirements document is already being utilized in different settings, which we plan to report on in detail elsewhere. Within our own research, the first author has led an effort to supplement the AADL architecture description with formal behavioral specifications written in his BLESS framework [6]. These specifications include contracts on component boundaries written in the BLESS behavioral interface specification language and formal proofs of behavioral conformance to contracts using the BLESS proof tool. We are also using the requirements specification to support research on inter-operability, safety, and security for devices that interoperate following the architecture specified in the ASTM Integrated Clinical Environment standard [7]. The requirements document is also supporting collaborative work with the FDA and Underwriters Laboratory on safety standards for interoperable medical devices. Finally, we are contributing to the Software Certification Consortium [8], which is seeking such problems from many safety-critical domains for “mock” certifications including assurance cases arguing for safety and effectiveness from evidence, especially from formal methods.
C. Previous Work
Kansas State’s PCA Pump requirements document builds upon University of Pennsylvania’s and FDA’s Generic Infusion Pump (GIP) project [9]. The GIP project includes a smaller set of requirements, and an initial hazard analysis for the pump. These GIP artifacts were utilized in follow-on work by researchers at UPenn / FDA and elsewhere on the application of verification techniques that tended to emphasize properties that could be checked by real-time model checkers like UPPAAL [10]. Our work aims to further the objectives of the GIP project by expanding on the original requirements document along several dimensions, e.g., by significantly expanding the requirements to address a much broader set of functionality and additional safety requirements, by adding clinical motivation and use case descriptions, by adding formal architectural descriptions, by introducing a safety architecture, and by aligning the document with the methodology suggested in the FAA REMH.
II. PCA Pump Background
A PCA infusion pump is used to infuse a pain killer. Pain medication is prescribed by a licensed physician, which is dispensed by the hospital’s pharmacy. The drug is placed into a vial labeled with the name of the drug, its concentration, the prescription, and the intended patient. A clinician loads the drug into the pump, and attaches it to the patient. The pump infuses a prescribed basal flow rate which may be augmented by a patient-requested bolus or a clinician-requested bolus. This allows additional pain medication in response to patient need within safe limits.
PCA pumps, unfortunately, have been associated with a large number of adverse events [11], [12]. The FDA notes [13] that while PCA pumps (and infusion pumps in general) have allowed for a greater level of control, accuracy, and precision in drug delivery—thereby reducing medication errors and contributing to improvements in patient care—infusion pumps have been associated with persistent safety problems. From 2005 through 2009, 87 infusion pump recalls were conducted by firms to address identified safety problems. Infusion pump problems have been observed across multiple manufacturers and pump types. Through analysis of pump-related adverse event reports and device recalls, FDA has concluded that many of these problems appear to be related to deficiencies in device design and engineering.
Through the Infusion Pump Improvement Initiative [13], FDA is taking broad steps to prevent infusion pump problems. Specifically, FDA aims to establish additional requirements for infusion pump manufacturers, proactively facilitate device improvements, and increase user awareness of problems and best engineering practices. As an example of emphasizing best engineering practices, the FDA Draft Guidance for Infusion Pumps [14] now requires pump manufacturers to provide an assurance case with their regulatory submissions.
These activities indicate the significant concerns that FDA has regarding pump safety, and they provide an impetus for research in the areas of software engineering, safety, security, and verification & validation applied to pump development; research which we hope to enable to some extent with the requirements document described here.
III. The Requirements Document
A. Sources of Information
What sources of information were used in the construction of the requirements document?
These requirements simulate the result of domain experts working with systems engineers to define function that will be safe for patients, and effective for some medical need. For PCA, that medical need is to provide narcotics to dull excruciating pain. Delivering medication as prescribed is what makes a PCA pump effective. Avoiding overdose, and all other harms to patients, is what makes a PCA pump safe.
These simulated requirements are provided as a public-domain example, because real requirements are highly-confidential to medical device manufacturers, often using proprietary clinical data. However, it should be noted that the authors are not clinical experts in PCA infusion therapy. Our primary sources of information were the FDA’s guidance documents on infusion pumps (in particular, the description of hazards for pumps) [14], the FDA Infusion Pump Infusion Pump Initiative [13], earlier versions of requirements from the GIP [9], feedback from FDA engineers, and the first author’s previous experiences in the medical device industry. The primary contribution to the research community is a collection, in one place, of relevant domain knowledge sufficient for driving realistic research investigations of techniques for developing pumps. Even though we made every attempt to be a clinically accurate as possible, we will not claim to have provided the accuracy or detail sufficient for developing a device for which regulatory approval could be obtained. However, one of our goals is to develop examples of risk assessment artifacts and mock regulatory submissions that would provide futher insight into the regulatory submission and approval process.
B. Methodology
What methodology did we use in the process of writing and organizing the requirements?
There are a number of potential sources to appeal to for guiding elicitation and capture of requirements. Traditional Software Requirements Specification (SRS) guidelines such as IEEE-830 are general purpose guidelines and fall short of the methodology and insights needed when dealing with safety-critical systems. Our principle source of inspiration has been the US Federal Aviation Administration (FAA) Requirements Engineering Management Handbook (FAA-REMH) written by Rockwell Collins engineers David Lempia and Steven Miller [3]. FAA-REMH focuses directly on recommended practices for requirements engineering for safety-critical embedded systems and provides illustrations using three systems, including a medical system–an Isolette Thermostat for a neonatal incubator. We found it a reasonable resource as it met two key criteria: 1) it is targeted at safety-critical embedded systems, and 2) it was written by experts in the field.
FAA-REMH lists eleven steps that developers should take in order to “progress from an initial, high-level overview of the system... to a detailed description of its behavioral... requirements.” The steps are:
Develop the System Overview
Identify the System Boundary
Develop the Operational Concepts
Identify the Environmental Assumptions
Develop the Functional Architecture
Revise the Architecture to Meet Implementation Constraints
Identify System Modes
Develop the Detailed Behavior and Performance Requirements
Define the Software Requirements
Allocate System Requirements to Subsystems
Provide Rationale
C. Document Structure
What is the overall structure and content of the document?
The document has twelve sections, briefly summarized here, with more important sections further elaborated below.
Introduction: purpose, references, terms, and acronyms
System Overview: clinical background and need, synopsis, context, external interactions, goals, boundary, and monitored/controlled variables
System Operational Concepts: use and exception cases
PCA Pump Function: functional requirements; basal flow rate, patient- or clinician-requested bolus
PCA Pump Interfaces: interface requirements; sensors, actuators, alarms, control panel, logging, network, reservoir, drug library, scanner
Safety Requirements: safety architecture, anomaly detection and response, power, diagnostics, tamper-resistance, biocompatibility
Security Requirements: authentication, confidentiality, provisioning
Requirements Allocation: requirements must be allocated to functional architecture or labeling
Labeling of Nonfunctional Requirements: environmental assumptions for use
Functional Architecture: decomposition of system into functional components
Initialization and Configuration: simplified set-up instructions
Rationale: reasons for requirements
D. System Operational Concepts
FAA-REMH recommends creating use and exception cases during requirements development to capture system operational concepts. Our requirements document contains roughly 15 pages of use cases. Table I lists the use cases in the requirements document; Table II list exception cases. Use case (UC1) is presented as an example.
TABLE I.
Summary of PCA Use Cases
| ID | Actor | Title | Description |
|---|---|---|---|
| UC1 | Clinician | Normal Operation | initialization, attachement, basal infusion, detatchment |
| UC2 | Patient | Patient-Requested Bolus | extra dose upon patient-determined need |
| UC3 | Clinician | Clinician-Requested Bolus | extra dose upon clinician-determined need |
| UC4 | ICE app | ICE-Detected Hazard | switch to KVO infusion rate upon ICE app-determined need |
| UC5 | Clinician | Resume Operation After ICE-Detected Hazard | resume prescribed infusion after clinician determines it is safe |
| UC6 | Clinician or App | ICE-Initiated Audible Alarm Inactivation | suspend audible alarm from ICE |
TABLE II.
Summary of PCA Exception Cases
| ID | Actor | Title | Description |
|---|---|---|---|
| EC1 | Patient or Clinician | Bolus Request Too Soon | bolus request denied because minimum time between boluses had not elapsed |
| EC2 | Clinician | Drug Library Soft Limit | basal rate or bolus VTBI exceeded soft limit |
| EC3 | Clinician | Drug Library Hard Limit | basal rate or bolus VTBI exceeded hard limit |
| EC4 | Power-On Self Test Failure | power-on self test fails | |
| EC5 | Internal Electronic Failure | PCA pump detects its own failure | |
| EC6 | Clinician | Pump Priming Failure | pump fails to prime after loading drug reservoir |
| EC7 | Over-Flow Rate Alarm | measured flow rate exceeds setting | |
| EC8 | Under-Flow Rate Alarm | measured flow rate below setting | |
| EC9 | Pump Overheating | pump temperature exceeds 55 C | |
| EC10 | Downstream Occlusion | blockage between pump and patient | |
| EC11 | Upstream Occlusion | blockage between reservoir and pump | |
| EC12 | Air-in-line Embolism | bubble detection | |
| EC13 | Maximum Safe Dose | dose reaches maximum allowed by drug library | |
| EC14 | Clinician | Clinician Authentication Failure | clinician not authorized to operate pump |
| EC15 | Clinician | Patient Authentication Failure | patient not admitted to hospital |
| EC16 | Clinician | Prescription Authentication Failure | drug or prescription not intended for this patient |
| EC17 | Clinician | Sound Failure | no audible alarm |
| EC18 | ICE Failure | indication of no ICE alarms enabled | |
| EC19 | Drug Library Not Present or Corrupted | the drug library fails authenticity or integrity check |
Use Case: Normal Operation of PCA Pump (UC1)
Related System Goals: G1 and G2
Primary Actor: Clinician
Precondition
Patient is ready for infusion
Physician has prescribed drug
Pharmacy has filled prescription
Pharmacy has installed drug library into PCA pump
Drug has been delivered to clinician
PCA pump is off
Postcondition
PCA pump is turned off
Infusion needle removed from patient
Main Success Scenario
Clinician turns on PCA pump (Exception Case: Power-On Self Test Failure)
Clinician presses button when hearing audible alarm sound (Exception Case: Sound Failure)
Clinician scans own badge
Clinician is authenticated to operate PCA pump for Δauth = 5 minutes (Exception Case: Clinician Authentication Failure)
Clinician enters scans patient information
Patient is authenticated to receive medical care for Δauth = 5 minutes (Exception Case: Patient Authentication Failure)
Clinician scans drug information and patient’s prescription from drug container (vial)
Prescription is authenticated as originating from an authorized pharmacist (Exception Case: Prescription Authentication Failure)
Prescription is authenticated for the patient for Δauth = 5 minutes (Exception Case: Prescription Authentication Failure)
PCA pump compares prescription with its drug library (Exception Cases: Drug Library Soft Limit and Drug Library Hard Limit and Drug Library Not Present or Corrupted)
PCA pump unlocks and clinician opens the reservoir door
Clinician puts drug vial into the reservoir and closes the door
PCA pump locks reservoir door and terminates clinician authentication
Clinician attaches infusion tube and needle to pump
Clinician primes pump (Exception Case: Pump Priming Failure)
Clinician inserts infusion needle into patient’s vein
Clinician presses Start button to begin basal-rate infusion
Bolus dose infused upon request; see Use Cases UC2 and UC3: Bolus Infusion
Clinician presses Stop button to halt infusion
Clinician removes infusion needle from patient’s vein
Clinician scan own badge
Clinician is authenticated to operate PCA pump for Δauth = 5 minutes (Exception Case: Clinician Authentication Failure)
PCA pump unlocks and clinician opens the reservoir door
Clinician removes drug vial, closes the door, returning remaining drug to pharmacy
PCA pump locks reservoir door and terminates clinician authentication
Clinician turns off PCA pump.
We do not claim to provide an exhaustive collection of use cases. We aimed for a level of coverage that would allow us to establish traceability of each of the subsequent requirements to one or more use cases.
E. Individual Requirements
Each requirement has its own paragraph, which allows unique numbering and tracing to functional architecture components responsible for implementing them. There are subsections of statements for functions such as Basal Flow Rate, Patient-Requested Bolus, Clinician-Requested Bolus. Requirements are given for the actions of pump sensors (for detecting flow rate, occlusion of drug delivery tubes, and air-in-line embolism (bubbles)), actuators (the pumping action itself), and alarms. Requirements are also given for supporting functionality such as the clinician interface, on-device drug library, logging, and drug reservoir. Safety and security are broken out into their own distinct sections.
Finally, an important goal of our work is the investigation of issues associated with network-enabled interoperable devices that conform to the ICE architecture [7]. Accordingly, the document gives some initial requirements related to the ICE interface of the device (we expect these to evolve as our research progresses).
Here are some samples of requirements from the patient bolus function.
Upon patient’s press of the PCA pump’s patient-button, a prescribed bolus volume-to-be-infused, V TBI, of the drug loaded in the pump is delivered to the patient.2
A patient-requested bolus shall be delivered at its prescribed rate, Fbolus, in addition to the prescribed basal flow rate, Fbasal, but no more than the maximum flow rate for the pump, Fmax.
Patient-requested bolus shall not be delivered more often than a prescribed number of minutes, Δprb.
Prescribed V TBI and rate shall not exceed the hard limits set by the drug library from the hospital pharmacy for the drug loaded in the PCA pump.
Patient-requested bolus shall not be delivered if infusing prescribed V TBI will exceed hard limits retrieved from the drug library for the volume of drug infused over a period of time. Pump rate shall be reduced to KVO and a max dose warning be issued.
Patient-requested bolus delivery shall be immediately halted when alarms sound.
Here are some from the audible alarm safety requirements.
Alarms shall cause audible alarms signals that meet the requirements of Tables 203 and 204 of standard IEC 60601-1-8 for alarm pulses, bursts, and harmonics.
The auditory volume of audible alarms signals shall conform to Section 201.3.3.2 Volume of auditory ALARM SIGNALS and INFORMATION SIGNALS of standard IEC 60601-1-8.
The alarm melody of audible alarms signals shall conform to Table AAA.1 of standard IEC 60601-1-8 for drug or fluid delivery. “C d g” shall be used for medium priority alarms; “C d g - C d” shall be used for high priority alarms; “e c” shall be used for warnings and low priority alarms.3
Each tone in the alarm melody shall be composed of a minimum of 4 harmonic components in the range 300 Hz to 4000 Hz comprising an inverted 9th jazz chord.
Temporarily paused alarms shall reactivate Δap = 10 minutes after inactivation.
F. Functional Architecture
Following the REMH methodology, the PCA Pump functional architecture partitions system operation into smaller, simpler pieces, recursively. The PCA function is partitioned into the functional components in Table III, depicted in Figure 2.
TABLE III.
Functional Components
| Component | Behavior |
|---|---|
| fluid | holds and moves drug |
| operation | controls pump operation |
| safety | checks for faults; inhibits possibly hazardous infusion; signals alarms and warnings |
| power | coordinates battery and power supply; detects power anomalies |
Fig. 2.
PCA Functional Components
G. Safety Architecture
Another distinguishing feature of our requirements work is the illustration of a safety architecture. This is a notion that the FDA wishes to expose and illustrate to the academic and industrial communities. A medical device safety architecture is a hardware and software subsystem, separate from that which performs the normal operations of the device. The subsystem detects potential safety hazards, acts to prevent or mitigate a detected hazard, notifies a person that a hazard was detected, and records its occurrence for later investigation. In the case of a PCA pump, the safety subsystem detects faults that may harm the patient (e.g., improper flow rate, air bubbles in tube, etc.), signals an alarm or warning, and stops infusion or reduces infusion to a keep vein open rate depending on the fault(s) detected. The components in the safety system are listed in Table IV, and depicted in Figure 3.
TABLE IV.
Safety Components
| Component | Behavior |
|---|---|
| pump fault manager | handles pump fault signals |
| alarm process | holds thread which controls alarms |
| fault logger | record faults |
| error detector | handle hardware-detected faults |
| failure led | indicates hardware failure |
Fig. 3.
Safety Subsystem
IV. Ongoing and Future Work
The PCA requirements document forms the basis for creation of interrelated design artifacts to serve as public examples of model-based engineering of safety-critical medical devices.
The AADL model of the PCA Pump is being augmented with behaviors defined using Behavioral Language for Embedded Systems with Software (BLESS) annex subclauses and specifications using BLESS Assertion properties [15]. The goal of this work is compositional correctness proofs of high-level safety and efficacy properties.
We plan to add Error Modeling annex subclauses (EMV2) [16] to model fault initiation, error transmission and transformation, and failure occurrence. Failure modes and effect analysis (FMEA), fault-tree analysis (FTA), and other reliability or safety analyses can be applied to the architectural model. We also plan to link requirements to architectural elements with the Requirements Definition and Analysis Language (RDAL) [17] used by the RDAL Tool Environment (RDALTE) plug-in to the Open-Source AADL Tool Environment (OS-ATE) [5] provided by the Software Engineering Institute of Carnegie Mellon University. The assurance case for the PCA pump will trace to RDAL for evidence, and then to implementing architectural components and verification artifacts like tests and proofs.
The PCA pump model exemplifies safety architecture in [18]. Prototype hardware is being designed by Kansas State’s Electrical and Computer Engineering department. The model is subject for security architecture development, and definition of interoperability using AADL’s polymorphic type checking of architectural components.
Under the NSF FDA Scholar-In-Residence (SIR) program, we regularly meet with the U.S. FDA Office of Science and Engineering Laboratories (OSEL) to report research progress and receive guidance to ensure FDA can commend the artifacts as examples of good design. Assurance cases, required in submissions for FDA approval of infusion pumps, must be kept confidential to protect manufacturer’s trade secrets, thus cannot be used as examples of clear and convincing arguments that a pump is both safe and effective. Hopefully, the public PCA pump design artifacts will help manufacturers write applications for approval that FDA’s Office of Device Evaluation can easily and quickly ascertain that argument from evidence convincingly demonstrates the device(s) will be acceptably safe and effective.
Fig. 1.
Independant PCA Pump Use
TABLE V.
Top-Level Components
| Component | Behavior |
|---|---|
| ICE Bus Adaptor | translates events and data into bus transactions |
| PCA | performs pump operation |
| Maintenance | technician access to logs, drug library |
Acknowledgments
This work is supported in part by the National Science Foundation under Grants #0932289,1065887,1238431,1239543 and by the NIH/NIBIB Quantum program.
Footnotes
Subject to safety constraints.
The characters c, d, e, g, C refer to relative musical pitches and C is one octave above c.
Contributor Information
Brian R. Larson, Email: brl@ksu.edu.
John Hatcliff, Email: hatcliff@ksu.edu.
Patrice Chalin, Email: chalin@ksu.edu.
References
- 1.PCA Pump Requirements and Architecture. 2013 info.santoslab.org/-research/pca.
- 2.Boston Scientific. PACEMAKER system requirements specification. 2007 http://sqrl.mcmaster.ca/pacemaker.htm.
- 3.Lempia D, Miller S. Federal Aviation Administration. 2009. DOT/FAA/AR-08/32. requirements engineering management handbook. [Google Scholar]
- 4.Feiler PH, Gluch DP. Model-Based Engineering with AADL: An Introduction to the SAE Architecture Analysis & Design Language. Addison-Wesley; 2013. [Google Scholar]
- 5.Architecture Analysis & Design Language. 2013 www.aadl.info.
- 6.Larson BR, Chalin P, Hatcliff J. Bless: Formal specification and verification of behaviors for embedded systems with software. 2013 NASA Formal Methods Conference; ACM; 2013. pp. 99–105. [Google Scholar]
- 7.ASTM F2761-2009. Medical Devices and Medical Systems — Essential Safety Requirements for Equipment Comprising the Patient-Centric Integrated Clinical Environment (ICE), Part 1: General Requirements and Conceptual Model. ASTM International; 2009. [Google Scholar]
- 8.The Software Certification Consortium. 2013 website www.cps-vo.org.
- 9.Generic Infusion Pump Project Homepage. http://rtg.cis.upenn.edu/gip.php3.
- 10.Arney D, Jetley R, Jones P, Lee I, Sokolsky O. Formal methods based development of a pca infusion pump reference model: Generic infusion pump (gip) project. Proceedings of 2007 Joint Workshop on High Confidence Medical Devices, Software, and Systems and Medical Device Plug-and-Play Interoperability; jun 2007. [Google Scholar]
- 11.Hicks RW, Sikirica V, Nelson W, Schein JR, Cousins DD. Medication errors involving patient-controlled analgesia. American Journal of Health-System Pharmacy. 2008 Mar;65(5):429–440. doi: 10.2146/ajhp070194. [DOI] [PubMed] [Google Scholar]
- 12.J Commission. Preventing patient-controlled analgesia overdose. Joint Commission Perspectives on Patient Safety; October 2005.p. 11. [Google Scholar]
- 13.US FDA Infusion Pump Improvement Initiative. 2010 http://info.santoslab.org/research/pca.
- 14.Guidance for Industry and FDA Staff - Total Product Life Cycle: Infusion Pump - Premarket Notification [510(k)] Submissions (Draft Guidance) 2010 http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm206153.htm.
- 15.Larson BR. Behavior language for embedded systems with software annex sublanguage for aadl. 2013 info.santoslab.org/research/aadl/bless.
- 16.AADL Error Model Annex. SAE International; 2013. [Google Scholar]
- 17.AADL Requirements Definition and Analysis Language. SAE International; 2013. [Google Scholar]
- 18.Larson BR, Jones P, Zhang Y. Tech Rep. Kansas State and US FDA; Amherst, MA, USA: 2013. Medical device safety architecture. [Google Scholar]



