Characterization of Departures from Regulatory Requirements Identified During Inspections Conducted by the US Federal Select Agent Program, 2014–15

Abstract

We studied departures from regulatory requirements identified on US Federal Select Agent Program (FSAP) inspections to increase transparency regarding biosafety and security risk at FSAP-regulated entities and identify areas for programmatic improvement. Regulatory departures from inspections led by Centers for Disease Control and Prevention inspectors during 2014–15 were grouped into “biosafety,” “security,” and “other” observation categories and assigned a risk level and score reflecting perceived severity. The resulting 2,267 biosafety (n = 1,153) and security (n = 1,114) observations from 296 inspections were analyzed by frequency and risk across entity and inspection characteristics. The greatest proportion of biosafety observations involved equipment and facilities (28%), and the greatest proportion of security observations involved access restrictions (33%). The greatest proportion of higher-risk observations for biosafety were containment issues and for security were inventory discrepancies. Commercial entities had the highest median cumulative risk score per inspection (17), followed by private (13), academic (10), federal government (10), and nonfederal government (8). Maximum containment (BSL-4) inspections had higher median biosafety risk per inspection (13) than other inspections (5) and lower security risk (0 vs 4). Unannounced inspections had proportionally more upper risk level observations than announced (biosafety, 21% vs 12%; security, 18% vs 7%). Possessors of select agents had higher median biosafety risk per inspection (6) than nonpossessors (4) and more upper risk level security observations (10% vs 0%). Programmatic changes to balance resources according to entity risk may strengthen FSAP oversight. Varying inspection methods by select agent possession and entity type, and conducting more unannounced inspections, may be beneficial.

As of December 2017, the Federal Select Agent Program (FSAP) regulates the possession, use, and transfer of 66 biological select agents and toxins that have been determined by the US government to have the potential to pose a severe threat to public health, animal or plant health, or to animal or plant products.¹ Federal oversight of biological select agents and toxins is established by federal statute to decrease the risk of terrorism by constraining access to these agents and to decrease the risk of accidental release through enforcement of biosafety standards.² Common examples of biological select agents and toxins include the organisms that cause anthrax, Ebola virus disease, and bubonic plague, as well as the botulinum and ricin toxins. While these agents can be hazardous to health, work with them provides important scientific discoveries that have led to improved detection, prevention, diagnosis, and treatment options for diseases considered to be some of the most threatening to public and agricultural health. FSAP regulations allow laboratories to conduct lifesaving research on these materials while increasing assurance that it is done as safely and securely as possible.

The Division of Select Agents and Toxins (DSAT) of the Centers for Disease Control and Prevention (CDC) partners with Agriculture Select Agent Services (AgSAS)/Animal and Plant Health Inspection Service (APHIS) of the US Department of Agriculture (USDA) to administer FSAP. CDC oversees entities working with biological select agents and toxins that affect humans, APHIS oversees entities working with agents that affect agriculture, and CDC and APHIS collaborate to oversee entities working with agents that affect both humans and agriculture. FSAP encompasses the following functions to ensure the successful management of risk:

• Registration: An entity must be approved to possess, use, or transfer biological select agents and toxins. Their registration lays out their plans for biological select agents and toxins and how they will fulfill regulatory requirements, including local oversight, biosafety, security, reporting, and training needs. FSAP may suspend entities from working with biological select agents and toxins until corrective actions are completed or notify the appropriate authorities (eg, the Federal Bureau of Investigation [FBI]) for further investigation.

• Inspection: FSAP evaluates the current registration and, through on-site inspection, ensures appropriate measures are in place at each registered entity to prevent the unauthorized access to, theft or loss of, or unintended exposure to or release of biological select agents and toxins.

• Security: An individual who will have access to biological select agents and toxins must first undergo a security risk assessment performed by the FBI’s Criminal Justice Information Services Division. Personnel and physical security requirements help prevent access to biological select agents and toxins by bioterrorists or others likely to misuse them.

• Communication: All entities must promptly report instances of a theft, loss, or release of biological select agents and toxins outside of intended containment. These may include exposure (eg, a needlestick, spill, or animal bite), laboratory-acquired infection, or missing inventory. FSAP follows up with each entity to ensure that proper corrective actions are taken to prevent the incident from happening again. Requests to transfer biological select agents and toxins between entities must be approved in advance. Discovery of biological select agents and toxins in diagnostic samples must be reported whether or not the work was performed by a registered facility to ensure proper management of these materials until they can be secured at a registered facility or properly destroyed. FSAP also provides guidance to those working with biological select agents and toxins (including the publication of policies and guidelines), interprets the regulations to help entities meet requirements, and conducts training and outreach to increase knowledge of, and compliance with, the select agent regulations.

Three federal reviews released in October 2015, following high-profile incidents involving biological select agents and toxins that occurred at FSAP-regulated laboratories, included recommendations designed to strengthen the federal government’s biosafety and security practices and its oversight through FSAP.^3–5 The US Government Accountability Office also completed a review in 2016 of pathogen inactivation failures as a source of risk from high-containment laboratories.⁶ Each of these reports highlighted the importance of a risk-based approach to regulation to minimize unnecessary burdens and to direct resources where the need is greatest. These reports also called for improving data management and active analysis of program data to better understand opportunities for program improvement to reduce biosafety and security risks associated with the legitimate use of biological select agents and toxins.

To increase transparency regarding the biosafety and security conditions associated with biological select agents and toxins work and to identify areas for programmatic attention, we conducted a study of regulatory departures identified during inspections conducted by DSAT between January 1, 2014, and December 31, 2015.

Methods

Data Abstraction

Data were obtained from the National Select Agent Registry (NSAR) and transferred to a Microsoft Access relational database on February 4, 2016. DSAT-only inspections and DSAT-led joint inspections with AgSAS with start dates between January 1, 2014, and December 31, 2015, were included. Data describing laboratory characteristics and research activities for each entity at the study midpoint (January 1, 2015) were abstracted from entity registration documents.

Inspection reports include inspector-written narratives describing each finding that was deemed a departure from regulatory requirements (ie, deviation from an accepted standard of practice relevant to the select agent regulations). When stored in NSAR as image files (ie, scanned documents) rather than searchable data variables, departure narratives were manually copied and pasted into the database. Table 1 describes the variables abstracted and created for this analysis.

Table 1.

Entity and Inspection Data Variables Studied

Variable	Description of Data Entered
Inspection duration	Number of days
Inspection type	Compliance, maximum containment, new entity, new space, renewal, verification
Notification status	Announced inspection, unannounced inspection
Entity type	Academic, commercial, federal government, non—federal government, private (nonprofit)
Departure narrative	Inspector-written description of each departure
Observation category	Coded by the study team (see Table 2)
Observation subcategory	Coded by the study team (see Table 2)
Observation risk level and score	Level 1 = 1 point, level 2 = 2 points, level 3 = 4 points
Entity complexity scorea	Sum of the component scores below
Biosafety level of entity’s lab(s)	BSL-2 = 1 point, BSL-3 = 2 points, BSL-3 Ag = 3 points, BSL-4 = 3 points
Possession of any BSAT	No = ‒1 point
Possession of any Tier 1 BSAT	Yes = 1 point
Arthropod research	Yes = 1 point
Animal (nonarthropod) research	Yes = 1 point
Plant research	Yes = 1 point
Aerosols generated	Yes = 1 point
Large volume propagation/purification	1–10L = 0.5 points, >10L = 1 point
Large volume (≥10L) BSAT fermentation	Yes = 1 point
Commercial manufacturing of BSAT	Yes = 1 point
Restricted experiments involving genetic manipulation	Yes = 1 point
Research involving highly pathogenic avian influenza virus, reconstructed 1918 pandemic influenza virus, SARS coronavirus, African swine fever virus, and/or classical swine fever virus	Yes = 0.5 points

^aThe score was calculated based on National Select Agent Registry records at the study midpoint, January 1, 2015. BSAT = biological select agents and toxins; BSL = biological safety level.

The 5 entity types registered with FSAP are:

• Academic: a university, college, or other institution of higher learning

• Commercial: a for-profit company

• Federal government: an entity that is part of an agency of the federal government

• Non–federal government: an entity that is part of an agency of a state or local government (excluding academic entities)

• Private: a privately owned, nonprofit entity whose principal purpose is charitable or benevolent

The 6 inspection types conducted by FSAP are:

• Renewal: review, typically conducted every 3 years, of entire entity program, including all registered spaces and documents, to renew an existing registration

• Verification: intermediate review of entity program, including assessment of responses to previous inspection departures, which is conducted within 12 to 18 months of a renewal inspection

• Maximum containment: review of entity program, including laboratory spaces and documents, for biosafety level 4 (BSL-4) and animal biosafety level 4 (ABSL-4) laboratories

• New entity: review of all laboratory spaces and documents for an entity that is submitting a new application. The entity does not possess, use, or transfer biological select agents and toxins at the time of inspection.

• New space: review of laboratory space and documents for adding new laboratory space to an existing registration. No possession, use, or transfer of biological select agents and toxins occurs within the space at the time of inspection.

• Compliance: review of entity program for compliance issues, including inspections to investigate whistle-blower reports

Renewal, verification, and maximum containment inspections are the most consistent and complete inspections of laboratory spaces approved for the possession, use, or transfer of biological select agents and toxins. We refer to them as routine inspections.

Some entities registered for biological select agents and toxins by FSAP do not routinely possess them but register in order to be preapproved to possess, use, or transfer them should there be a need in the future. A “possessor” is an entity that holds biological select agents and toxins onsite at the time of the inspection. Tier 1 refers to specifically designated biological select agents and toxins (http://www.selectagents.gov/SelectAgentsandToxinsList.html) that represent the greatest risk of deliberate misuse with the most significant potential for mass casualties or devastating effects on the economy, critical infrastructure, or public confidence.⁷ Tier 1 entities are those that must meet more stringent requirements for security and biosafety and have been approved to possess Tier 1 biological select agents and toxins whether or not they possess them at the time of inspection.

Observation Categories

Departure narratives were assessed by an 8-person team of CDC scientists with laboratory research or inspection backgrounds who were not involved in the inspections reviewed. Regulatory departures were classified into 3 observation categories and 19 subcategories (Table 2). Single departures in an inspection report were classified as multiple observations when the departure narratives described deficiencies corresponding to more than 1 of the observation categories of this study. For example, a single departure narrative describing doors in a hallway with a decontamination airlock that did not self-close and lock was coded as 2 observations to describe both the biosafety concern (category: biosafety; subcategory: equipment and facilities) and the security concern (category: security; subcategory: restricted access).

Table 2.

Overview of Observation Categories Used to Classify Regulatory Departures Described by Inspectors in Inspection Reports

Category	Subcategory	Focus of Regulatory Departures Reported
Biosafety	Containment	Packaging, storage, aerosolization, decontamination, inactivation, airflow, HEPA filtration, animal housing, other containment issues
	Equipment and facilities	Equipment function/certification, exhaust system, mechanical and structural facilities, laboratory verification, ancillary supplies
	Hygiene and safety	Handwashing, sharps management, occupational health, pest control, personal protective equipment, labels, signage, environmental monitoring
	Plans and procedures	Completeness, date of last update, filing/posting location of required documents
	Records and reporting	Timeliness and completeness of required files, logs, certifications, FSAP form submissions
	Training and drills	Timing, frequency, activities/topics covered
Security	Inventory discrepancy	Consistency between observed inventory and recorded inventory
	Inventory records	Timeliness, completeness
	Operations	Alarm/monitoring systems, information systems, risk assessments, shipping and transport, other security operations
	Plans and procedures	Completeness, date of last update, filing/posting location of required documents
	Records and reporting	Timeliness and completeness of required files, logs, certifications, FSAP form submissions
	Restricted access	Restricted access requirements, security risk approval, personnel suitability assessment, visitor procedures
	Training and drills	Timing, frequency, activities/topics covered
Othera	Annual inspections	Timing, frequency, completeness of required entity-led annual inspections
	Entity registration	Timeliness, accuracy of entity’s FSAP registration information
	Plans and procedures	Completeness, date of last update, filing/posting location of required documents
	Records and reporting	Timeliness and completeness of required files, logs, certifications, FSAP form submissions
	Responsible official	Responsibilities of the entity’s responsible official
	Training and drills	Timing, frequency, activities/topics covered

^aObservations that do not fall strictly within the biosafety or security primary categories. FSAP = Federal Select Agent Program; HEPA = high-efficiency particulate air.

Conversely, multiple departures were combined into a single observation when closely related deficiencies were reported as multiple regulatory departures. For example, record-keeping requirements include inventory records that must capture and retain the agent or toxin name, the quantity acquired, the month it was acquired, the source from which it was acquired, the room number of storage, the dates of removal from storage, and the names of persons removing it from storage. Failure to record all these elements could lead to multiple regulatory departures in an inspection report, including failure to record an accurate, current inventory (42 CFR Section 73.17(a)(1)), the date of acquisition (Section 73.17(a)(1)(ii)), which freezer it is stored in (Section 73.17(a)(1)(iii)), and when it was returned after removal from storage (Section 73.17(a)(1)(iv)). Such related groups of departures were considered in concert and coded as a single observation (eg, category: security; subcategory: inventory records), although the observation risk level (described below) reflects the perceived magnitude of the aggregated departures. The combination of multiple departures into a single observation was far more common for plans/procedures and for training/drills issues than other observation subcategories, because the a priori decision was made to assess all biosafety plans/procedures deficiencies as a single observation and all biosafety training/drills deficiencies as a single observation for each inspection. The same process was used for coding plans/procedures and training/drills in the security and “other” observation categories.

Records were reviewed without the entity name; however, departure narratives occasionally contained entity identifiers, and blinding of reviewers was therefore not complete. Each departure narrative was processed by a single reviewer, but team members worked in the same room throughout the 3-week review process and regularly discussed the coding of departures to maximize consistency among reviewers.

Observation Risk Level

Each categorized observation was assigned to 1 of 3 risk levels of increasing perceived severity (levels 1–3). “Risk” in this context reflects the relative probability of an adverse biological select agent and toxin outcome (eg, theft, loss, or unintended exposure) and the consequence of that adverse outcome should it occur. Because no thefts have been identified in the history of FSAP and losses and worker exposures are infrequent,⁸ the assignment of a risk level was a subjective ranking of relative seriousness of the observation. While level 2 and level 3 observations represent the middle and upper categories of perceived risk assessed in this study, they do not indicate that an adverse event is necessarily likely to occur. A risk score was assigned immediately following the classification of a departure to an observation category and subcategory by the same reviewer, but, as described above, team members communicated throughout the process to maximize consistency. Parameters for each risk level were established in advance in consultation with DSAT inspectors. To calculate an overall risk score for each inspection and analyze inspection risk by inspection and entity characteristics, risk levels were weighted to reflect their nonlinear relative risk as follows: level 1 (1 point), level 2 (2 points), and level 3 (4 points). Inspection risk was defined as the sum of assigned points for the observations associated with each inspection.

Examples of level 1 observations include out-of-date or incomplete plans and procedural documents, partially incomplete inventory records (eg, incomplete recording of acquisition date or time of removal from, or return to, storage), recently expired equipment certifications or respiratory fit test certifications, and chips in wall paint or floor tiles. The identification of many level 1 observations during an inspection points to systemic management and operations problems and, ultimately, elevated risk. Examples of level 2 observations include significant deficiencies in the training entities provided for personnel or visitors, failure to remove outer gloves when exiting animal holding rooms, exposed sharps placed in a padded sheet or plastic bag for transport to an autoclave, absence of entire sections of required plans or procedural documents, and variance in the concentration of disinfectant found in dunk tanks. Examples of level 3 observations include a malfunctioning motion sensor inside an anteroom that made it possible to enter a laboratory without a cardkey, interviews with entity staff revealing that work with toxin solutions was sometimes performed on the open bench, failure to record entries by approved individuals or visiting scientists into a select agent storage room, smoke-testing of a biosafety cabinet that revealed air flowing outward at the front opening into the laboratory room, and discovery of vials of toxin that the entity was not registered to possess.

Context provided in departure narratives influenced the risk level assigned by the review team. For example, discovery of a damaged or missing o-ring for a centrifuge rotor was coded as being at risk level 1 if the machine was still present in the laboratory but clearly labeled as out-of-order while awaiting repair. The observation was coded as level 2 if no such precaution was being taken.

Complexity Score

DSAT inspectors identified 12 entity characteristics that describe the type and scale of work conducted. The presence or absence of each of the 12 components contribute to entity complexity because they indicate associated safeguards the entity is required to implement or install to reduce biosafety or security risk. Data for the 12 variables at the study midpoint were retrieved from registration information in NSAR. Point values were assigned to each variable and summed to determine a complexity score for each entity (Table 1). The points assigned to each variable are not based on a standardized scale of complexity, but their sum is meant to capture inter-entity variation in risk potential, with higher complexity scores indicating higher risk.

Statistical Analyses

Study data are summarized by median values, and non-parametric statistical methods were used for comparisons across entity and inspection characteristics, because the null hypothesis of normality was rejected for all test variables (P < .001) using the Shapiro-Wilk test for normality.⁹ Statistical significance of differences in inspection risk (ie, the sum of observation risk scores for each inspection) across entity and inspection characteristics was assessed using the Kruskal-Wallis rank sum test.¹⁰ For tests significant at α = .05 for characteristics with more than 2 categories, the post-hoc Conover-Iman test of multiple pairwise comparisons¹¹ was used with Benjamini and Hochberg adjustment for false discovery.¹² The G-test of statistical independence¹³ was used to assess the significance of differences in the distribution of observation counts by risk level (ie, level 1, level 2, and level 3) across entity and inspection characteristics. For tests significant at α = .05 for characteristics with more than 2 categories, post-hoc pairwise G-tests were used with the Benjamini and Hochberg adjustment for false discovery. The best-fit linear relationship between inspection risk and entity complexity scores and between inspection risk and inspection duration were assessed using ordinary least squares regression.

Final Dataset

A total of 230 FSAP-registered entities were involved in the 324 inspections meeting the study criteria, for a mean of 1.4 inspections/entity during the 2-year study period. These inspections contained 5,061 regulatory departures, which were categorized into 3,280 observations. The decrease in number from raw departures to classified observations resulted from more instances of multiple departures being combined and classified as 1 observation than of single departures being split into multiple observations. The total number of observations per inspection ranged from 1 to 37, with a median of 9 observations per inspection. The 3,280 observations included 1,254 (38%) biosafety observations, 1,183 (36%) security observations, and 843 (26%) “other” observations (described in Table 2) not clearly attributable to biosafety or security.

Academic entities were inspected most frequently, representing 111 (34%) of 324 inspections, followed by 88 (27%) non–federal government, 53 (16%) federal government, 48 (15%) commercial, and 24 (7%) private inspections. These proportions are comparable to those of all entities registered with FSAP at the study end-point⁷ (32% academic, 29% non-federal government, 15% federal government, 18% commercial, and 6% private). While 57% (130) of the study’s 230 entities were registered as Tier 1 with DSAT, Tier 1 entities represented a slightly higher percentage of the 324 total DSAT inspections conducted (209, or 65%).

Routine inspections combined to represent 296 (91%) of the 324 total inspections (maximum containment: 11 [3%]; renewal: 175 [54%]; verification: 110 [34%]). Nineteen (6%) compliance and 9 (3%) new entity or new space inspections made up the remainder. The range (1 to 37) and median (9) of observation frequency per inspection for the 296 routine inspections were equivalent to the range and median for all 324 inspections combined. The final dataset included biosafety and security observations from routine inspections only.

Results

A combined total of 2,267 biosafety and security observations were classified across all 296 routine inspections (median = 6 observations/inspection). By entity type, commercial entities had the highest median combined (ie, biosafety plus security) observation frequency and the highest median risk per inspection (9 observations/inspection; inspection risk: 17), followed by private (7 observations/inspection; inspection risk: 13), academic (6 observations/inspection; inspection risk: 10), federal government (6 observations/inspection; inspection risk:10), and non–federal government (5 observations/inspection; inspection risk: 8) entities. The median combined observation frequency per inspection was similar across inspection types (verification: 6.5; renewal: 6; maximum containment: 6). The median combined per-inspection frequency of observations was slightly greater for possessors of biological select agents and toxins (6) than for nonpossessors (4), for entities registered for Tier 1 biological select agents and toxins (6) than for non-registered entities (5), and for unannounced inspections (7) than for announced inspections (6). The median frequencies of biosafety observations (3) and security observations (3) per inspection were the same; however, proportionally more biosafety observations were scored above risk level 1 than security observations (70% vs 57%), resulting in a higher per-inspection biosafety risk (6) than security risk (4) (Table 3).

Table 3.

Distribution of Biosafety and Security Observations by Entity and Inspection Characteristics and by Observation Risk Level

					Observation Riska
						level 1		Level 2		Level 3
Entity or Inspection Characteristic	No. Observations	No. Inspections	No. Observations/Inspectionb	Risk/Inspectiona,b		No. Observations	Percent of Row	No. Observations	Percent of Row	No. Observation	Percent of Row
All	2,267	296	6 (4–9.8)	10 (6–19)		820	36%	1,171	52%	276	12%
Biosafety observations	1,153	296	3 (2–5)	6 (3–10)	a	340	29%	646	56%	167	14%	A
Entity Type
Academic	416	100	3 (2–6)	6 (4–12)	a	118	28%	234	56%	64	15%	A
Commercial	206	43	4 (2–7)	7 (3–13)	a	53	26%	116	56%	37	18%	A
Federal government	183	48	2 (1–5.8)	4 (2–11.5)	ab	54	30%	98	54%	31	17%	A
Non—federal government	247	84	3 (2–4)	4 (3–7)	bc	91	37%	139	56%	17	7%	B
Private	101	21	5 (3–7)	7 (5.5–13.5)	a	24	24%	59	58%	18	18%	A
Inspection T
Maximum containment	65	11	6 (3–8)	13 (7–26)	a	6	9%	34	52%	25	38%	A
Renewal	636	175	3 (1–5)	5 (2–8)	b	211	33%	347	55%	78	12%	B
Verification	452	110	3.5 (2–5)	6 (3.8–10)	c	123	27%	265	59%	64	14%	B
Inspection Notification Status
Announced	862	231	3 (2–5)	5 (2–10)	a	263	31%	493	57%	106	12%	A
Unannounced	291	65	4 (2–6)	7 (4–12)	b	77	26%	153	53%	61	21%	B
BSAT Possession Status
Possessor	1,106	275	3 (2–5)	6 (3–11)	a	323	29%	620	56%	163	15%	A
Nonpossessor	47	21	2 (1–3.5)	4 (2–6.5)	b	17	36%	26	55%	4	9%	A
Tier 1 BSAT Registration Status
Registered	789	186	3 (2–6)	6 (3–12)	a	232	29%	441	56%	116	15%	A
Not registered	364	110	3 (1.8–4)	5 (2–8.3)	b	108	30%	205	56%	51	14%	A
Security observations	1,114	296	3 (1–5)	4 (2–8)	b	480	43%	525	47%	109	10%	B
Entity Type
Academic	359	100	3 (2–4)	4 (2–6)	a	169	47%	163	45%	27	8%	A
Commercial	254	43	5 (3–8)	8 (4–16)	b	95	37%	123	48%	36	14%	B
Federal government	148	48	2 (1–5)	4 (1–8)	a	50	34%	80	54%	18	12%	B
Non—federal government	278	84	3 (1–4.8)	4 (2–7)	a	141	51%	125	45%	12	4%	A
Private	75	21	2 (1–6)	4 (1.5–10)	a	25	33%	34	45%	16	21%	B
Inspection Type
Maximum containment	6	11	0 (0–1)	0 (0–2)	a	2	33%	3	50%	1	17%	A
Renewal	694	175	3 (2–5)	5 (2–8)	b	303	44%	332	48%	59	9%	A
Verification	414	110	3 (1.8–5)	4 (2–8.3)	b	175	42%	190	46%	49	12%	A
Inspection Notification Status
Announced	840	231	3 (1–5)	4 (2–8)	a	376	45%	405	48%	59	7%	A
Unannounced	274	65	3 (1.5–5.5)	4 (2–11.5)	a	104	38%	120	44%	50	18%	B
BSAT Possession Status
Possessor	1,066	275	3 (2–5)	4 (2–8)	a	459	43%	498	47%	109	10%	A
Nonpossessor	48	21	2 (1–3)	3 (2–5)	a	21	44%	27	56%	0	0%	B
Tier 1 BSAT Registration Status
Registered	747	186	3 (1–5)	4 (2–9)	a	316	42%	355	48%	76	10%	A
Not registered	367	110	3 (2–5)	4 (2–7)	a	164	45%	170	46%	33	9%	A

^aLower case and upper case letters indicate results of pairwise comparisons. A lack of any matching letters between 2 entity or inspection characteristic subcategories within the same parent category represents a significant pairwise difference at α = .05. Analyses of risk per inspection (ie, the observation risk sum for each inspection), indicated by lower case letters, represent results of the Kruskal-Wallis rank sum test and the Conover-Iman test of multiple pairwise comparisons with Benjamini and Hochberg adjustment for false discovery. Analyses of the distribution of observation counts across risk levels 1, 2, and 3, indicated by upper case letters, represent results of the G-test of statistical independence with Benjamini and Hochberg adjustment for false discovery. For example, when considering all observations, the biosafety risk per inspection is significantly greater than the security risk per inspection (a vs b), and the distribution of biosafety observation counts across risk levels is skewed significantly less toward level 1 than is the distribution of security observation counts (A vs B). When considering only biosafety observations and comparing across entity types, the inspection risk scores of academic, commercial, federal government, and private entities do not differ significantly (all have an a), and the inspection risk scores of non-federal government entities (the only entity type with a c) are significantly lower than all other entity types except federal government entities (because they share a b). The distribution of biosafety observation counts across risk levels among non-federal government entities (B) is significantly different (ie, skewed more toward level 1) than are the distributions of all other entity types (all have an A).

^bPresented as median (interquartile range).

There were 1,153 biosafety-related observations and 1,114 security-related observations (Figure 1). A combined 74% of biosafety observations were attributed to equipment and facility (28%), biological select agents and toxins containment (24%), or hygiene and safety (22%) issues (see Table 2 for observation category descriptions). The biosafety categories with both the highest raw number and percentage of observations scored above risk level 1 were containment (258 observations, 93%) and equipment and facilities (233 observations, 71%). A combined 81% of security observations were attributed to restricted access (33%), inventory records (29%), or plans and procedures (19%) issues (see Table 2). The security categories with the highest percentage of observations scored above risk level 1 were inventory discrepancy (79%) and security operations (77%). Restricted access and inventory records issues accounted for the highest raw number of security observations scored above risk level 1 (249 [68%] and 167 [51%] observations, respectively). Plans and procedures issues accounted for the lowest percentage of observations scored above risk level 1 among both biosafety (49%) and security (32%) observations.

Figure 1.

Number of Observations by Observation Category and Risk Level. Biosafety and security observations (N = 2,267) from all 296 routine inspections are grouped by observation subcategory and risk level (Level 1, blue; Level 2, red; Level 3, green). Numbers above each set of bars represent the sum of observations across the 3 risk levels for each observation subcategory.

The frequency of observations and the observation and inspection risk varied by entity and inspection characteristics (Table 3). Private and commercial entities had the highest frequency of biosafety observations per inspection (5 and 4, respectively). The median biosafety risk per inspection was significantly higher for private and commercial entities when compared to non-federal government entities (biosafety risk/inspection: 7 vs 4; percentage of biosafety observations scored at risk level 3: 18% vs 7%). The median frequency of biosafety observations per inspection was greater among the 11 maximum containment inspections (6) than among renewal (3) and verification (3.5) inspections. The distribution of observations among maximum containment inspections was significantly skewed toward risk level 3 (38% of observations) relative to renewal and verification inspections combined (13%), which is also evident when comparing the median biosafety risk per inspection (13 vs 5). The percentage of observations scored at risk level 3 was 1.8 times higher for unannounced (21%) relative to announced (12%) inspections. Entities possessing biological select agents and toxins at the time of the inspection had higher median biosafety observations per inspection (3 vs 2) and biosafety risk per inspection (6 vs 4) compared with entities not possessing biological select agents and toxins. Entities registered and not registered to work with Tier 1 biological select agents and toxins each had a median of 3 biosafety observations per inspection, but Tier 1 entities had slightly higher median biosafety risk per inspection (6 vs 5).

Commercial entities had 1.7 times the median frequency of security observations per inspection (5) compared to all other entity types combined (3) and double the median security risk per inspection (8 vs 4). Compared with renewal and verification inspections combined, the median per-inspection security observation frequency and the per-inspection security risk for maximum containment inspections were lower (median observation frequency: 0 vs 3; median risk total: 0 vs 4). Unannounced inspections had the same median frequency of security observations per inspection (3) and the same median security risk per inspection (4) as announced inspections, but had 2.6 times the percentage of risk level 3 observations (18% vs 7%). As with biosafety observations, security observations were proportionately more serious among biological select agent and toxin-possessing versus nonpossessing entities, with proportionally more risk level 3 observations (10% vs 0%) and a higher median security risk per inspection (3 vs 2). Neither the security risk nor the risk distribution of security observations differed significantly among Tier 1-registered versus non-Tier 1 entities.

Complexity scores for inspected entities ranged from −1 to 11.5 with a median of 3. The median complexity of entities inspected varied by entity type (academic: 4; commercial: 3; federal government: 3; non-federal government: 2; private: 5.5), inspection type (maximum containment: 9.5; renewal: 3; verification: 3.5), biological select agent and toxin possession status (possessors: 3.5; nonpossessors: 1), and Tier 1 registration status (registered: 4; not registered: 2). The median complexity of entities inspected was the same for announced and unannounced inspections (3). Despite significant correlations, entity complexity score explained little of the variation in biosafety and security risk per inspection (biosafety R² = .16, security R² = .03) (Figure 2). No meaningful change was seen when maximum containment inspections (ie, the inspections with the highest median complexity score) were removed from the regression. Similarly, while the linear regressions were significant, inspection duration explained little of the variation in biosafety or security risk (biosafety R² = .12, security R² = .01). Maximum containment inspections were longer (median = 8 days) than renewal (2 days) and verification (2 days) inspections. When maximum containment inspections were removed from the regression, the correlations strengthened but remained low (biosafety R² = 0.14, security R² = 0.10).

Figure 2.

Relationship of Risk per Inspection to Entity Complexity and Inspection Duration. Ordinary least squares regressions, while significant, described little of the variation in relationships between risk per inspection and entity complexity score for biosafety and security observations combined (y = 1.76x + 7.88; R² = 0.10; df = 294; P < .001), (a) biosafety observations only (y = 1.22x + 3.22; R² = 0.16; df = 294; P < .001), or (b) security observations only (y = 0.53x + 4.66; R² = 0.03; df = 294; P = .004), or between risk per inspection and inspection duration for biosafety and security observations combined (y = 2.76x + 6.81; R² = 0.07; df = 294; P < .001),(c) biosafety observations only (y = 2.04x + 2.16; R² = 0.12; df = 294; P < .001), or (d) security observations only (y = 0.72x + 4.65; R² = 0.01; df = 294; P = .039). See text for additional regression test results.

Discussion

The major contribution of this study—the first risk-based review of Federal Select Agent Program inspection findings—is the distillation of detailed departure information into observation categories and risk scores amenable to quantitative analysis and reporting. Descriptive details were lost when aggregating departures into broad observation categories, but patterns emerged, and potential lessons and actionable conclusions emerged for strengthening biosafety and security oversight at FSAP-regulated entities. The DSAT program has been implementing changes since late 2015 to better understand risk in FSAP-regulated facilities and to strengthen risk-based oversight.¹⁴ It is hoped that this study will help to further those efforts.

When analyzing biosafety and security observations together, for example, commercial and private entities had the highest—and non-federal government entities had the lowest—observation frequency per inspection and risk per inspection. This suggests that inspection effort, including duration and frequency, might be better weighted by entity type than applied equally across the universe of registered entities. Nonpossessors of biological select agents and toxins also had substantially fewer observations and lower risk. Given the absence of biological select agents and toxins in storage or use, inspections of nonpossessing entities do not allow for direct observation and review of biological select agent and toxin work or security procedures, so it is likely that these inspections offer less opportunity for departure findings than inspections of active facilities. Alternative inspection protocols might therefore be considered for these entities until they perform work on biological select agents and toxins. While Tier 1 registration requires additional security protections, and thus provides a greater opportunity for security observations, comparisons of Tier 1 and non-Tier 1 inspections revealed no differences in the median number or risk of security observations, suggesting that Tier 1 entities have successfully stepped up their security to meet the enhanced requirements.

The entity complexity score demonstrated low predictive value for biosafety risk in this study and even lower predictive value for security risk. The entity complexity score was an attempt by DSAT inspectors to quantify and standardize opportunity for regulatory departures based on the additional biosafety and security procedures, installations, and safeguards required of entities as the scope of biological select agent and toxin work expands. Greater precision in weighting entities by risk is likely possible by further exploring the underlying factors of risk, beyond those assessed in this study.

Maximum containment (BSL-4) inspections, while few in number, differed significantly from renewal and verification inspections. This is not surprising given the duration and intensity of the inspections, which include the review of engineering controls and practices for biosafety inherent in the standards for a maximum containment laboratory. The more stringent access controls and physical barriers at maximum containment facilities likely contributed to the lower security risk per inspection observed at these facilities.

Even following the extensive combining of related departures identified by inspectors describing deficiencies in plans/procedures and training/drills when coding observations for this study (described in Methods), observations in these subcategories still combined to represent 24% of all biosafety and security observations (Figure 1). Inspectors invest a great deal of time in the review of entity plans and standard operation procedures and in the review of records and materials from training sessions and drills. Further evaluation is needed of the opportunity costs of this work compared with the direct observation of biosafety and security conditions and activities.

While announced inspections provide a predictable timeline for entities to review their biosafety and security practices on a regular basis and make improvements in preparation for their inspection, FSAP is able to schedule routine inspections only every 18 months. That frequency of review and improvement may not be seen as satisfactory for acceptable biosafety or security risk management. The unannounced inspection has the potential to increase routine attentiveness of internal entity oversight programs to biosafety and security practices. As demonstrated in this study, the unannounced inspection identifies greater numbers of biosafety observations and higher-risk findings than announced inspections, suggesting a more realistic view of routine practice and greater opportunities for improvement. Unannounced inspections are more challenging to arrange and can be more disruptive to regulated entities; however, they appear to be an important component of a risk-based oversight program.

Because regulatory departures identified in actual inspection reports were altered to categorize and score relative severity by a team of staff who did not participate in the actual inspections, these results do not reflect the exact number or characterization of inspection findings. The severity scoring differentiates relative levels of risk, but even the highest category of risk may not reflect a regulatory departure putting biological select agent and toxin security or worker safety at meaningful risk. These findings should be interpreted in a comparative rather than absolute sense. Considerable efforts were made to maintain consistency when coding departure narratives into observation categories and risk levels. Some variation in observation coding among study team members likely remained, but any such variation would probably have been distributed randomly across inspections.

This study primarily assessed the relationship of the entity and inspection characteristics with risk in a dichotomous fashion. The factors associated with biosafety and security risk are undoubtedly greater. More sophisticated, multivariate analytic techniques may help to refine the understanding of risk characteristics needing increased attention. For example, nonpossessors may be more likely to have other low-risk factors or be treated differently during inspection. Additional analyses will also help to understand how risk—as reported during inspections and as quantified for this study—relates to reported biological select agent and toxin loss or release incidents at FSAP-regulated entities. Improvements in data management by FSAP should allow more thorough and routine analysis of inspection findings in the future.

Conclusion

The primary purpose of this report is to ensure that a central aspect of the biological select agent and toxin oversight process, the facility inspection, is more transparent, allowing for a better understanding of the risk associated with this work. We have described the nature and relative risk of regulatory departures identified from 2014 through 2015 and identified several areas of variation that can inform further study and programmatic attention. This initial analysis has identified factors that differentiate risk associated with inspection findings and allows consideration to apply inspection resources differently by entity type, inspection type, and biological select agent and toxin possession status. The oversight requirements of nonpossessors of biological select agents and toxins and the non-federal government entities, such as public health laboratories involved in diagnosis and emergency response, appear to be different than other entities and may warrant changes to the oversight process. Further consideration is warranted to increasing the frequency of unannounced inspections so inspectors are more likely to observe routine day-to-day operations that pose biosafety or security risk. This could result in improved biosafety and security by increasing the specificity and impact of post-inspection remediation efforts recommended by inspectors. It could also help FSAP to prioritize and customize its training, guidance, and outreach efforts. Assessing factors associated with biosafety and security risk in greater detail may allow for a more efficient use of program resources to identify and correct biosafety and security risks.