Assessing the Reproducibility of Research Based on the Food and Drug Administration Manufacturer and User Facility Device Experience Data

Abstract

Objective:

This article aims to assess the reproducibility of Manufacturer and User Facility Device Experience (MAUDE) data-driven studies by analyzing the data queries used in their research processes.

Methods:

Studies using MAUDE data were sourced from PubMed by searching for “MAUDE” or “Manufacturer and User Facility Device Experience” in titles or abstracts. We manually chose articles with executable queries. The reproducibility of each query was assessed by replicating it in the MAUDE Application Programming Interface. The reproducibility of a query is determined by a reproducibility coefficient that ranges from 0.95 to 1.05. This coefficient is calculated by comparing the number of medical device reports (MDRs) returned by the reproduced queries to the number of reported MDRs in the original studies. We also computed the reproducibility ratio, which is the fraction of reproducible queries in subgroups divided by the query complexity, the device category, and the presence of a data processing flow.

Results:

As of August 8, 2022, we identified 523 articles from which 336 contained queries, and 60 of these were executable. Among these, 14 queries were reproducible. Queries using a single field like product code, product class, or brand name showed higher reproducibility (50%, 33.3%, 31.3%) compared with other fields (8.3%, P = 0.037). Single-category device queries exhibited a higher reproducibility ratio than multicategory ones, but without statistical significance (27.1% versus 8.3%, P = 0.321). Studies including a data processing flow had a higher reproducibility ratio than those without, although this difference was not statistically significant (42.9% versus 17.4%, P = 0.107).

Conclusions:

Our findings indicate that the reproducibility of queries in MAUDE data-driven studies is limited. Enhancing this requires the development of more effective MAUDE data query strategies and improved application programming interfaces.

To monitor and identify potential safety issues associated with the use of medical devices, the Food and Drug Administration (FDA) collects reports of device malfunctions, adverse events, and deaths and maintains the reports in the Manufacturer and User Facility Device Experience (MAUDE) database. The MAUDE database is a valuable and publicly available resource for patient safety research. The MAUDE data can be leveraged to identify patterns and trends that may indicate a need for further safety investigation or regulatory action.^1–3

The use of MAUDE-based studies to investigate patient safety issues has grown rapidly in recent years. These studies have provided valuable insights into the adverse events associated with medical devices and have been used to inform medical device safety training and regulatory decisions.^4–6 However, concerns have been raised about the transparency and reproducibility of the data extraction process used in these studies.^7–9 To improve the credibility and trustworthiness of MAUDE-based studies, it is important to assess the reproducibility of the queries used to create study cohorts. This is considered the first step in ensuring the reproducibility of MAUDE-based studies.

Query reproducibility could be impacted by various factors, including query design, research topic, or data processing flow. An in-depth investigation of those factors would provide an insightful understanding of query reproduction. This article reports our comprehensive investigation on assessing query reproducibility in MAUDE-based studies. Our study creates measurements for query reproducibility, sheds light on why data queries could not be reproduced, and provides potential solutions for addressing the problems of query reproducibility.

MATERIALS AND METHODS

Search Strategy for MAUDE-Based Studies

The key words connected with Boolean operators expressed as “(MAUDE[Title/Abstract]) OR (Manufacturer and User Facility Device Experience[Title/Abstract])” were used in PubMed to retrieve MAUDE-based articles. Articles containing queries to extract MAUDE data were identified through manual review, followed by a summary statistics analysis.

Selecting Articles With Executable Data Queries

The queries in each of the retrieved articles were reviewed according to the criteria listed in Table 1. Based on the criteria, articles with inexecutable queries were excluded.

TABLE 1.

Criteria to Exclude Inexecutable Queries

Query Property	Criteria for Inexecutable Queries
Timespan	a) Queries did not provide a precise time range for MDR retrieval. b) Queries contained MDRs 10 y earlier than the time the reproduction experiments were conducted. (MAUDE API, see details in Supplementary Materials S1, http://links.lww.com/JPS/A617) c) Queries presented start and end dates with defined days, months, and years, which are not supported by the simply search. (MAUDE API, see details in Supplementary Materials S1, http://links.lww.com/JPS/A617)
Query fields	a) Queries did not clearly describe the fields used. b) Queries used simple search and advanced search APIs simultaneously. (MAUDE API, see details in Supplementary Materials S1, http://links.lww.com/JPS/A617)
No. MDRs reported	a) Queries did not report the exact number of MDRs reported. b) Queries provided the number of MDRs after postquery data processing and missed the raw number of MDRs returned right after the query was executed. c) The number of returned MDRs shown in the articles was too large to be reproduced (the threshold of the number was set to 2.9 million).

The review process was performed by 2 authors and finalized through group discussions involving domain experts with expertise in patient safety, health informatics, and medicine. The articles containing executable queries were maintained for further analysis. An example of executable query described in a article¹⁰ appears as: “A review of adverse events associated with the Watchman device was performed using the MAUDE database from March 2015 (FDA approval date) through February 2019. Within the database, the search was conducted using the term ‘XXX’ in the ‘brand name’ section to achieve the broadest possible search.”

Of 523 articles retrieved from PubMed, 336 featured a query to the MAUDE database. From 336 articles, 21 that downloaded MAUDE data files were set aside for further analysis.^11–31 Upon manual review, 135 articles with unclear queries^32–164 and 120 with incomplete queries^165–284 were excluded. Ultimately, 60 executable queries were found.^{5,10,285–342} A comprehensive flow-chart of this selection process is depicted in Figure 1.

FIGURE 1.

The flowchart shows the process of identifying articles with reproducible queries.

Measuring the Reproducibility of Executable Queries

The executable queries were reproduced by using the MAUDE Application Programming Interface (API) fields with exact key words as described in the original articles. The reproducibility of the queries was measured by comparing the number of medical device reports (MDRs) returned by the reproduced queries with the number reported in the original articles. Specifically, reproducibility is defined by the reproducibility coefficient as follows:

$$\text{Reproducibility coefficient =}\frac{\text{The number of MDRs returned by the reproduced query}}{\text{The number of MDRs reported in the paper}}$$

Given the FDA’s routine cleansing of received MDRs, identical queries conducted precleaning and postcleaning may yield varying MDR counts, a ± 5% threshold was applied to determine the query reproducibility. Specifically, a query is considered reproducible if:

$$0.95\le \text{reproducibility coefficient}\le 1.05$$

An example of our workflow to determine the reproducibility of a query is elaborated in Supplementary S1, http://links.lww.com/JPS/A597.

Evaluating Reproducibility Ratios: Intrasubgroup Assessment and Intersubgroup Comparisons

The reproducibility coefficient of each query could be influenced by factors such as the complexity of the query, the category of the device involved, and whether there is a data processing flow included in the study. Therefore, we divided the articles with executable queries into subgroups based on the query complexity, the device category, and the presence of the data processing flow. The reproducibility ratio, defined as the proportion of reproducible queries in a subgroup, was calculated for each subgroup and compared via χ² tests.

Query Complexity: API Fields and Estimated API Runs

The complexity of a query, which can influence its reproducibility, is affected by the number of API fields used. Single-field queries use one API field, while multifield queries use 2 or more. The choice of specific fields (like brand name or product code) also adds to the complexity. In addition, the estimated number of runs (ENRs) required to complete a query is another factor, given the limitations of the MAUDE API (detailed in Supplementary S2, http://links.lww.com/JPS/A597). Articles are categorized based on single- versus multifield usage, field selection, and ENR levels.

Device Category

The focus of an article on either a single device category or multiple device categories could affect the reproducibility of its queries. Consequently, the articles are categorized into subgroups of single- and multicategory device focus.

The Presence of a Data Processing Flow

The data processing flow describes how the raw data (returned by queries) are processed in each study step. The presence of a data processing flow can improve the transparency and reproducibility of a study. The articles are divided into subgroups with and without a data processing flow.

RESULTS

Geographic and Expertise Diversity in MAUDE-Based Research Publications

The 336 surveyed articles, categorized by the corresponding authors’ home countries, originated from 18 different countries/international organizations. The majority, with 299 articles, were from the United States. Figure 2 illustrates the nationality distribution and the annual comparison of U.S. and non-U.S. article counts. Despite MAUDE being a U.S. initiative, it has garnered global research interest, evidenced by 28 articles from 14 non-U.S. countries and 8 internationally collaborative publications.

FIGURE 2.

Distribution of nationalities in 336 MAUDE research articles (top) and annual comparison of U.S. and non-U.S. article counts (bottom).

We examined the expertise of the corresponding authors among the 336 articles, discovering that 283 were authored by experts from 18 medical fields (such as dermatology and ophthalmology), while 53 articles had authors from 15 nonmedical disciplines (including informatics, biomedical engineering, and statistics, as well as government and corporate entities). Figure 3 presents the breakdown of these articles across the identified 18 medical and 15 nonmedical fields. Notably, nearly half of the medical-related publications were predominantly contributed by authors specializing in surgery and cardiovascular fields.

FIGURE 3.

Distribution of 336 corresponding authors’ expertise: 18 medical fields (left) and 15 nonmedical domains (right).

Reproducibility Coefficients and Ratios

The reproducibility coefficients of 14 articles^10,285–297 are in the range of 0.95 to 1.05. Results of reproducibility ratios for each subgroup are shown in Table 2.

TABLE 2.

Reproducibility Ratios of Subgroups Divided by Query Complexity, Device Category, and the Presence of Data Processing Flows

No. Articles With Executable Queries (n = 60)	No. Articles With Reproducible Queries
Query complexity
Single-field query (n = 51)	12 (23.5%)
Simple search (n = 23)	2 (8.7%)
Brand name (n = 16)	5 (31.3%)
Product class (n = 3)	1 (33.3%)
Product code (n = 8)	4 (50.0%)
Manufacturer (n = 1)	0 (0.0%)
Multifield query (n = 9)	2 (22.2%)
Brand name + event type (n = 1)	0 (0.0%)
Brand name + manufacturer (n = 1)	0 (0.0%)
Brand name + simple search (n = 1)	0 (0.0%)
Brand name + product code (n = 1)	0 (0.0%)
Event type + product class (n = 1)	1 (100.0%)
Manufacturer + product code (n = 2)	1 (50.0%)
Product class + product code (n = 1)	0 (0.0%)
Brand name + event type + product code (n = 1)	0 (0.0%)
ENRs
1 (n = 11)	4 (36.4%)
2 (n = 9)	2 (22.2%)
3 (n = 4)	1 (25.0%)
4 (n = 5)	3 (60.0%)
5 (n = 7)	1 (14.3%)
6 (n = 2)	1 (50.0%)
7 (n = 3)	1 (33.3%)
≥8 (n = 19)	1 (5.3%)
Device category
Single category (n = 48)	13 (27.1%)
Multicategory (n = 12)	1 (8.3%)
Presence of data processing flows
Show (n = 14)	6 (42.9%)
No show (n = 46)	8 (17.4%)

Single-field queries use only 1 API field, whereas multifield queries use 2 or more API fields. The ENRs required to complete a query is based on the limitations of the MAUDE API (see Supplementary S2, http://links.lww.com/JPS/A624). A single device category implies that the device under investigation falls into one specific category. In contrast, multiple device categories indicate that the device pertains to several categories. The term “presence of data processing flow” denotes whether the article describes the methods used to handle the MDRs retrieved through these queries.

Single- Versus Multifield Queries

Twelve of the 51 single-field queries (23.5%) were reproducible, while 2 of the 9 multifield queries (22.2%) were reproducible. The difference between the reproducibility ratio of single- and multifield queries (P = 1.000) is insignificant.

Individual API Field

Simple search (23), brand name (16), and product code (8) were the top 3 most frequently used fields in the single-field query. Queries using product code (reproducibility ratio, 50.0%), product class (33.3%), and brand name (31.3%) had the highest reproducibility ratios in the single-field query. Single-field queries using these three fields had significantly higher reproducibility than those using other fields (37.0% versus 8.3%, P = 0.037). For multifield queries, the query using event type + product class was reproducible; however, 1 of the 2 queries using manufacturer + product code was reproducible.

Queries With Different ENRs

Queries with ENR greater than 7 had extremely low reproducibility ratio. Only 1 of the 19 queries was reproducible.

Single- Versus Multicategory Queries

There were 48 single-category and 12 multicategory queries. Thirteen of the 48 single-category queries (27.1%) were reproducible, while 1 of the 12 multicategory queries (8.3%) was reproducible. Single-category queries had higher reproducibility ratio than the multicategory queries without a significant difference (P = 0.321).

Presence of a Data Processing Flow

Fourteen of the 60 articles (25.0%) included a data processing flow. Six of the 14 queries (42.9%) were reproducible, while 8 of the 46 queries (17.4%) without a data processing flow were reproducible. Queries with a data processing flow had higher reproducibility ratio than those without a data processing flow. However, there is no significant difference (P = 0.107).

DISCUSSION

Principal Findings

This article presented an analysis to assess reproducibility of queries used in the MAUDE-based studies. Our study found that the transparency and reproducibility of data extraction from the MAUDE database in MAUDE-based studies is questionable, with only a small percentage of articles and queries being reproducible. Specifically, our study revealed that only 11.5% of 523 identified MAUDE-based articles contained executable queries, and only 23.3% of those executable queries were reproducible. This lack of transparency and reproducibility can negatively impact research, practice, and policy by hindering progress in understanding patient safety issues related to medical devices and making it difficult to use these findings to inform decision making.

Our further analysis results revealed that the single-field query with common search such as product code or brand name, the single device category, and the presence of a data processing flow may enhance the reproducibility of MAUDE data queries. Only 8.3% of the multifield queries were reproducible, compared with 27.1% for the single-field queries. Studies focusing on multiple device categories tended to have more complicated query logics, which required concatenating MDRs retrieved by a larger number of subqueries. Postprocessing procedures like concatenation may affect the query reproducibility in 2 ways: (1) concatenating MDR sets required intensive postquery processing (e.g., deduplication), which was prone to mistakes. (2) the authors were more unlikely to report the number of MDRs both before and after postquery processing. Therefore, it is recommended that MAUDE-based research focus on a single device category. If multiple device categories must be used, it is better to provide the number of MDRs returned by each subquery with a clear description of postquery processing steps.

A large number of authors failed to provide a clear description of queries, which posted barriers to reproduce the results. One hundred thirty-five of the 336 articles (40.2%) exhibited unclear descriptions of queries. Of the 135, the first authors of the 113 (83.7%) had a medical background. One potential reason for unclear query description might be that many authors did not receive formal or sufficient training for data extraction or information retireval. To create a clear query, it is essential to provide query fields, key words in each query field, exact time range for the query, and to report the number of MDRs returned by every single subquery.

Only 14 of the 60 articles (25.0%) with executable queries presented a data flow chart, a standard component of data-driven studies. The data processing flow is indispensable to describe the overall procedure to process the MDRs and the number of MDRs proceeded in each step. One potential reason for studies not providing data processing flows might be due to the lack of expertise of data analysis.

Twenty-three of the 60 articles (38.3%) with executable queries used the simple search function without the selection of specific API fields. The simple search function inclines to yield inconsistent query results. One potential reason for authors to use the simple search function is that they did not fully explore how to use the structured fields in the MAUDE API and the benefits for using these fields. For instance, brand name, manufacturer and product class are the 3 API fields that could be matched with the research topics of 29 articles; however, only 9 articles actually used these structured fields in their queries. Therefore, it is recommended that researchers learn about structured API fields before planning to use simple search. Having an FDA device category at hand also helps decide proper query key words. In addition, a generic name field could be added to the MAUDE API. Generic name is a mandated MDR field where clinicians give phrase-level descriptions of the malfunctioning device. Generic name is characterized by its flexibility: any phrase can be used in this free-text field to describe the reported medical device. This flexibility makes generic name an ideal field to match a large variety of research topics. Nevertheless, generic name cannot be used directly for indexing since there may exist multiple equivalent expressions for the same concept (e.g., ultrasound and ultrasonic, humidification system, and humidifier). An Natural Language Processing-based recommendation system may be needed to retrieve MDRs relevant to the query input via fuzzy matching.

The MAUDE API complicated some of the queries. In situations where the query hit the 500 MDR limitation, the authors had to split the time span used in the query and run extra subqueries for each divided subtime span to obtain the result. The 60 executable queries had 8.63 ENRs on average, while the maximum ENR was 132. A desired query API should be designed to handle queries with a large number of MDRs returned. Another effort made by OpenFDA³⁴³ was a URL-based API³⁴⁴ released in June 2014 for MAUDE queries that allows for queries on a total of 121 MDR fields, far more than those included in the original API (shown in Supplementary S3, http://links.lww.com/JPS/A597), and thus can handle various research needs. This API returns up to 1000 MDRs per query run and provides a “skip” parameter that helps decrease the ENR and improve query reproducibility (see Supplementary S4, http://links.lww.com/JPS/A597). However, it requires users to write a long URL with numerous parameters from scratch and returns JSON structured data as the result, which sets a high threshold for researchers without computer science backgrounds. One future direction is to develop a user-friendly version of such an API.

Scope of This Study and Its Limitations

This study has limitations that must be addressed to improve future studies’ understanding and reproducibility of MAUDE queries. First, the size of the executable queries is insufficient to draw statistically significant conclusions. Second, the reproducibility coefficient measurement only considered the numbers of MDRs returned between original and reproduced queries rather than the contents of the MDRs. Investigating the content of the MDRs seems to be very challenging or even impossible because the detailed contents of MDRs are seldom elaborated in articles. Furthermore, reproducibility is subject to numerous influencing factors, and our study serves as a preliminary exploration of these elements. We assessed reproducibility ratios in relation to query complexity, involved device categories, and the presence of data processing flows. Future research ought to broaden this scope, encompassing MAUDE-based articles from more sources, examining reproducibility more comprehensively, and pinpointing additional factors impacting reproducibility. Such efforts will improve the robustness and relevance of studies driven by MAUDE data.

Availability of data and materials.

Not applicable.