Harnessing a Health Information Exchange to Identify Surgical Device Adverse Events for Urogynecologic Mesh

Abstract

We sought to create an automated means to conduct surveillance of complications related to urogynecologic mesh because current postmarket surveillance fails to detect the true incidence of device-related adverse events. Using health information exchange data, we developed a search algorithm to identify urogynecologic surgeries with mesh implantation and associated inpatient adverse events. We validated the algorithm search results against those obtained from a manual case review of mesh surgical records. Our refined automated search strategy matched 93% of the 2874 mesh cases manually identified, and further identified 97% of 2103 vaginal mesh cases. Complications were identified in 380 of the 2874 mesh cases. This is the first known report of an automated process for identifying urogynecologic surgical mesh implantation cases from a health information exchange. Automated surveillance of health information exchange data may contribute to tracking of device-related adverse events.

Introduction

Adverse events caused by FDA-approved surgically-implanted medical devices can cause serious injury to patients. Recent FDA warnings describe serious injury and death caused by implanted urogynecologic mesh, a medical device used to treat pelvic organ prolapse (POP) and stress urinary incontinence (SUI). An estimated 283,000 women received mesh implants for these indications in 2010.¹ Adverse events are not unique to this type of medical device. Postmarket device failures and recalls affect many medical specialties, such as cardiology and orthopedics, with serious consequences.² Lack of data regarding expected failure rates of given medical devices can compromise the informed consent process with patients.

Robust methods for postmarket surveillance of surgical devices are not widely available. Not only do voluntary reporting methods underestimate the incidence of device failures and device-related adverse reactions, they cannot describe the prevalence of the problem.³ Device-specific registries can be effective, but they may suffer from poor data quality if they are populated solely through voluntary reporting. Longitudinal data in a health information exchange may provide a means to follow adverse events related to postmarket medical devices.

Pelvic organ prolapse (POP) is caused by a weakening of the muscles and ligaments that support the pelvic organs, leading to protrusion of the uterus or vaginal walls into or out of the vagina. Stress urinary incontinence (SUI) is the involuntary loss of urine typically caused by a cough, sneeze, or similar increase in intra-abdominal pressure. These conditions are often repaired surgically, but traditional surgical techniques have been associated with high recurrence of POP.⁴

The FDA considers urogynecologic surgical mesh a medical device. Mesh is an elastic web made from absorbable or non-absorbable synthetic material or absorbable biologic material. It has been used in urogynecologic surgery for POP or SUI to enhance the strength of a patient’s own fascia, thus reducing the risk of recurrent prolapse or incontinence. Biologic meshes and absorbable synthetic meshes are less likely to erode through surrounding tissue, but because they are absorbed by the body over time, they may be more prone to surgical failure with recurrent POP or SUI. Most FDA-approved gynecologic mesh kits utilize monofilament non-absorbable mesh, often synthetic polypropylene.⁵

In July 2011 the FDA issued a Safety Communication Update on serious complications associated with transvaginal placement of surgical mesh for POP. This update was triggered by 3,979 reports of complications, including death, received by the FDA since 2005 regarding surgical mesh devices used to treat POP or SUI. The most frequent complications reported included mesh erosion through the vagina (also called exposure, extrusion, or protrusion), pain, infection, bleeding, pain during sexual intercourse, organ perforation, and urinary problems. There were also reports of recurrent prolapse, neuromuscular problems, vaginal scarring or shrinkage, and emotional problems.⁶ The FDA Update was followed in January 2012 by an FDA requirement for postmarket surveillance studies by manufacturers of urogynecologic surgical mesh devices. The FDA ordered 33 manufacturers of mesh for POP and seven manufacturers of single-incision mini-slings for SUI to conduct studies of complications.⁷

With urogynecologic surgical mesh as a use case for surgically-implanted medical devices, we sought to determine if data contained in a large health information exchange could be used to track the use of urogynecologic mesh and the incidence of complications associated with it. This study had the following objectives:

• Objective One: Using data in the health information exchange, determine how well an automated search strategy identified 1) all surgeries performed to treat POP or SUI involving implantation of urogynecologic mesh, and 2) the subset of those surgeries with transvaginal mesh placement.

• Objective Two: Determine which of the surgeries involved complications during the index admission or readmission within the six-year study period.

• Objective Three: Identify the type of urogynecologic mesh device used in procedures for SUI or POP involving transvaginal mesh placement.

Background

Automated surveillance systems for infection control and pharmacoepidemiology study have been established for many years. Automated postmarket surveillance of regulated medical products using observational data contained in electronic health records developed more recently, and is an international priority⁸. Two examples of developing automated postmarket surveillance systems in the United States include the Vaccine Safety Datalink run by the Center for Disease Control and the Sentinel System of the US Food and Drug Administration (FDA).

Several considerations are important in developing an automated medical product surveillance system. Ideally, the system should identify known associations between a product and a specific health outcome of interest (HOI), such as a known adverse event for a given surgical procedure. In addition, the system should also detect unexpected adverse events. Most current surveillance systems examine targeted monitoring of specific pairs of products and HOIs, usually looking for 5–10 adverse events for a given product exposure ⁹. Following this model, we sought to identify common known serious adverse events, including those that happen as a result of the surgical procedure: hemorrhage, fistulas, bowel injuries, bladder injuries, ureteral injuries, and wound infections. We also searched for delayed complications of surgery, including mesh erosions, mesh exposures, and mesh infections.

Algorithms to identify both the product (in this case, urogynecologic mesh) and the HOI vary depending on the available data within the database. The FDA’s Mini-Sentinel project primarily used administrative and claims data. The database for our study, the Indiana Network for Patient Care, lacks claims data for most participating hospitals, so we focused on administrative codes (ICD-9 CM and CPT), as well as search terms from operative notes and discharge summaries. Additionally, in a study conducted by the Mini-Sentinel Core project, attempts had been made to search medical literature to identify previously validated identification algorithms for HOIs, and the study found that locating these algorithms was difficult¹⁰. We also could find no previously validated search algorithms for urogynecologic mesh and its known complications in another database similar to the Indiana Network for Patient Care.

Other factors that influence the design of a surveillance system include the exposure characteristics of the medical product (whether the patient’s exposure to the product is transient, sustained, or both), and the timing of onset of the HOI (whether abrupt or insidious). Consideration of whether the patient can act as her own control (self-controlled design) or whether a cohort design (between-person comparison) is preferred depends on both the exposure characteristics of the medical product and the timing of the HOI ¹¹. In April, 2011, the FDA published draft guidance on best practices for reporting pharmacoepidemiologic safety studies to reduce bias using data from electronic health records,^12,13 which can guide these decisions.

The potential for false positive and false negative adverse event identification in medical product monitoring is substantial, due to the size of the electronic health databases, the large numbers of exposures and outcomes to be monitored, and the possibility of identifying the same adverse event on more than one occasion when sequential monitoring for events is done over time ¹⁴. Identification of adverse events needs to be sensitive, specific, and timely. Calibration of a surveillance system involves trade-offs between identification of cases (recall) and false positive alerts (precision). This was demonstrated by Murff and others, who compared natural language processing (NLP) techniques against administrative claims data on a manually-validated dataset of inpatient records from the Veterans Affairs Surgical Quality Improvement Program with the goal of identifying specific post-operative complications. NLP techniques were associated with higher sensitivity, but lower specificity than the use of administrative data, and all of the search strategies required a balance of the two¹⁵.

An automated monitoring system must be capable of generating alerts when adverse events are detected beyond a specific threshold so appropriate response action can be taken. Gagne and others¹⁶ evaluated five classes of alerting algorithms in a simulated matched-cohort framework to replicate monitoring of cervistatin-induced rhabdomyolysis and found that no single algorithm performed best in all scenarios. He concluded that alerting algorithms for adverse events or HOIs should be tailored to the expected event frequencies and the trade-offs between false-positive and false-negative alerting.

Methods

The steps to create and test our search algorithm first involved the creation of a set of true positive cases by manually reviewing a large number of dictated operative notes which contained the word “mesh.” Our inclusion criterion was any surgery performed on a female patient for the treatment of SUI or POP where urogynecologic mesh of any type was implanted. The manually-identified dataset of surgeries became our gold standard for comparing the results of our automated search strategies.

We began with a broad search of medical records within the Indiana Network for Patient Care (INPC), a 16-year-old health information exchange operated in Indiana which includes clinical data from more than 20 hospital systems and contains more than 4 billion clinical observations and more than 79 million text reports. We limited the search to five hospitals in two healthcare systems where mesh was used from 2005 to 2011. We first performed a search of operative notes and discharge summaries for the word “mesh” or any of 17 urogynecologic mesh device names (e.g., “Prolift”). The specific search tokens used are listed in Figure 1, Search Strategy 1. This approach yielded 9838 records (8236 operative notes and 1602 discharge summaries). We deliberately pulled a large number of records in order to reduce the likelihood of missing appropriate cases. We decided not to use discharge summaries to identify the surgical cases, and instead worked with the operative notes alone. Discharge summaries were later used to identify surgical complications.

Figure One.

Search Strategies. This figure summarizes the serial searches we conducted to maximize the recall and precision of our search algorithm. Search Strategy 2 is our preferred method.

We then manually reviewed the 8236 operative reports to determine which cases were surgeries for POP, SUI, or both, where urogynecologic mesh had been implanted. We found 2874 cases, identified as Dataset 3 (DS3), which became our first gold standard for comparison. We extracted from this group the reports that included only vaginal urogynecologic mesh surgeries for POP, SUI or both, resulting in 2103 cases, which became our second gold standard for comparison, Dataset 1 (DS1). For each of these cases, we recorded the available details about the specific urogynecologic mesh surgical device used (e.g., manufacturer or product name). We gathered this subset of vaginal-approach cases because the postmarket studies ordered by the FDA in January 2012 specifically addressed transvaginal mesh placement medical devices. We also separately documented “redo” reparative operations and readmissions for complications of any mesh surgeries to see if our automated search could identify the complications.

We then conducted an automated search of the 8236 operative reports using refined search tokens to collect all POP or SUI surgical cases where mesh was used (Figure 1, Search Strategy 2). This approach excluded 4792 cases, leaving 3444 cases for primary analysis, which became Dataset 2 (DS2). For comparison we ran a supplemental search of data from the five hospitals in the two healthcare systems looking for only administrative codes (e.g., ICD-9-CM and CPT codes) used for coding urogynecologic surgeries (Figure 1, Search Strategy 3). This resulted in 53,692 cases, referenced as Dataset 4 (DS4). The results of these two automated searches were then compared to the first and second gold standards to determine recall. Because the precision for the vaginal cases was low, we further refined our automated search to exclude non-vaginal urogynecologic mesh surgeries (e.g., abdominal sacral colpopexy) (Figure 1, Search Strategy 4). This resulted in 1766 cases, which we identified as Dataset 5 (DS5).

Results

The demographics of the population that underwent vaginal surgery for POP or SUI are listed in Table 1. The patients ranged in age from 23 to 95 years, with a median age of 59 years. The number of surgical cases per year increased steadily from 165 cases in 2005 to 449 cases in 2009.

Table 1.

Demographics of participants

Characteristic	N (%)
Age (years)
20–29	19 (0.9)
30–39	142 (6.6)
40–49	461 (22)
50–59	598 (28)
60–69	554 (26)
70–79	305 (14)
80–89	51 (2.4)
90–99	7 (0.3)
Race
White	1983 (93)
Black	89 (4.2)
Other	38 (1.8)
Unknown	29 (1.4)
Year of procedure
2005	165 (7.7)
2006	300 (14)
2007	381 (18)
2008	396 (19)
2009	449 (21)
2010	448 (21)
Hospital
1	116 (5.4)
2	809 (38)
3	607 (28)
4	401 (19)
5	206 (9.6)

Comparison of the results of the automated refined search DS2 (3444 cases) against the manually validated vaginal approach DS1 (2103 cases) showed 2045 cases common to both sets, for a recall of 97% (2045/2103). We then manually reviewed a random sample of the remaining 1399 DS2 that did not match (41%) and discovered that they were not vaginal SUI or POP surgeries where mesh was implanted. The precision was 62%.

Comparison of refined search DS2 (3444 cases) against the DS3 manually-validated data set for all urogynecologic surgeries where mesh was implanted (2874 cases) resulted in 2660 matched cases, for a recall of 93% (2660/2874). The precision was 84%.

Comparison of the results of the DS4 ICD-9/CPT code sets (53,692 cases) against DS1 (2103 cases) found 1974 cases (94%). Comparison of DS4 against DS3 (2874 cases) resulted in 88% recall. However, because of the large number of cases pulled by this search strategy (most of which likely did not involve urogynecologic mesh), it was infeasible to use as a surveillance technique.

Our fourth search strategy, the results of DS5 (1766 cases) compared against DS1 (2103 cases), was an attempt to improve the precision for detecting vaginal approach mesh surgeries. This strategy successfully improved the precision of 62% from Search Strategy 2 to 74%; however, the recall dropped to 63% (1321/2103).

Results of all comparisons are represented in Table 2.

Table 2.

Search strategy results

Search Strategy	Recall (%)	Precision (%)
Strategy 2 (Vaginal Cases)	97	62
Strategy 2 (All Cases)	93	84
Strategy 3 (Vaginal Cases)	96	Not calculated
Strategy 3 (All Cases)	88	Not calculated
Strategy 4 (Vaginal Cases)	63	74

Strategy 2: Automated Refined Search Tokens

Strategy 3: Administrative Codes

Strategy 4: Automated Further Refined Search to Identify Vaginal Approach Cases

The manually-reviewed cases for vaginal approach surgery were examined to determine if the specific urogynecologic mesh device had been identified in the operative note. The specific surgical device was identified in 86% of all cases performed to treat POP, including combined cases, and 28% of cases performed to treat SUI, including combined cases. In the combined cases (cases where both POP and SUI were treated), 25% identified the mesh type used for both procedures. In addition, transcription errors were detected on manual review (Examples were that “Perigee” was transcribed as “Para-G” and “Lynx” was transcribed as “Links”). Results are listed in Table 3.

Table 3.

Mesh type identification within vaginal approach cases

Case Type	Number of cases	Number (%) of cases with mesh
All Cases SUI	1598	442 (28)
All Cases POP	827	715 (86)
All Cases SUI & POP	322	79 (25)

POP = Pelvic organ Prolapse SUI = Stress Urinary Incontinence

Surgical complications were identified in 380 of the 2874 cases (13%) of all urogynecologic mesh surgeries; 130 of these were vaginal SUI, 103 were vaginal POP, 51 were a combination of the two, and 96 were a result of complications arising from abdominal or laparoscopic surgery for SUI or POP. The most common complication was mesh erosion, found in 286 cases (74%) -- 82 vaginal SUI, 82 vaginal POP, 35 vaginal combined procedures, and 87 abdominal or laparoscopic procedures (Table 4).

Table 4.

Mesh complications occurring intra-operatively, post-operatively or requiring subsequent surgery

Complication	SUI	POP	SUI/POP	ABD or L/S	ABD or L/S with SUI	Total
Mesh erosion	72	73	30	74	8	257
Mesh contraction	2	3	1	1	1	8
Mesh erosion with organ injury	8	6	1	2	0	17
Mesh erosion with infection	0	0	0	1	0	1
Mesh erosion with urinary retention	0	0	3	0	0	3
Hematoma	0	3	3	0	0	6
Hemorrhage/anemia with transfusion	2	3	0	0	0	5
Organ injury (without erosion)	3	4	3	3	1	14
Deep venous thrombosis (post-op)	0	0	2	0	0	2
Urinary retention (with sling revision or removal)	34	3	6	0	0	43
Pain (causing mesh revision)	5	8	0	1	0	14
Infected mesh	4	0	0	4	0	8
Medical admissions*	0	0	2	0	0	2
Any complication	130	103	51	86	10	380

^*Medical Admissions for chest pain and pain control

POP = Pelvic Organ Prolapse SUI = Stress Urinary Incontinence ABD = Abdominal L/S = Laparoscopic

Discussion

This study demonstrated for the first time that a health information exchange can be used to identify cases where urogynecologic surgical mesh was used for POP or SUI. Current postmarket surveillance systems can fail to detect significant device defects before large patient populations are exposed. Published studies examining computer-based surveillance of adverse events related to medical devices have demonstrated that methods using diagnostic discharge codes, patients’ survey data, and computer-based flagging within electronic health record systems were inadequate at identifying these events.¹⁷ More recent published reports, however, have recognized success with automated surveillance using device registry data.¹⁸ Registries do not exist for all implanted medical devices, and many registries rely upon voluntary reporting. An informatics approach using longitudinal information contained in large health information exchanges may facilitate automated surveillance and creation of registries and may greatly improve this process. Although we did not implement or evaluate longitudinal tracking, the method described herein could be used for such purposes to build and maintain a database of the use of urogynecologic mesh.

Our process required identification of surgical cases where mesh was implanted, determination of the surgical approach (vaginal, abdominal, or laparoscopic), identification of associated complications, and identification of the surgical device implanted. Urogynecologic mesh is not a discrete data element within our database. In addition, urogynecologic mesh is used in several gynecologic procedures often identified by different names (e.g., “posterior repair,” “posterior colporrhaphy,” and “rectocele repair” refer to the same procedure). We used several text mining strategies to maximize the results of automated search methods. The use of our refined Search Strategy 2 worked best for identification of urogynecologic surgeries to treat POP or SUI, but was ineffective at segregating those with an abdominal or laparoscopic approach from those with a vaginal approach. Administrative codes (e.g., ICD-9 CM and CPT codes) lacked precision for adequate automated detection. Our case identification using the refined search strategy resulted in good recall, but a relatively low precision. Manual review, or an alternative approach, would be required to eliminate cases that were not the desired result of our search.

This study was designed to be a preliminary step in assessing the data available in a large health information exchange to determine if an automated surveillance method could be developed to detect complications from urogynecologic mesh. Future steps in this development process include the incorporation of NLP techniques specifically to identify adverse events from operative notes, admission history and physical examination reports, and discharge summaries. The output of these NLP techniques can be compared to adverse events identified by manual review. A practical value added by using the more basic text-based search approach described in this study is that this type of search strategy may be used by a hospital system that does not necessarily have the expertise to use more sophisticated NLP techniques.

To develop an effective surveillance system, surgical complications and adverse events resulting from the use of urogynecologic mesh must be identified. We identified 380 re-admissions for complications of urogynecologic mesh through manual review. Our refined search method using keyword searches identified 96%of these complications, while a search of administrative codes identified only 91%. The refined search methods can effectively identify repeat surgeries or readmissions following the index surgery. Some patients, however, had subsequent admissions due to the management of a new urogynecologic problem (e.g., new SUI independent of effective repair of POP) rather than for the treatment of a complication. Unless we can further refine our analysis of these readmissions, manual review may be necessary to accurately determine whether a patient’s subsequent hospitalization reflects a complication rather than treatment of a new urogynecologic problem. In addition, our dataset includes many patients who were referred into the hospital networks in our study for the treatment of complications that occurred from surgery at other facilities. We lack the ability to capture the index surgery in these instances.

Among all possible types of adverse events, only serious adverse events were targeted by this study. These were events requiring prolonged hospital stays, readmissions, or subsequent surgical procedures. Identification of less severe adverse events was not feasible due to a lack of outpatient records within the health information exchange. Only one of the hospital systems included in the study contributed outpatient records to the health information exchange, where less severe adverse events are more readily identified. Serious adverse events were primarily identified in operative notes (90%), followed by discharge summaries (9%), and outpatient clinic notes (1%). Our project suggests operative notes identified patients who required surgical correction of complications related to urogynecologic mesh, particularly mesh erosions. Discharge summaries identified complications related to the surgical procedure such as post-operative hemorrhage and urinary retention.

Identification of the specific urogynecologic surgical device used in the procedure was problematic. Lot numbers or other identifying data are not captured as structured or coded data within the health information exchange. Some implanted mesh devices may carry a higher risk of post-operative complication than others, making device identification a significant concern. Text search for device identification was ineffective because dictating physicians did not consistently identify the device used within their operative notes, particularly in procedures for the treatment of SUI. A unique code in the medical record specifying the implanted medical device would greatly improve the ability not only to identify the device used, but to locate the surgical cases where mesh was implanted. A unique device identification code could likely be tracked in the medical record with greater recall and precision than achieved by our refined search term method.

Legislation for the creation of Unique Device Identification (UDI) Codes was signed into law as part of the FDA Amendments Act of 2007. The FDA ran a proof-of-concept test in 2009 to determine what data would make UDIs useful and meaningful, but has not yet issued a final regulation. In the Final Report on Unique Identification of Surgical Devices prepared for the FDA,¹⁹ several potential advantages of UDI numbers were described, including simplification of medical device recalls, reduction of medical errors, unintended device interaction problems, curbing distribution of counterfeit medical devices, and use of UDI numbers for tracking and surveillance. These potential benefits can best be realized if UDI numbers are implemented extensively and consistently and if electronic health record systems capture these data.

In 2009 the FDA launched its Sentinel Initiative to supplement the current and mostly passive system for monitoring postmarket adverse reactions with active surveillance from electronic data sources that can be analyzed to identify safety or efficacy issues with new drugs or devices on the market. This network has linked claims data of more than 100 million health records. Unfortunately, medical device-specific data are often not included in this data source. This is expected to improve over time as more data sources are added to the Sentinel System.²⁰

Limitations

As noted in the methods section, outpatient reports were not included in identifying adverse events, due to variation in availability of such reports through the health information exchange. We did not have the resources to review the daily inpatient progress notes. Advanced approaches to NLP were not used in this study, but they could enhance the performance of the search strategy. We did not create a fully automated system capable of performing ongoing surveillance for serious adverse events related to the use of implanted urogynecologic mesh.

Conclusion

Robust postmarket surveillance techniques for surgical devices are lacking. Patients may suffer serious and tragic adverse events from FDA-approved devices that malfunction. Voluntary reporting systems under-report device failures and complications and lack information on the prevalence of the problem. We demonstrated that we can detect surgical cases where urogynecologic mesh was implanted for the treatment of POP with a recall of 93% and a precision of 77%. Adverse events were identified in 13.2% of cases. We are able to accurately identify the type of mesh used in 86% of cases of vaginal POP repair and 28% of cases of vaginal SUI repair. Further refinements in our search criteria, as well as the use of natural language processing techniques, may lead to improved results.

Registries can serve as a surveillance data source, but need to be populated with accurate, complete, and detailed clinical data. Health information exchanges such as the INPC contain this type of data and could serve as a means for surveillance of surgical device adverse events or as a data source for longitudinal population of medical device registries. This process would be simplified and data quality would be improved if unique device identification codes were part of electronic health records.