Design of a Study to Measure Patient-Perspectives in Adverse Event Reporting (the PPAR Study): Supplementary Study to the ASPIRe Trial

Abstract

Objectives:

The primary objective of this study is to compare patient versus physician rankings of adverse event (AE) and adverse symptom (AS) severity after pelvic reconstructive surgery. Secondary objectives include to estimate the association between patient rankings of AEs/ASs with decision-making and quality-of-life outcomes and to determine whether patient perspective about AE/AS changes over time.

Methods:

This is a supplementary study, Patient-Perspectives in Adverse Event Reporting (PPAR), to the index trial, ASPIRe (Apical Suspension Repair for Vault Prolapse In a Three-Arm Randomized Trial Design). During the trial, AEs/ASs will be assessed by physicians longitudinally every 6 months, which includes a determination of the AE/AS grade severity. For PPAR, additional patient perspective will be measured for 19 predetermined AEs/ASs at the time of identification and again at 12 and 36 months post-operatively. Decision-making and quality-of-life questionnaires will be collected at these time points. The primary outcome, the overall interrater agreement between patient and physician rankings for AE/AS severity, will be determined using a repeated-measures concordance correlation coefficient.

Results:

To date, the index trial has completed enrollment, and follow-up is ongoing.

Conclusions:

The PPAR methods for incorporating patient perspective in the measurement of AEs/ASs to determine their agreement with physician ranking, long-term relevance, and impact on treatment decision making and quality of life are described. This will contribute to improved measurements of AEs/ASs in future research with the goal of improving patient counseling and informing expectations and treatment decision making.

The reporting of complications and adverse outcomes after medical treatment is an important component in the assessment and comparison of new and existing interventions. Although many trials use robust methodology to detect potential treatment benefits, there has been less focus on the assessment of adverse outcomes and complications and their long-term relevance and association with quality of life. There are several challenges to adverse outcome reporting and measurement, including a lack of understanding of the patient perspective. There is more recent emphasis that knowledge about adverse outcomes from the patient perspective is a key component required to capture all important harms that could influence patient decision making.^1–3

The overarching aim of this supplementary study, the Patient-Perspectives in Adverse Event Reporting (PPAR) study, is to improve our understanding and capture of the patient perspective regarding adverse surgical outcomes, their associated harms, and their effect on patient decision-making outcomes for female pelvic floor disorders (PFDs). Both patient and clinician perspectives are meaningful, and an improved understanding will inform the development of a shared model for adverse outcome reporting in PFD treatments. The aims include comparing patient versus physician rankings of complication grade (severity), to estimate the association between patient rankings of adverse outcomes with decision-making and quality-of-life outcomes, and to determine if patient perspective about adverse outcomes changes over time after pelvic reconstructive surgery.

MATERIALS AND METHODS

Study Design Overview

Patient-Perspectives in Adverse Event Reporting is a supplementary study to the Apical Suspension Repair for Vault Prolapse In a Three-Arm Randomized Trial (ASPIRe trial) conducted by the Eunice Kennedy Shriver National Institute of Child Health and Human Development–sponsored Pelvic Floor Disorders Network (PFDN). ASPIRe is a multisite, prospective, 3-armed randomized trial comparing apical transvaginal mesh, sacral colpopexy, and native tissue vaginal repair of posthysterectomy vaginal vault prolapse.⁴ Patient-Perspectives in Adverse Event Reporting is designed to compare patient and physician rankings of adverse events (AEs) and symptoms (ASs) and determine which are associated with a negative impact on decision-making and quality-of-life outcomes. In other words, PPAR will provide information to inform which adverse outcomes are relevant and result in harm from the patient perspective.

Figure 1 provides an overview of the PPAR study design. The primary aim of PPAR is to compare patient rankings versus physician rankings of AE/AS grade. Secondary aims are to assess the impact of AEs/ASs on long-term treatment decision regret, decision satisfaction, quality of life, and patient global impression of outcomes and to determine if patient perspective about AE/AS changes over time.

FIGURE 1. Study design overview.

*Abbreviations:

PPAR - Current study name

ASPIRe - Parent study name

AE - Adverse event

AS - Adverse symptom

†Decision-making and quality of life outcome measures:

The Decision Regret Scale (DRS) and Satisfaction with Decision Scale (SDS)

12-Item Short Form Health Survey (SF-12) and Patient Global Impression of Improvement (PGI-I)

Study Population

All women who participate in ASPIRe will be included in this supplementary study, with the same inclusion and exclusion criteria (see ASPIRe methods paper). Only randomized and treated participants (N = 360) will be included in the analysis.

Description of Study Interventions

Both physicians and patients will provide information and perspectives about AEs/ASs that occur in the study period. The physicians will be masked to the patient reporting and the patients will be masked to the physician reporting.

In PPAR, adverse outcomes are categorized as AE or AS for measurement and descriptive purposes. Adverse events can be described as discrete events that may not have ongoing or long-term symptoms once they are resolved (eg, visit to the emergency room, port site hernia). Adverse symptoms can be described as undesired symptoms experienced by a patient that may be ongoing and/or may not have a clear etiology (eg, vaginal discharge, constipation).

Physician Rankings

Physician ranking of AEs/ASs follows a commonly used process in PFDN trials that meets several regulatory guidelines required for adverse outcome reporting. The initial AE/AS reporting will be captured by research staff either by telephone or in-person follow-up visits. Collection includes both active capture (patients are queried about a specific list of AEs/ASs) and passive capture (relies on patient reporting to staff that an AE/AS occurred). These are logged on an ASPIRe Physician AE Log Form, for which there are 7 fields to be completed by research staff and the local sites’ physicians/investigators (Table 1). Research staff gather data for the “Action taken,” “Outcome,” and “Continuing” fields of the log form, and physician/investigators determine “Grade,” “Attribution,” “Expected,” and “Serious” fields. Completed reports are batched and reviewed centrally by an AE Adjudication Committee, which consists of 3 investigators in the PFDN to help standardize rankings.

TABLE 1.

ASPIRe Physician AE Log Form

AE Descriptor	Response Options (Determined by Investigators)
Grade	Mild, moderate, severe, life-threatening, death
Attribution: relation to study Intervention	Unrelated, unlikely, possible, probable, definite
Action	None, sought medical attention, additional surgery, hospitalization, other, unknown
Outcome	Resolved, resolved with sequelae, unresolved, fatal, unknown
Expected AE?	Yes or no
Serious AE?	Yes or no
Continuing?	Yes or no

The grading of events is guided by the Common Terminology Criteria for Adverse Events⁵ (CTCAE) published by the National Cancer Institute of the National Institutes of Health in an effort to provide standardization and consistency in reporting treatment-related toxicity in cancer trials. Based on the CTCAE, AEs are graded as mild (grade 1), moderate (grade 2), severe (grade 3), or life-threatening (grade 4), with specific parameters according to the specific organ system. Death (grade 5) is used to denote a fatality that occurred during treatment. Approximately 10% of items in the CTCAE represent symptoms, and 90% represent events: both symptoms and events are graded by clinicians and not by patients. This is used as a guide for ASPIRe AE physician grading and thus represents the physician perspective.

ASPIRe AE capture will follow guidelines described here:

• 0–6 weeks postoperatively: All AE/AS grade 2 or higher are captured.

• 6 weeks to 6 months: All AEs/ASs and severe AEs involving death, hospitalization, or ER visit and grade 2 or higher, which are at least possibly related to surgery in the opinion of the investigator(s) are captured.

• After 6 months: Deaths, AEs/ASs, and severe AEs grade 2 or higher that are at least possibly related to pelvis or surgery in the opinion of the investigator(s) are captured.

Patient Rankings

For PPAR, patient input will be obtained on 19 preselected AEs/ASs through active capture (study staff will directly question patients about PPAR AEs/ASs). There are 8 AEs and 11 ASs to be captured in PPAR (Tables 2 and 3). These were selected based on relevance and review of the Pelvic Floor Complications Scale (PFCS),⁶ the Patient-Reported Outcomes Version of CTCAE (PRO-CTCAE) library,³ previous work through the PFDN,⁷ and consensus among the surgeons and investigators in the PFDN.

TABLE 2.

PPAR Inventory of Adverse Symptoms^*

Adverse Symptoms	Description
Vaginal discharge	New or worsening vaginal discharge
De novo vaginal bleeding	New-onset or persistent vaginal bleeding beyond 6 wk
Pelvic pain	Any pain associated with worsening bother compared to preoperative occurring in the lower abdomen or genital area beyond 12 wk postoperatively (excluding neuromuscular pain and dyspareunia)
Dyspareunia	Any new-onset pain associated with sexual activity that was not present during sexual activity preoperatively (AE = anything other than never)
Stress urinary incontinence	New or worsening stress urinary incontinence
Urgency incontinence	New or worsening urgency urinary incontinence
Fecal incontinence	New or worsening fecal incontinence
Constipation (new or worsening)	New-onset or worsening condition in which medical treatment is required
Difficulty emptying bladder (new or worsening)	The new-onset or worsening inability to completely empty the bladder during urination
Symptomatic mesh exposure	Vaginal bleeding, problems with sexual relations, or other symptoms reported by patient due to vaginal mesh exposure
Neurovascular event related to surgery	Numbness/weakness

^*Refer to Table 4 for data collected about each reported AS.

TABLE 3.

PPAR Inventory of Adverse Events^*

Adverse Events	Description
Vaginal infection	Infection of the vagina determined by a physician using clinical or radiological indicators to be uncommon to vagina and requiring treatment
Lower urinary tract infection (UTI)	UTI based on clinical judgment or confirmation of a culture proven by laboratory criteria also includes empiric antibiotic treatment for symptoms thought to be secondary to UTI
Other infection* (specify)	Infection diagnosed using clinical or radiological indicators not including vaginal infection, lower UTI, pelvic infection/abscess, or infection/inflammation of bone
Laparoscopic/robotic port site hernia
Emergency department visit for intervention related complication	Patient was evaluated in an emergency department for a problem related to the surgery
Intraoperative or perioperative event that changed management	Cautery burn, corneal abrasion, retained foreign body, ureteral or bladder injury requiring additional surgery, conversion to open surgery, ileus, bowel obstruction, need for blood transfusion
Return to operating room for intervention related complication	Patient returned to the operating room for a problem related to the surgery
Event requiring additional treatment for intervention related complication	Trigger point, etc

^*Refer to Table 4 for data collected about each reported AE.

When a PPAR AE/AS is identified, in addition to completing the physician ranking process described above, the research staff will also capture patient-perspective attributes based directly on patient input (see Table 4 for PPAR Patient AE Log Form). The PPAR Patient AE Log Form was developed using the PRO-CTCAE³ as a guide. The PRO-CTCAE is an item library of AS that was developed to measure patient-reported outcomes for symptomatic toxicity for patients in cancer clinical trials. It is intended to be used as a companion to the CTCAE. Attributes measured for AS in the PRO-CTCAE include frequency, severity, and/or interference with usual or daily activities.

TABLE 4.

PPAR Patient AE Log Form

AE Descriptor	Patient Question*	Response Options
l. AEs
Severity at its worse (grade) Impact/interference	How would you grade or rank this event? How much did the event/symptom interfere with your usual or daily activities?	Mild, moderate, severe, life-threatening Not at all/a little bit/somewhat/quite a bit/very much
2. ASs
Frequency	How often did you experience the symptom	Rarely, occasionally, frequently, almost constantly
Severity at worst (grade)	What was the severity level of the symptom at its worst?	Mild, moderate, severe, very severe
Impact/interference	How much did the event/symptom interfere with your usual or daily activities?	Not at all/a little bit/somewhat/quite a bit/very much

In PPAR, for discrete AEs, patients will be asked to grade the severity and rate the impact/interference of the event. For AS, patients will be asked to grade severity, frequency, and impact/interference of the symptom. These will be recorded on the PPAR Patient AE Log Form, which represents “Patient Ranking.” Patients will be asked to provide information based on their opinions and experiences and will not be guided by the research team. The attributes will be captured in an ongoing fashion such that the highest severity reported by the participant will be recorded. Her perspective will be reassessed at 12 months and 3 years after surgery, and she will be asked to provide information about her perspective in real time, which will allow assessment of whether her perspective changed over time. To minimize bias, patients will complete decision-making and quality-of-life outcome scales (see below) prior to completing the PPAR Patient AE Log Form with research staff.

Measures of AE/AS Impact and Harm

Although short-term effects of adverse outcomes are important, PPAR is focused on assessing the long-lasting consequences of events, specifically those that have long-term impact on quality of life and/or may cause regret or dissatisfaction with a patient’s decision to have surgery. The ASPIRe trial provides an opportunity to gather these long-term data.

The Decision Regret Scale⁸ (DRS) and Satisfaction with Decision Scale⁹ (SDS) are patient-centered decision-making outcome scales that have demonstrated good psychometric properties in various patient populations. They have been previously adapted and shown to have good psychometric properties in women undergoing surgical treatment for PFDs (DRS-PFD and SDS-PFD).¹⁰ The DRS-PFD is a 5-item questionnaire with a 5-point Likert response scale, and higher scores indicate a higher degree of regret with the treatment decision. The SDS-PFD is a 6-item questionnaire with a 5-point Likert response scale, and higher scores indicate a higher degree of treatment decision satisfaction. Distress from complications has been shown to be associated with significantly higher regret scores. In a study evaluating recall of surgical consent for midurethral slings, the number of complications was independently associated with higher level of decision regret.¹¹ The DRS-PFD and SDS-PFD will be collected when a PPAR AE/AS is identified and at 12, 24, and 36 months after surgery per the ASPIRe protocol.

The 12-item Short Form Health Survey (SF-12)¹² and Patient Global Impression of Improvement (PGI-I)¹³ will also be collected. The SF-12 is a multipurpose short-form survey that provides a measure of overall health-related quality of life. It is a generic measure and includes physical and mental health domains. The SF-12 will be collected at baseline, 6 and 12 months after surgery, and then every 12 months throughout the patient’s participation in ASPIRe. The PGI-I will be collected at the same postoperative time points and provides a measure of global improvement from the patient’s perspective. All questionnaires are paper based.

Statistical Design

Sample Size/Power Calculations

Adverse event rates were estimated using data from previous PFDN trials evaluating apical suspension procedures.^14,15 Additionally, at the time of protocol development, approximately 300 adverse outcomes had been reported in an ongoing apical trial of 183 women comparing native tissue apical suspension versus transvaginal mesh hysteropexy, with slightly more than 60% having a relationship of at least “possibly related” to the surgical intervention (unpublished data). Based on this information, it was estimated that at least 400 PPAR adverse outcomes would be identified among the 360 ASPIRe participants. Assuming that the distribution of AE/AS severity is comparable to previous apical surgical trials, and if there was no correlation between different events for the same patient, the ASPIRe study sample size (N = 363) will provide a 95% confidence interval width for estimating the agreement between patients’ and physicians’ AE/AS severity grading of +/−0.11. That is, if the study estimated κ coefficient is 0.5, there will be 95% confidence that the true agreement is in the range of 0.39 to 0.61. Further, a tighter confidence interval width is expected by taking into account the correlation between events using the repeated-measures extension of the κ coefficient.

Data Analysis

The primary outcome, the interrater agreement between Patient Rankings and Physician Rankings for AE/AS grade (severity), will be determined by estimating the repeated-measures concordant correlation coefficient, which is an extension of the κ coefficient for repeated assessments.¹⁶ In addition to this test, which provides an overall assessment of whether the ratings between physicians and patients agree or disagree, extensions of generalized linear models will be used for the ordinal nature of these outcomes and the paired structure of the patient/clinician ratings to evaluate how patients and clinicians disagree, should the initial test suggest differences. Specifically, classes of events for which the rankings of patients are consistently lower or higher than those of the clinicians will be explored, and a descriptive κ coefficient will be calculated for each of the 18 PPAR AE/AS terms.

The short- and long-term impact and relevance of AEs/ASs and their effect on decision making and quality of life will be measured using DRS-PFD, SDS-PFD, SF-12, and PGI-I scores. The association of patient grade and impact of AEs/ASs with decision-making and quality-of-life scores will be estimated using linear model approaches and graphical summaries. If appropriate, mean scores for these scales will be compared across women who exhibit different AE/AS grades; if general linear methods are not appropriate, then we will use appropriate methods to analyze these outcomes. These scores at the initial AE/AS identification and 12-month and 3-year time points will be compared.

To determine if patient perspective about AE/AS changes over time, the intraobserver agreement will be estimated for patient rankings of AE/AS grade and impact over time using κ coefficients. The impact of specific adverse outcomes over time will be explored, taking into account the patient’s perception of if and when the AEs/ASs resolved.

Based on PPAR, a ranked list of AEs/ASs will be developed and compared with the previously published PFCS.⁵ Adverse events ranked by severity based on physician rankings will serve as 1 ranking list. A similar ranking list will be developed based on patient rankings, and the lists will be compared qualitatively to explore differences and similarities between these 2 ranking lists.

DISCUSSION

Adverse event, AS, and subsequent harms assessments in clinical trials are critical to allow patients and health care providers to weigh the potential benefits, harms, and invasiveness of different treatment options. There are at least 2 important reasons why good-quality data about events and complications after surgical treatment are needed. First, a comprehensive understanding of how and which complications can affect quality-of-life and decision-making outcomes is important for patient treatment decision making and to allow balanced counseling of options between a health care provider and the patient. Second, purchasers and payers have an increasing interest in surgical complications and the quality of surgical care, as complications can be very costly.¹⁷ Despite these strong reasons, the measurement and monitoring of complications, particularly from the patient perspective, is often imprecise and of uncertain validity.¹⁸ Weak measures and inconsistent methodology ultimately result in weak data.

In a study by Gutman et al,⁶ surgeons from 2 clinical trials networks rated complications to develop a PFCS, which were rated based on surgeon perceptions about perceived patient bother, severity, and duration of disability. Using this ranking, PFCS scores were calculated for 977 subjects in 2 surgical trials, and it was found that the higher PFCS scores were associated with longer hospitalization, lower satisfaction, health utility scores, and quality of life. Extending this work, Fitzgerald et al¹⁹ evaluated 33 women (16 preoperative and 17 postoperative) through telephone interviews and focus groups and identified that patients considered new recurrent urinary tract infections, new persistent constipation, worsening postoperative constipation, blood transfusion, readmission, and reoperation to be severe complications. Common patient concerns also included surgical failure, anesthesia, mesh erosion, discharge with catheter, and pain.

More recently, Dunivan et al⁷ conducted a qualitative study through the PFDN to describe patient perceptions on adverse outcomes following PFD surgery. This study included 81 women attending 12 focus groups and included patients who were preoperative, short-term postoperative, and long-term postoperative. Patients were asked to rank patient-identified and surgeon-identified adverse outcomes in order of perceived severity. The study found that women considered functional outcomes including incontinence, sexual dysfunction, and recurrence of symptoms as severe. Patient-Perspectives in Adverse Event Reporting will build on this work. These functional outcomes will be captured in PPAR and in the ASPIRe protocol, and we will have the opportunity to assess their long-term effects on decision making and quality of life.

A limitation of PPAR will be the active capture approach of preselected AEs/ASs, because it is possible there may be AEs/ASs that patients consider important that will not be included.²⁰ However, PPAR is an early step, and for this supplementary study, active capture was considered to be the most feasible first-step approach to collecting data in a reliable and consistent manner that will result in meaningful data.

A primary strength of PPAR is that potential harm will be assessed in several ways including (1) measurement of impact/interference of the AEs/ASs, (2) association with decision-making outcomes, and (3) association with quality-of-life and global impression outcomes. Another advantage of capturing patient perspective at the time of AE/AS reporting is that we can capture her experience in real time, instead of relying on retrospective recall or hypothetical scenarios. Although other studies have focused on complications in the immediate perioperative period, this study will provide insight into the patient’s experience of AEs/ASs after discharge and what is experienced more remote from the time of the index surgery. Patient-Perspectives in Adverse Event Reporting will also combine the complementary perspectives of clinicians and patients.^21,22 Clinician reporting of AEs is highly associated with clinical endpoints of death and hospitalization in cancer trials, as their impressions are based on professional training and experience. Patient reporting is more highly associated with measures of day-to-day health status and better reflects the short- and/or long-term impact of AEs/ASs. Therefore, these are complementary perspectives both captured in PPAR to support a shared reporting model in the future. Which events are associated with harm (short-term and/or long-term impact on decision making and quality of life) will be determined. Finally, the results of PPAR will help guide and streamline future AE/AS reporting in clinical trials of PFDs.

In order to help patients make informed treatment decisions, comprehensive information about treatment efficacy, options, quality of life, and adverse outcomes is needed. This supplementary study is designed to improve the way we capture data about adverse treatment outcomes by incorporating the patient perspective. Ultimately, we anticipate this study will help to present a more balanced representation of patient experience and provide more robust data about AEs and symptoms associated with treatments for PFDs.