Adapting to change: experiences and recommendations from the Pain Management Collaboratory on modifying statistical analysis plans

Abstract

Background

Best practices for clinical trials stipulate that statistical analysis plans (SAPs) need to be finalized before initiation of any analysis. However, there is limited guidance about when changes to SAPs are acceptable and how these changes should be incorporated into the research plan with appropriate documentation.

Methods

We conducted a survey of 12 pragmatic clinical trials (PCTs) in the Pain Management Collaboratory that evaluated nonpharmacological interventions for pain to assess the following SAP information: (1) location of statistical analysis details, (2) types of statistical analyses planned, (3) sponsor requirements, (4) templates used for development, (5) publication plan, (6) changes since trial launch, (7) process of documenting changes, and (8) process of updating the trial registry.

Results

All 12 PCTs provided details of their SAPs for the primary outcomes in the institutional review board–approved trial protocol; 8 included plans for secondary outcomes, and 6 included plans for tertiary/exploratory outcomes. Most PCTs made SAP changes after trial initiation, many as a result of COVID-19–related issues. Eleven of the PCTs were actively recruiting participants. Changes were made to sample size, study design, study arms, and analytical methods, all before the data lock/unblinding. In all cases, justification for the changes was documented in the trial protocol or SAP, signed off by the trial biostatistician and principal investigator, and reviewed/approved by an institutional review board, data and safety monitoring board, or sponsor.

Conclusions

We recommend that SAP changes can be acceptable up to the time of data lock/unblinding. To maintain full transparency and necessary rigor, clear documentation of such changes should include details, rationale, date(s) such changes were implemented, and evidence of approval by relevant oversight bodies.

Introduction

To ensure research integrity, reduced risk of bias, and optimal reproducibility, a clear, comprehensive, and transparent account of prespecified statistical analyses should be established and made available before implementation of a clinical trial.¹ Trial results can vary substantially by the statistical approach used,² and different conclusions about treatment effects can be reached for the same trial by using different statistical methodology.³ A prespecified statistical analysis plan (SAP) helps ensure research integrity by establishing the analytic approach for a clinical trial before the trial begins and documenting any subsequent modifications throughout the trial’s cycle. This practice also promotes transparency and reproducibility by peer reviewers and other partners so they can understand how the data were analyzed and interpreted.

The US Pain Management Collaboratory (PMC) is a joint clinical research initiative established in 2017 involving the National Institutes of Health (NIH), the Department of Defense (DOD), and the Department of Veterans Affairs (VA). It supports the conduct of several large, pragmatic clinical trials (PCTs) studying the effectiveness and implementation of nonpharmacological approaches for the management of pain and common co-occurring conditions in real-world military and veteran health care systems.⁴

At the time of this project, there were 12 ongoing PCTs in the PMC evaluating patients with acute to chronic pain; study populations ranged from those with more specific conditions (eg, low back pain) to those with generalized pain. Nonpharmacological interventions and health care pathways are compared with alternative strategies, usual care, or self-care. Pain intensity or interference, disability, and opioid use are the primary outcomes assessed across these PCTs; some of the secondary outcomes include quality of life, mental health, and health care utilization. These PCTs comprise a variety of study designs, ranging from individual to cluster-randomized, from parallel to stepped-wedge design, and from single randomization to a sequential multiple-assignment randomized trial (SMART).

The COVID-19 pandemic caused unprecedented disruption not only to ongoing patient care, but also to the conduct of clinical trials, which, in turn, impacted their SAPs. Disruptions to PCTs in the PMC included delays in participant enrollment, interruptions in intervention delivery and assessments, and modifications to intervention protocols. As a result, SAPs needed to be updated accordingly to allow for adaptation in the context of unforeseen challenges in real-world settings. However, limited guidance is available about when changes to SAPs are acceptable, what changes should be documented, or how to document such changes.

The purpose of this article is to describe survey results about the SAPs from the 12 PMC PCTs. The survey queried about the SAP development process, specifics of data analysis plans, and changes to the SAP during the trial planning and implementation phases, including how changes were documented. On the basis of survey results, we provide recommendations about modifications to SAPs.

Methods

At the annual in-person Steering Committee meeting held in May 2023, PMC investigators, biostatisticians, funders, leadership, and external content experts discussed best practices in developing and making changes to SAPs as one of several topics on best practices for the conduct of pragmatic trials. The meeting included a plenary and multi-day breakout sessions on the topic of SAPs. The breakout discussion focused on the need to better characterize the processes for developing and changing SAPs for the PCTs. Plenary session leaders (Q.Y., C.R.L., P.P.) used detailed minutes of all sessions to develop an outline of relevant questions that assessed how and when SAPs might be modified. After the meeting, the preliminary questions were distributed to the PMC Biostatistics and Design Work Group for input. The Work Group includes biostatisticians from each of the PMC trials, representatives from funding agencies, 2 biostatistician co-chairs, a project manager, and a co–principal investigator from the PMC Coordinating Center. Members meet monthly to develop methodological standards for the design and implementation of pragmatic clinical pain trials. Writing team members were solicited from attendees at the breakout session and the Work Group. After several rounds of edits by the writing team, a survey assessing how individual studies made adjustments to SAPs was finalized.

Each PCT was asked to complete the SAP survey (Table 1). The questions were answered by the lead biostatisticians from each PCT, with input from the principal investigator(s). The survey consisted of 14 questions covering 8 areas related to the SAP: (1) location of statistical analysis details, (2) types of statistical analyses planned for the primary outcome, (3) sponsor requirements, (4) guidance or templates used for development, (5) SAP publication plan, (6) changes since trial launch, (7) process of documenting changes, and (8) process of updating the trial registry. Writing group members reviewed the survey result, identified shared practices and gaps, and compiled general recommendations over several follow-up sessions.

Table 1.

Pain Management Collaboratory survey on managing the statistical analysis plan.

1	Name
2	What study trial do you belong to?
3	Where do you provide details of your trial’s statistical analysis (check all that apply):	☐ Protocol ☐ Stand-along Statistical Analysis Plan (SAP) ☐ Design paper ☐ Other (Specify)
4	What analyses are included in your analysis plan (check all that apply)?	☐ Primary Outcomes ☐ Secondary Outcomes ☐ Tertiary/Exploratory Outcomes
5	Did your trial sponsor have any requirements for your SAP (contents, level of detail, etc.)?	☐ Yes ☐ No
6	Did you follow a guidance or use a template for creating your SAP?	☐ Yes ☐ No
7	Are you planning on publishing your SAP (eg, open-source journal)?	☐ Yes ☐ No
8	Have you made any changes to your SAP or statistical section in the protocol since launching your study?	☐ Yes ☐ No
9	Who reviews and approves changes to the SAP (check all that apply)?	☐ IRB ☐ DSMB ☐ Sponsor ☐ Other
10	Who signs off on changes?	☐ Biostatistician ☐ PI ☐ Sponsor ☐ Other
11	How do you document changes to the SAP?	☐ Tracked changes ☐ Version control table ☐ Other
12	Do you document the reason/rationale for the change?	☐ Yes ☐ No
13	How do you maintain consistency with ClinicalTrials.gov?
14	Is there anything else you would like to share?

Abbreviations: DSMB = data and safety monitoring board; IRB = institutional review board; PI = principal investigator; SAP = statistical analysis plan.

Results

All trials

All 12 PCTs provided details of their SAPs in the institutional review board (IRB)–approved trial protocol: 12 for their primary outcomes; 8 for their secondary outcomes; and 6 for their tertiary and exploratory outcomes. Among these, 4 of 12 reported that they also have a standalone SAP. Because one PCT had not yet launched their trial, only 11 responded to the remainder of the survey questions.

Active trials

General guidance for content and reporting of SAPs has been established⁵; however, sponsors have their own requirements. There were 3 sponsors for these PCTs: NIH, DOD, and VA. Five of the 11 PCTs indicated that the trial sponsor had requirements such as provision of detailed descriptions of sample size calculations and statistical modeling, adjustment for multiple comparisons of secondary outcomes, and provision of more detail on approaches to handling missing data. None of the trials used a template or followed any formal guidance in writing their SAP. Initially, 2 teams indicated that they used an SAP template or guidance; however, upon further review, it was determined that both teams were using a protocol template with a statistical analysis section, rather than an SAP template. Trial protocol papers including some level of the SAP were published by 11 of the PCT teams^6–16 before or early in trial implementation. Only 2 plan to publish their SAP as a standalone document.

Seven out of 11 PCTs reported that they made changes to their SAP or statistical section of the protocol after starting the trial. Changes were made to sample size (n = 3), study design (n = 1), study arms (n = 2), and analytic methods (n = 4), and one study added a formal analysis of COVID-19 impact (n = 1). All 7 PCTs that reported making changes had those changes approved and signed off by their biostatistician. Some also reported having changes reviewed and approved by the IRB, data and safety monitoring board, or sponsor.

Many of the changes were in response to the COVID-19 pandemic. For example, one PCT encountered a 6-month suspension of enrollment during the COVID-19 pandemic, and the few study participants enrolled before the pandemic were subsequently excluded from the primary statistical analysis because of the impact of COVID-19 restrictions. Several PCTs changed the plans for sensitivity analyses to assess the impact of COVID-19 on the effect of the intervention or subgroup analysis, while others revised both their study timelines (to allow for longer recruitment periods) and their target sample sizes (by choosing to forego some secondary analyses that had been originally planned). Because of the challenges imposed by guidance on social distancing/quarantine and the COVID-19–related inaccessibility of in-person visits, some PCTs switched the intervention delivery from in person to virtual. Other examples of changes include sample size revisions (eg, variability in the intracluster correlation coefficient necessitated revising the required number of clusters) or interim power projections, as well as intention-to-treat vs per-protocol analytic approaches. Another PCT changed the primary outcome from total Brief Pain Inventory to the interference subscale of the Brief Pain Inventory.

Changes to the SAP were documented via tracked changes (n = 7/11) or with version control tables (n = 4/11). Other methods of documentation included naming of the documents with the version number and storage of all versions of the trial protocol, which included the statistical analysis section. Most PCTs (n = 9/11) documented the reason/rationale for the changes captured in the revised protocol, the SAP, supportive commentary, or a cumulative summary of changes. Various strategies were used to maintain consistency with ClinicalTrials.gov and to communicate the changes to the SAPs, including manual reviews, monitoring/reviewing the entry with the IRB-approved protocols, and posting the updated SAP on a preprint server.

Discussion

Pragmatic trials often encounter unforeseen challenges or disruptions in the real-world setting that require modifying the SAP. Best practices stipulate that SAPs should be specified and finalized before any data analyses are conducted.^5,17,18 Nevertheless, there is limited guidance about when changes to SAPs are acceptable, what changes need be documented, or how to document such changes. A recent publication provides guidance for developing and reporting an SAP for a clinical trial through the use of a checklist and provides guidance that an SAP should be written or amended before the data lock and unblinding of outcomes, and we agree.⁵ However, it does not provide guidance on review/approval process of SAP changes.

Regulatory agencies such as the US Food and Drug Administration (FDA) and European Medicines Agency (EMA) often require that details of the SAP be included as part of the clinical trial protocol.^18–20 These agencies provide some recommendations for SAPs. For example, the FDA encourages sponsors to include the SAP as part of the protocol, rather than providing it in a separate document, even if the SAP has not been finalized. The FDA considers the SAP to be a key document for decision-making; primary and secondary analyses and outcomes must be prespecified, with all other analyses considered post hoc.¹⁸

According to the guidance provided in the International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH) E9. 75, the SAP may be written as a separate document to be completed after the trial protocol has been finalized.¹⁹ In a standalone SAP, a more technical and detailed elaboration of the principal features stated in the trial protocol may be included (see Section 7.1¹⁷). The plan may include detailed procedures for executing the statistical analysis of the primary and secondary outcome variables and other data. The plan should be finalized before the data are unblinded. Formal and signed-off records should be kept when the SAP is finalized, as well as when the database is locked and the data are unblinded.

The majority of PCTs made changes to their SAPs after trial initiation, many of which were related to issues arising from the COVID-19 pandemic. Changes were made to sample size, study design, study arms, and analytic methods. Justification for the changes made to the PCT SAPs was documented in the trial protocol or SAP. All changes were signed off by the biostatistician and principal investigator and reviewed and approved by the IRB, data and safety monitoring board, or sponsor. The PCTs were in the planning or active stage at the time of the survey, so these changes were made before the database was locked and before unblinded outcomes were available.

Ensuring transparency when the SAP is updated during the conduct of a clinical trial is critical for maintaining the integrity of the research. Any major changes to the SAP should be documented with details about what was changed and when the change was made. Justification of the changes should also be included to explain why the change is necessary and how it affects or improves the data analysis and interpretation of the results. Publishing the SAP changes on a preprint server is a good way to communicate the change, particularly when it has been described in a published protocol.

Major changes typically include modifications to sample size, study design, randomization, and primary or secondary endpoints, as well as changes in the analytic methods for primary analysis. Reasons for sample size modifications in the PMC PCTs included issues in recruitment or follow-up assessments, as well as operational challenges resulting from COVID-19 disruptions. Modifications to study design included changes to study arms, mode of intervention delivery, or trial design. For example, some PCTs switched from in-person to virtual intervention delivery because of COVID-19 restrictions. Furthermore, changes in the analytic methods for primary analyses included altering the modeling approach or hypothesis testing. One of the PCTs had such changes because of increased participant withdrawals and missing data, and accordingly, they updated their SAP.

Trials need to ensure that major changes comply with ethics guidelines and regulatory requirements. Furthermore, all major changes have been or will need to be updated on the ClinicalTrials.gov registry to maintain public transparency. Trial alterations such as changes in masking status, addition/removal of study sites, or changes in the inclusion/exclusion criteria need to be documented and justified in the trial protocol but might not be required in the SAP.

Changing the SAP during the conduct of the trial might be necessary and beneficial under certain circumstances, such as difficulties in enrollment, changes in the standard of care, or health emergencies like the COVID-19 pandemic. Modifying the SAP can help mitigate these issues and preserve trial integrity. Another scenario is when breakthrough statistical methods or data analysis techniques become available; adopting such new approaches can improve the validity and reliability of results. A good example is the integration of innovative machine learning or artificial intelligence techniques in areas such as outcome prediction, adaptive trial design, and natural language processing.

Conclusion

Our survey results show that changes to pragmatic trial SAPs are common and can be appropriately implemented and justified. These modifications are typically driven by unforeseen circumstances, emerging data, or evolving scientific knowledge and can be pivotal to the success and integrity of the trial. We recommend detailed documentation of changes to SAPs as part of the trial protocol, in a standalone document that would include information about what was changed and when the change was made. At minimum, a brief summary of the changes should be included in the publication reporting the primary results. Alternatively, these changes can be housed on preprint servers, in the trial registry, or as an addendum to the published protocol. Communication of the changes to the SAP allows results to be interpreted within the appropriate context of these changes. The documentation should also include the rationale for each modification, its impact on the trial’s outcomes and analysis, and approval by relevant oversight bodies. Our recommendations are shown in Table 2.

Table 2.

Summary recommendations for SAPs and their modifications.

Recommendation 1: Adhere to best practices that stipulate that a prespecified SAP needs to be written before any data analyses are conducted. Details of the SAP need to be included as part of the clinical trial protocol. A separate standalone SAP may be written, providing a more technical and detailed elaboration of the principal features stated in the trial protocol.

Recommendation 2: Allow for changes to SAPs up to the time of data lock/unblinding, and comply with ethics guidelines and regulatory requirements.

Recommendation 3: Provide detailed documentation on SAP changes and, at minimum, include: Detailed information on changes and their chronology. A clear and detailed rationale for each modification (typically, sample size, study design, randomization, and analytic plan) and its impact on the trial’s outcomes and analysis. Documentation of approval by relevant oversight bodies (eg, IRBs, DSMBs, sponsors).

Recommendation 4: Update the trial’s ClinicalTrials.gov registry entry to reflect the changes and to maintain public transparency.

Abbreviations: DSMB = data and safety monitoring board; IRB = institutional review board; SAP = statistical analysis plan.

Funding

Research reported in this publication was supported by the National Center for Complementary and Integrative Health (NCCIH) and the Office of Behavioral and Social Sciences Research (OBSSR) of the National Institutes of Health (NIH) under Award Number U24AT009769.

Research reported in this publication was supported by the National Institutes of Health under Award Numbers UG3/UH3 AT009790 (Hastings/George AIM-back trial), UG3/UH3 AT009763 (Fritz/Rhon SMART-LBP), UH3/UG3 AT012257 (Lovejoy/Morasco CORPs trial), and UG3/UH3AT009761 (Long/Goertz VERDICT trial).

Conflicts of interest: None declared.

Supplement statement

This article appears as part of the supplement titled “Pain Management Collaboratory: Updates, Lessons Learned, and Future Directions.”

This article is a product of the Pain Management Collaboratory. For more information about the Collaboratory, visit https://painmanagementcollaboratory.org/.