The Snapshot Audit Methodology – Design, Implementation and Analysis of Prospective Observational Cohort Studies in Surgery

Abstract

Purpose:

For some surgical conditions and scientific questions, the “real world” effectiveness of surgical patient care may be better explored using a multi-institutional time-bound observational cohort assessment approach (termed a “snapshot audit”) than by retrospective review of administrative datasets or by prospective randomized control trials. We discuss when this might be the case, and present the key features of developing, deploying, and assessing snapshot audit outcomes data.

Methods:

A narrative review of snapshot audit methodology was generated using the Scale for the Assessment of Narrative Review Articles (SANRA) guideline. Manuscripts were selected from domains including: audit design and deployment, statistical analysis, surgical therapy and technique, surgical outcomes, diagnostic testing, critical care management, concomitant non-surgical disease, implementation science, and guideline compliance.

Results:

Snapshot audits all conform to a similar structure: being time-bound, non-interventional, and multi-institutional. A successful diverse steering committee will leverage expertise that includes clinical care and data science, coupled with librarian services. Pre-published protocols (with specified aims and analyses) greatly helps site recruitment. Mentored trainee involvement at collaborating sites should be encouraged through manuscript contributorship. Current funding principally flows from medical professional organizations.

Conclusion:

The snapshot audit approach to assessing current care provides insights into care delivery, outcomes, and guideline compliance while generating testable hypotheses.

Introduction

Improving surgical outcomes requires reliable evidence discovery that informs care. Such data flows both from retrospective studies that generate testable hypotheses as well as prospective inquiries that test those hypotheses. Information from both sources guide future trial design and revises current care approaches. Similar to other disciplines, surgical science prioritizes multicenter inquiries over single-center ones for all assessments other than local quality or performance improvement projects. Neither the retrospective review, nor the prospective multi-institutional trial assess current care, however. Instead, usual care for a specific disease process may be captured using a prospective time-limited multi-institutional observational cohort approach. These studies are termed “snapshot audits” and have helped assess a variety of surgical conditions.[1–5] Importantly, this approach does not limit data to those that fit interventional trial inclusion criteria but instead includes every patient receiving therapy for the condition of interest during the short assessment period. Since snapshot audits are less common that other study designs, its structure, specific elements, benefits and pitfalls have been less robustly characterized. Therefore, this narrative review explores the snapshot audit method to describe strengths and weaknesses, identify enablers and barriers, explore team composition, and highlight its overlap with implementation science.

Methods

We developed a Scale for the Assessment of Narrative Review Articles (SANRA) guideline conforming narrative review to provide historical context and practical guidance regarding prospective time-bound multi-institutional observational cohort studies.[6] To that end, the Thomson Reuters Web of Science, OVID, and PubMed databases were queried for English language manuscripts regarding snapshot audit studies in surgery (inception through February 1, 2022, using keywords such as “prospective observational cohort” OR “snapshot audit” OR “implementation science” AND “surgery”). Focused domains included surgical therapy and technique, diagnostic testing, critical care management, interaction with concomitant non-surgical disease (such as SARS-CoV-2), trial design, statistical analysis, as well as implementation science approaches to guideline compliance. Selected manuscripts detailed snapshot audit related primary data, narrative reviews, systematic reviews, and meta-analyses [Supplemental Table 1].

EVOLUTION OF SCIENTIFIC REPORTING IN SURGERY

Surgical techniques, patient care approaches, and outcome data reporting continue to increase in complexity. Early publications consisted of case reports that shared a clinical innovation, observation, or discovery that was practice-altering. They illustrated expert advice regarding specific conditions and treatments. As care became more complex, so too did reporting, moving to case series of similar patients while often presenting optimal outcomes. Epidemiologists and biostatisticians developed tools to evaluate disease patterns and etiology, and established public health reporting as a unique discipline. At the same time, early 20th century clinician-scientists laid the ethical and methodological foundations required to conduct prospective research and randomized controlled trials (RCTs) that assess outcomes after a specified intervention.[7,8]

Virtues and Limitations of RCTs in Surgery

Recognizing that retrospective reviews of large administrative datasets are at inherent risk of problems, including finding potentially spurious associations in large datasets, prospective RCTs seek to circumvent these issues by standardizing groups, controls, interventions, and analyses “under ideal conditions”. RCTs employ randomization to minimize bias in treatment allocation, and they often use blinding. Unlike pharmaceutical trials where double-blinding is possible, the surgeon cannot be blinded to the operation. While RCTs may address a common clinical concern (i.e., perforated diverticulitis), enrollment criteria fit only a specific portion of all patients who present with the condition of interest. Using specific inclusion criteria improves group homogeneity and clarifies intervention impact, but limits generalizability. Therefore, RCT data may not apply to “usual” patients. Because large numbers of patients are often required to meet power analysis requirements – especially for low-frequency events - weighted randomization schemes have arisen to reduce the required number of enrollees. Variable adherence to study protocol – a common source of bias in prospective trials – further degrades trial data validity. Academic centers are overrepresented as RCT sites, perhaps providing care that is not standard or not available across all care settings including community care sites. Furthermore, RCTs often deliberately exclude those at the extremes of age or those currently pregnant, and inadvertently exclude those with cultures or languages that create participation barriers.

Relatedly, surgical intervention is binary within RCTs (e.g. there was or there was not an operation) and trial design rarely affords no adjustment for operator experience along a learning curve. Instead, outcomes from a seasoned surgeon are often identically assessed to those of a surgeon who recently completed training. Moreover, in low-to-middle income countries (LMIC), factors other than surgical technique or patient selection may account for adverse outcomes, despite RCT entry criteria fulfillment. For example, access to certain therapeutics may be limited (e.g. certain antibiotics, CT scanning, component transfusion), and 24/7 electrical power may be uncertain, adversely impacting operative and post-operative care. Nonetheless, the Hawthorne effect - short-term improvement that occurs as a result of explicit observation - has been raised as one explanation of better patient outcomes during RCT post-intervention follow-up. Despite their value in testing interventions, Acute Care Surgery (ACS) patient enrollment in RCTs is less than desired. Therefore, RCTs provide neither an environmental scan of who is receiving care within a facility, nor what care is being provided, and consequently provide a quite limited view of current care.

SNAPSHOT AUDIT – A ‘REAL WORLD’ PROSPECTIVE OBSERVATIONAL STUDY

Snapshot audits, which are non-interventional and catalog usual care, complement both retrospective studies and prospective RCTs. Snapshot audit origins are rooted in the Lothian Audit, a prospective primarily vascular surgery outcomes database created in early 1990’s Scotland. Indeed, the Lothian Audit was hailed as a trailblazing initiative within the context of limited computing power.[9–13] While the development of comprehensive clinical datasets improved the granularity and appropriateness of source data to evaluate disease-specific care, they still suffer the limitations of retrospective analyses, questioning whether derived conclusions are sufficiently robust to influence current care.[14–17] In 2015, to rapidly assess current care, the European Society of Coloproctology (ESCP) and others pioneered the “snapshot audit” as a novel adaptation of the traditional prospective collaborative cohort study to assess outcomes following sigmoid colectomy for colorectal cancer. This approach utilizes pre-publication of protocols and data elements (in protocol registries such as ClinicalTrials.gov), as well as planned group and sub-group analyses. As well as providing an environmental map of demographic data and outcomes for this common procedure, their data also provided testable hypotheses (such as the impact of pre-operative mechanical bowel preparation) for subsequent interventional studies. Such questions may include i) quality of care issues that can be assessed in between-hospital comparisons, ii) medical research questions concerning interventions or prognosis, and iii) clinico-epidemiologic questions, such as complications rates, patient characteristics, stage distribution, amongst others..A clearly defined protocol that is straightforward to follow encourages sites to participate, and therefore facilitates dynamic collaboration across an often-international defacto research network. [Figure 1]

Figure 1 -.

Snapshot Audit. Steps in the Contemplative, Study and Analysis/Reporting phases of a prospective multicenter observational cohort study

The ESCP promoted brief high-volume prospective snapshot audits across nearly 50 countries while recruiting over 10,000 patients.[18] Audits of right hemicolectomy (2015), stoma closure (2016), as well as left colon, sigmoid and rectal resection (2017) were completed. [19–21] The 2019 Management of Acute Severe Ulcerative Colitis Audit (MASC) recently closed data recruiting. Seeking to evaluate the contemporary management of complicated biliary calculous disease and acute appendicitis, the European Society for Trauma and Emergency Surgery (ESTES), adopted an identical approach.[4,22,23] Most recently, the SARS-CoV-2 pandemic catapulted the snapshot audit onto the world stage, with the massive pan-national COVIDSurg snapshot audit. This particular study established a new record in collaborative scientific inquiry, recruiting 1,677 centers in 112 countries caring for 142,815 patients (to date) while providing patient-level clinical care and outcome data.[3,24–28] Collected data are designed to be substantially more detailed and targeted than that of administrative clinical care databases (e.g. National Trauma Data Bank of the ACS or the German Trauma Bank). Indeed, once the snapshot audit ‘machinery’ is in place, the study could be redeployed at regular intervals to assess changes in clinical practice over time, or to assess the impact of evidence-based guideline implementation.[29,30] Snapshot audits capture unvarnished differences in epidemiology and clinical practice across contributing centers without impacting institutional care paradigms, pathways or protocols. Therefore, the spectrum of patients - and the care they receive - may be characterized, outcomes assessed, hypotheses generated, and key information identified to may inform interventional trial design. [Figure 2]

Figure 2 -.

Major recent snapshot audits, ongoing studies, and the potential for future work

LESSONS LEARNED IN DEPLOYING AND SHARING SNAPSHOT AUDITS

Crafting the question

One practical consideration when embarking on a snapshot audit is to ensure that the data collected answers the intended clinical question. While such guidance seems intuitively obvious, it is much more nuanced. Collecting extraneous data adds work, clutters data fields, and requires more storage, especially if data is stored using a cloud-based program. Conversely, not having sufficient information to answer the primary and clearly related questions devalues the work. For example, capturing data regarding acute perforated appendicitis managed using a percutaneous drain during index hospitalization to assess the time to interval appendectomy seems straightforward. Failing to capture interval hospitalizations prevents one from assessing drain management, additional antibiotics, or admission for unrelated comorbid or newly acquired conditions (e.g. C. difficile disease) and therefore limits dataset utility and examinable questions. For example, an error we made in creating the data dictionary for the acute biliary calculous disease snapshot audit was not anticipating the need to acquire readmission data between index admission and interval cholecystectomy. Therefore, we were unable to comment on failures or complications of initial non-operative management.[4,22] Snapshot audit leaders must ensure that the primary question – and related questions – all align with captured data points as the questions are immutable after trial launch. Therefore, the use of a pre-defined data collection instrument is a strength and a weakness of both RCT and snapshot audit inquiries. We strongly suggest employing a modified Delphi approach that systematically solicits expert opinions to build the questions to answer as well as the underpinning data dictionary[31,32]. An ideal group to construct the data dictionary is the audit’s steering group. (Figure 1) As it may be quite difficult to know which variables (in addition to age, gender, comorbidities, or socioeconomic status) are possible or likely confounders, literature searches are often necessary for a full and systematic understanding of which variables influence the main outcomes of interest.

Team composition

Each snapshot audit team fields a small steering group as well as a larger implementation group – the latter representing all of the collaborating centers. Steering group members should have clinical expertise in the condition of interest. However, planned recruitment of members with statistical, library science, and data science expertise into the steering group is invaluable. Large database management, data assessment, and presentation all represent unique skill sets that are uncommonly possessed by bedside clinicians. The diverse international authorship for this manuscript, reflective of a snapshot audit steering group, brings together expertise in emergency surgery, anesthesia and perioperative medicine, surgical outcomes research, biostatistics, study methodology and implementation science. Legislation such as the European Union’s General Data Protection Regulation (GDPR) and the Health Insurance Portability and Accountability Act (HIPAA) in the United States regulate the sharing and utilization of sensitive patient data. Moreover, intentional GDPR- and HIPAA-compliant database construction (as opposed to using an Excel spreadsheet, for example) facilitates high-quality data collection and analysis. Indeed, the ease with which a collaborating center can access and submit data to the database may strongly influence initial and sustained participation. The more broadly distributed are the collaborating centers, the more representative will be the data as an “environmental scan” of current patients and practice. Thus, a multinational group is preferable to a geographically constrained alliance of centers when investigating the global spectrum of usual care.

Data management

Assuring data fidelity and security are paramount to the success and integrity of the snapshot audit. There are several database management solutions available to snapshot auditors, of which REDCap and SMARTTrial are the most commonly deployed. REDCap, which originated at Vanderbilt University (Nashville, Tennessee, United States), is a freely-available secure web application for building and managing online surveys and databases (REDCap (projectredcap.org). It is specifically geared to support online and offline data capture for research applications. REDCap must be housed institutionally and generates maintenance costs associated with data storage and staff time; REDCap can integrate with some electronic health records[33]. SMARTTrial (SMART-TRIAL ApS, Aalborg, Denmark) is a commercial database that is purpose-built for each unique study. Both database solutions, designed with clinical research in mind, recognize that multi-center data entry comes at a risk of incomplete or erroneous data entry – a hazard in common with RCTs and large clinical databases. Thus, both have implemented built-in data-entry quality rules at point of record creation (i.e. not allowing submission when key fields are empty, or confining inputs to logical results) and post-hoc data-cleaning which can be deployed during interim analysis. As a behavioral nudge towards improved data fidelity and completeness, the protocols of many snapshot audits have tied 95% data completion to inclusion in publication contributorship.[2,4,34] Nonetheless, ICU data sharing can occur in a compliant fashion, as has been recently demonstrated.[35] Local Institutional Review Board concerns around data security and data de-identification are often the first hurdle auditors must clear prior to launching a snapshot audit.[35] Accordingly, obtaining pilot-site IRB approval prior to launching the public call for collaborating centers is ideal; this approach has been successfully used by the GlobalSURG consortium and others.[36,37] The approved IRB protocol may then serve as a template for the collaborating centers.

Once data have been accrued and analyzed, insights may be shared at medical professional society meetings as well as in peer-reviewed publications. Publication under single entity authorship (e.g., COVIDSurg) where a supplement or appendix lists every study collaborator is viable when collaborators are numerous. Examples include the aforementioned snapshot audits, but also large RCTs (e.g. ARDSNet) and clinical guidelines (e.g. Surviving Sepsis Campaign).[3,38,39] Criticism remains regarding whether collaborators who appear as authors fulfill all four of the International Council of Medical Journal Editors’ criteria for authorship.[40] As an alternative, and to precisely represent each individual’s contribution, one may instead deploy the Contributor Roles Taxonomy (CReDIT) approach.[41] Developed in 2014, this checklist defines fourteen distinct roles in which collaborators may contribute ranging from conceptualization to funding to investigation to writing (initial draft) or revising and editing (subsequent draft). While some large snapshot audits (e.g. COVIDSurg) have informally migrated in successive publications towards a tiered reporting of contributor roles (e.g., steering group membership, writing group, local principal investigators, and local data collators), we propose that contributorship, using the CReDIT taxonomy, be uniformly used in published works to precisely denote project involvement when there are large numbers of collaborators (as opposed to authors). Regardless of how authorship – or contributorship – is represented, the snapshot audit model affords opportunities for mentored trainee involvement and engagement in clinical inquiry. Furthermore, representation within a scholarly work sponsored by a medical professional society may encourage study participation as well as subsequent organization membership and service.

Statistical considerations

Two goals of snapshot database analysis are to determine the frequency of events, as well as how different aspects of care impact those events. When consecutive patients are prospectively recruited and followed, incidence rate ratios (IRR) may be calculated for binary outcomes such as survival or death. While Odds Ratios (OR) may also be used, the IRR have been recommended instead of OR, which are more vulnerable to effect size overestimation. [42]

Funding

Financial support for snapshot audits may come from a variety of sources. First, a team of individuals with access to a database platform may undertake a study without external funding. Success, however, may benefit from being nominally tethered to a medical professional organization or society, at least in terms of study promotion and participating center recruitment. This is the most team-intensive method of conducting a snapshot audit and runs the risk of fatigue and resource limitation, especially regarding securing expert data and database management. Second, securing funding from a medical professional organization couples name recognition with resource support and management.[2,3,24–28,46–49] Third, industry-based support may underpin audits that target device use and their impact on care. Fourth, philanthropic foundations or similar entities may provide financial as well as non-financial support including promotion across social media platforms, data analysis or visualization experts, data storage, or unique equipment. Fifth, governmental agencies may fund such studies, although this is presently uncommon. Instead, those agencies are more likely to fund trials informed by evidence derived from snapshot audit inquiries.

Study Timeline

In general, the snapshot audit assessment period is time-bound and quite short compared to that for RCTs. Moreover, the typical snapshot audit timeline from conception to completion spans nearly three quarters of a year - also much shorter than a RCT. (Figure 3) There are four distinct phases: design, promotion, accrual, and completion, with the shortest often being accrual. Scheduled steering committee activity occurs throughout the study period; additional impromptu meetings are the rule rather than the exception. Collaborating site activity is less frequent by comparison during data accrual than during promotion of completion. Dissemination of results is not included in the timeline as it spans a quite variable time-period following study completion. It is important to note that the study duration are dependent on the incidence and prevalence of the condition to be studied, and the estimated event rate for the outcomes of interest. Thus, a study of drug-induced pancreatitis, or of complications related to anastomotic failure following bowel resection in Crohn’s disease, may require a greater spread of centers and a longer follow-up period to capture meaningful observational data than a study looking at outcomes following treatment of a more common condition such as appendicitis.

Figure 3 -.

This sample project management Gantt chart illustrates a possible timeline for the conception, design, promulgation, accrual and analysis of a snapshot audit. The example presumes a 30-day window for patient enrolment, with 30 days follow-up for each patient. We highlight recommended points along the study course where there is need and utility in a steering group meeting; frequently in the Design phase to identify a clear knowledge gap, to create study questions to address this gap by modified eDelphi consensus, and to create a data dictionary that captures the patient-level information required to answer those questions. There is practical value in assessing progress early in the patient accrual window to allow for repair of any unforseen protocol issues. The study steering group should then reconvene after data extraction, clean-up and analysis to perform an environmental scan of those data and to plan the preparation of manuscripts mapped to each of the pre-defined study questions. The study group should then meet for a final time at study end to identify opportunities and plan for future work.

Guidelines and Implementation Science

Incremental outcome improvements for common surgical conditions are achievable by standardizing key aspects of patient care through clinical practice guidelines.[50–54] In complex ACS conditions, small cumulative relative risk reductions may be attached to early diagnosis, risk stratification, appropriate resuscitation, and directed antimicrobial therapy as well as prompt surgical, endoscopic, or percutaneous intervention.[50,55] Over recent years, concerted efforts to aggregate these separately identifiable gains into meaningful outcomes improvements have occurred by deploying evidence-based guidelines addressing the management of ACS conditions.[56–59] While guidelines do not supplant clinical experience, they do help reduce variations in care when guideline elements are adopted into practice. Unfortunately, guidelines appear to exert less robust than desired effects on physician behavior. Numerous studies document how infrequently clinicians routinely adhere to evidence-based guidelines.[51,52,60–64] Several barriers to guideline adherence exist, including awareness of new or existing guidelines, responsivity to new (or newly synthesized) knowledge, willingness to adapt current practice, external inhibiting factors (such as equipment access limitations in low-income or austere environments), outcome expectancy, and the inertia of prior practice.[60] Nonetheless, some guideline-based elements, often encoded into care bundles, have been utilized as measures of care quality – an approach that generates substantial controversy, especially around sepsis care. [65–67]

Implementation science, the field concerned with narrowing evidence-to-practice gaps,[68]offers approaches to “diagnose” and “treat” failures of healthcare system and individual caregiver adherence to guidelines and other evidence-based practices ⁷⁰. It also supports the de-implementation of practices thought to be harmful or ineffective.[69] Implementation science is increasingly prioritized within funding agencies such as the United States National Institutes of Health,[70–72] who may view the field as a way to recoup investments in basic and clinical research innovation. In surgery, implementation science can help characterize factors that influence the adoption of new practices, and identify approaches that support sustained practice implementation [73–76]. The snapshot audit presents a multicenter look at current practices across settings, allowing a precise determination of the evidence-to-practice performance gap. The ‘funnel of attrition’, demonstrates the potential interplay of implementation science with snapshot audits in identifying barriers to closing the evidence-to-practice gap.[77,78] (Figure 4) Once identified, such barriers can be targeted using quality improvement interventions or through mixed methods implementation research.

Figure 4 -.

Funnel of attrition (after Glasziou, Haynes and White) in Implementation Science, demonstrating the potential interplay with snapshot audit.[77,78] EBI: evidence-based intervention

Conclusions

Snapshot audits are large time-bound prospective multi-center observational cohort studies focused on specific clinical conditions, care delivery, and short-term outcomes. They provide an overarching view of current surgical care. Unique considerations that may differ from retrospective analysis and RCT regarding team membership, database construction, and data analysis (including statistical analysis) are often needed. With careful study design, the approach also generates patient-level data of sufficient granularity to assess treatment effectiveness. As evidence-based management of numerous surgical conditions is incorporated into clinical practice guidelines, snapshot audits also offer the opportunity to study the barriers to guideline implementation and the underlying evidence-to-practice gaps. The snapshot audit serves as a platform for multi-center and often multi-national collaboration assessing the spectrum of patients receiving care during a time limited period. This inquiry secures an environmental scan of current care and outcomes while generating testable hypotheses to inform interventional trial design.