Nondrug interventions for reducing SARS-CoV-2 transmission are frequently incompletely reported

Abstract

Objectives

To investigate the completeness of reporting of behavioral, environmental, social and system interventions (BESSI) for reducing the transmission of SARS-CoV-2 evaluated in randomized trials, to obtain missing intervention details and to document the interventions assessed.

Study Design and Setting

We assessed completeness of reporting in randomized trials of BESSI using the Template for Intervention Description and Replication (TIDieR) checklist. Investigators were contacted to provide missing intervention details and if provided, intervention descriptions were reassessed and documented according to the TIDieR items.

Results

Forty-five trials (planned or complete) describing 21 educational interventions, 15 protective measures, and nine social distancing interventions were included. In 30 trials with a protocol or study report, 30% (9/30) of interventions were completely described; this increased to 53% (16/30) after contacting 24 trial investigators (11 responded). Across all interventions, intervention provider training (35%) was the most frequently incompletely described checklist item, followed by the ‘when and how much’ intervention item.

Conclusion

Incomplete reporting of BESSI is a substantial problem with essential information necessary for implementation of interventions and for building on existing knowledge frequently missing and unable to be obtained. Such reporting is an avoidable source of research waste.

What is new?

Key findings

• •The behavioral, environmental, social, and systems interventions (BESSI) which have been the mainstay of COVID-19 pandemic management are often incompletely presented in reports of randomized trials that are evaluating them. Completeness of intervention descriptions can be improved after contacting investigators, though the investigators frequently do not respond to requests for information.

What this adds to what is known?

• •There is poor reporting of elements needed for the use of interventions, but further information can be obtained by contacting study investigators and a standard reporting format (Template for Intervention Description and Replication [TIDieR] checklist) for describing intervention can be used.
• •This study assessed the extent of the problem of incomplete intervention description reporting of BESSI that have been relied upon to limit COVID-19 transmission and evaluated in randomized trials

What is the implication, what should change now?

• •Incomplete reporting of BESSI hampers the ability to build on the findings of existing trials and the implementation of interventions that limit the transmission of SARS-CoV-2 in the context of pandemic urgency where rapid research replication and application are needed.
• •Incomplete intervention reporting is an avoidable source of research waste.
• •The TIDieR checklist can be used by researchers, registries and journal publishers to improve standardized descriptions of interventions.

1. Introduction

Controlling COVID-19 transmission has relied on behavioral, environmental, social, and systems interventions (BESSI) such as social distancing, face coverings, ventilation, hand hygiene, and quarantine. Despite the importance of BESSI the investment in their development and evaluation has been a fraction of that spent on pharmacological and vaccine interventions [1]. The RESIN (Research Investments in Global Health) study found that only 3–4% of the $3.3 B global funding for COVID-19 research to date was on BESSIs [2]. The BESSI collaboration website—www.bessi-collab.net—maintains a “scorecard” which compares the number of trials of the pharmacological treatments to trials of BESSI, showing over a 100-fold difference in trials registered or completed.

Given the importance of BESSI in the pandemic response, it is critically important that these interventions are described clearly and completely in trials evaluating them. However, previous studies have found that interventions are often incompletely described in the reports of trials, and this is especially the case for nonpharmaceutical interventions such as BESSI^.3 The Template for Intervention Description and Replication (TIDieR) checklist was developed to assist investigators in better reporting of details of interventions, but is also useful to assess the completeness of reporting of interventions in a sample of trials [3].

Detailed descriptions of interventions in BESSI trials–sufficient to allow replication—are important for several reasons: (i) the appropriate interpretation of and comparison to similar interventions and synthesis in systematic reviews, (ii) accurate implementation of effective interventions in practice, and (iii) the replication and improvement in new research studies. This study aimed to assess the completeness of reporting of BESSI evaluated in randomized trials, obtain missing details, and document details of the BESSI for COVID-19 that have been assessed in randomized trials to date.

2. Methods

2.1. Eligibility criteria

Randomized controlled trials comparing any BESSI to any other active or control intervention with the aim of preventing or reducing the transmission of COVID-19 or impacting behaviors that may prevent or reduce the chance of being infected with, or spreading COVID-19 were included. Trials planned, in progress or completed reporting intended, actual or self-reported behavioral outcomes were included. Trials suspended or terminated were excluded.

2.2. Search strategy

Trials were identified from a database of studies evaluating BESSI maintained by the BESSI collaboration (https://www.bessi-collab.net/). Studies in this database are identified through monthly searches of the COVID-19 Living Overview of Evidence (L.OVE) platform and by members of an email list interested in BESSI research. Two authors independently evaluated studies in the database on November 30, 2021, against the eligibility criteria and discrepancies regarding eligibility were resolved by discussion with other authors.

2.3. Assessment and extraction of intervention description

2.3.1. Rating of intervention description completeness

Intervention descriptions in each trial were rated using the TIDieR checklist [3]. The TIDieR checklist consists of 12 items related to the intervention that when completely described facilitates replication of the intervention. The checklist items are described in Appendix Table A.1 (and Figure 2).

Fig. 2.

Percent of interventions in BESSI trials rated as completely described for each checklist item before and after investigator contact.

We rated trials that were planned or in progress where the trial methods were reported in a protocol and trials that were completed and reported in a journal publication, preprint, or working report. We did not rate trials when trial information was reported in a trial registry only. When multiple documents about a trial were available (e.g., a trial registration and publication), we rated the completeness of intervention description using all the available information.

Trial interventions were rated for completeness of reporting of each checklist item as (i) “yes” (intervention completely described), (ii) “no” (intervention not completely described), or (iii) “not applicable”. Not applicable was used for item 10 (modifications) and 12 (how well, actual) when a protocol was rated, for item 12 (how well, actual) when item 11 (how well, planned) was not reported and for item 9 (tailoring) and 10 (modification) when tailoring or modification was not a feature of the intervention. If tailoring or modification were considered possible, the reporting was assessed as being complete, incomplete (and the investigator was contacted) or “not reported” when tailoring or modification was not mentioned in the report (it was assumed for the purposes of this study that there was nothing to report for items nine and 10 and investigators were not contacted for information). Authors (S.S. and E.G.) rating the interventions trialed the spreadsheet and rating process on one trial together then four trials independently. Differences in interpretation of the checklist items and ratings were discussed with another author (T.H.). The remaining trials were independently rated by two authors (S.L.S. and E.G.) and discrepancies were resolved through discussion with another author (T.H.). When trials evaluated different versions of the same intervention (e.g., different wordings of an informational message), only one version was rated. Descriptions of control group interventions were not rated.

2.3.2. Collection of intervention details from trial investigators

When details for a checklist item were assessed as missing from the intervention description or were incompletely described, details of the missing item were recorded, and the corresponding author of the trial contacted by email. The email to the investigator provided information about the aims and authors of this study and included a request to provide detail about the specific checklist item missing or incompletely described in their trial. A template of the investigator contact email is provided in Appendix Figure A.1. Investigators who did not respond to the first email request were sent a reminder email after 2 weeks.

2.3.3. Rerating of intervention description completeness after investigator contact

If a response was received from the investigators, two authors (S.L.S. and E.G.) independently rerated the intervention descriptions using the information received as “yes” (intervention description complete after investigator reply) or "no” (intervention description incomplete after investigator reply), resolving discrepancies through discussion or with another author (T.H.).

2.3.4. Extraction and recording of intervention description and trial characteristics

We extracted and tabulated details on the type of BESSI (categorized as educational interventions, social distancing measures, protective measures, and public space disinfection as per the taxonomy of BESSI in the COVID-19 L.OVE–Appendix Figure A.2), location of the trial (country), type of trial (individual or cluster randomized, factorial or adaptive), funding, and the type of outcomes reported (infections or behaviors) in each trial. TIDieR details of interventions were tabulated independently by two authors (S.S. and E.G.) with discrepancies resolved by discussion and with reference to another author if necessary. For completed trials reported in a journal publication or preprint, we checked the main article for reference to, and supplementary files for provision of a completed TIDieR or Consolidated Standards of Reporting Trials (CONSORT) checklist [3,4].

2.3.5. Data analysis

We calculated the proportion of rated trials (as a percent of trials where that item is applicable) that provided a complete description of the intervention before and after trial investigator contact. This was calculated as the number of trials rated as “yes” on items 1–9 of the TIDieR checklist/the number of rated trials. We considered items 1–9 to be essential for replication of the intervention itself. Items 10–12 were not included in this calculation as these items were considered to be providing information on study level factors that relate to interpretation of the effects of the intervention rather than information crucial to implementation of the intervention. Completeness of reporting of items 1–9 after investigator contact was compared between the types of interventions.

3. Results

Of 53 studies in the BESSI database on November 30, 2021, 45 trials met eligibility criteria for this study (Fig. 1 ). Thirty of these trials reported intervention details in a protocol, preprint, report or academic journal. Investigators of 24 of these 30 trials were contacted for additional intervention information, with 11 investigators responding (Fig. 1).

Fig. 1.

Flow chart showing study selection and investigator contact process, ∗proportion of interventions completely described when the item is applicable.

3.1. Characteristics of included trials

Characteristics of the included trials are summarized in Table 1 and details provided in Appendix Table A.2. Almost half (21/45; 47%) of the included trials evaluated an educational intervention. The remainder evaluated protective measures (15/45; 33%); most of these measures were face masks or shields, or social distancing measures (9/45; 20%) including reopenings, school practices, contact tracing, quarantine, and immunity passports. Most studies were conducted in the United Kingdom or United States (7 trials in each country) with 3 trials conducted in multiple countries. In 60% of trials, the outcome was a behavior or behaviors that impact on transmission/acquisition of COVID-19. At the time of manuscript writing, 51% (23/45) of included trials had been completed. Just over three-quarters of the trials (36/45; 76%) had a trial registration record and 11 trials (11/45; 24%) had provided a protocol as a preprint, journal publication or as an attachment to a trial registration record. One trial published as a preprint provided a completed TIDieR checklist in a supplementary file [5], and two trials published in journals provided a CONSORT checklist (the CONSORT-EHEALTH checklist and the CONSORT checklist for cluster trials) in supplementary files [6,7].

Table 1.

Characteristics of included trials (n = 45)

Category of BESSI intervention	Educational interventions n = 21 Protective measures n = 15 (face masks/shields n = 12, hand hygiene n = 3) Social distancing measures n = 9 (reopening n = 4; school practices n = 1; contact tracing n = 1; quarantine n = 1; immunity passports n = 1; cash transfer n = 1)
Status of trial (at 30/11/2021)
Trial planned or in progress	Trial registration only n = 15 Protocol ( ± trial registration) n = 7
Trial completed and results available	Preprint ( ± trial registration, protocol, working report) n = 5 Working report ( ± trial registration, protocol) n = 3 Journal publication ( ± trial registration, protocol, preprint, and working report) n = 15
Location of the trial	UK n = 7; US n = 7; India n = 5; Bangladesh n = 3; Norway n = 3; Multiple countries n = 3; Australia n = 2; Ireland n = 2; China n = 2; single studies (n = 11) in Germany, Switzerland, Egypt, Columbia, Pakistan, Denmark, Spain, France, Guinea-Bissau, Canada, Brazil
Type of trial	Individual n = 29 Cluster n = 13 Factorial n = 2 Adaptive n = 1
Trial outcomes	Behavior/s that may impact transmission/acquisition n = 27 SARS CoV-2 confirmed infections or COVID-19 like illness/symptoms n = 17 Both SARS CoV-2 confirmed infections and behaviors that may impact transmission/acquisition n = 1
Trials reporting funder/s	Public funder n = 14 Academic institution = 7 Private funder n = 6 Academic institution + charity n = 2

3.2. Completeness of intervention descriptions

Nine of the 30 rated trials (30%) provided a complete description (i.e., completely described checklist items 1-9) of the intervention evaluated prior to investigator contact. For all the checklist items (items 1-12), the percentage of interventions rated as completely described before investigator contact, is shown in Figure 2 . All the rated trials provided a brief name for the intervention (Item 1) and a rationale for the key components of the intervention (Item 2). The materials used in the intervention (Item 3 Part A) were completely described in 90% (27/30) of the trials, though details on where the materials could be accessed (Item 3 Part B) were provided in only 73% (22/30) of trials. The intervention provider (Item 5), how the intervention was delivered (Item 6 Part A and Part B), and where the intervention was provided (Item 7) were completely described in over 90% of trials. The training of intervention providers (when training was applicable) was the least frequently described item (35%; 5/14). Whether fidelity was assessed or whether strategies were used to maintain or improve fidelity and how this was done was also often incompletely described (46% of trials; 14/30). The number of checklist items and item numbers incompletely reported in each trial are presented in Appendix Table A.3. Sixty percent (18/30) of assessed trials incompletely reported one, two, or three of the 12 checklist items and 20% incompletely reported more than 3 items.

Of the 24 investigators contacted 11 responded (46%), with 10 providing sufficient description to change the rating of intervention description to completely described for some (n = 2) or all (n = 8) item/s (items 1-12). After investigator contact, 16/30 rated trials (53%) provided a complete description (i.e., completely described checklist items 1-9) of the intervention evaluated. For each checklist item, the percentage of interventions rated as completely described after investigator contact is shown in Figure 2. Examples of incompletely described and completely described interventions before and after investigator contact are provided in Table 2 . Details of the trial interventions according to TIDieR checklist items including intervention details obtained from investigators are provided in Appendix Table A.4.

Table 2.

Examples of intervention descriptions before and after investigator contact

TIDieR item	Brief description of the intervention	Incomplete item description before investigator contact	Complete item description after investigator contacta
4. What (procedures)	Germ Defense interactive website to improve infection control	Contact with general practices to share the weblink to the Germ Defense website and asking the general practitioners to provide the Germ Defense link to their patients.	Contact with general practices to share the weblink to the Germ Defense website and asking the general practitioners to provide the Germ Defense link to their patients. Investigators recommended the link be provided to patients via text or email using templates providedb, but practices could choose to place the link on the practice webpage or share it via social media with the suggested wording.
5 B. Who provided (training)	Preventive intervention for a mass-gathering indoor event	Training provided to nurses and security crew.	Brief training was provided to nurses of how to perform a nasopharyngeal swab and how to organize the screening structure so as to screen 1,000 participants in a single morning. Training provided to security crew about how to control the flow of the participants in the venue, and to monitor wearing of face masks by participants when in the venue.
8. When and how much	Instructional videos about COVID-19 and how to stay safe a. 10 min fact based video b. 22 min fact based plus underlying scientific concepts video	Interventions delivered over 2 d between April 21–24 2020. Video length for a. 10 min, b. 22 min	Interventions delivered over 2 days between April 21–24 2020. The video could be rewound or re-watched as many times as participant wanted but once the participant pressed “Continue” they could not go back and watch the video again. Video length for a. 10 min, b. 22 min.

^aAdditional information provided by investigator in italics.

^bExample of mass text message (160 characters) that can be sent (using MJog, accurRx, iPLATO, or similar) to patients. You may choose to edit this text, for example by adding your practice name so that patients know who the text is from: National Health Service general practitioners practices recommend this website www.germdefence.org with scientifically proven advice to reduce COVID-19. It only takes 10 min. Example of an email that can be sent to patients: We are letting you know about a very useful website called Germ Defense which was created by a team of doctors and scientists to give you advice that has been proven to reduce the spread of viruses in the home. It can help you plan how to protect yourself and members of your family from infection by COVID-19 and flu. It's easy to use and only takes 10 min–just click this link” www/germdefence.org/ (if this link does not open when you click on it, please copy and paste it into your web browser) Please pass details of the Germ Defense website to your friends and family. There is a button at the bottom of the Germ Defense website for sharing by social media. If you'd like to know more: (bullet points) over 20,000 people previously took part in research about Germ Defense. People who followed the advice in Germ Defense had fewer and less severe illnesses–and so did the people they lived with. Results of the study were published in The Lancet medical journal. Germ Defense has been updated with COVID-19 advice to help prevent a wave of COVID-19 and flu this Autumn/Winter. Information about how the Germ Defense website is being evaluated is available here (http://www.bristol.ac.uk/primaryhealthcare/researchthemes/roll-out-of-germ-defence-website/).

Completeness of the reporting of TIDieR items varied according to the type of intervention (Appendix Figure A.3). Access to materials of educational interventions (Item 3 Part B) were more often completely reported than social distancing or protective measures (75%; 15/20 for educational vs. 60%; 3/5 for social distancing; and 60% (3/5) for protective measures). Intervention procedures (Item 4) were more frequently completely described for educational interventions (90%; 18/20), then social distancing (80%; 4/5), and protective measures (60%; 3/5). Training of intervention providers (Item 5 Part B) was completely described by only half (3/6 to whom training of providers is applicable) of educational interventions and protective measure interventions (2/4), but by none of the social distancing interventions (0/4). Social distancing interventions also infrequently completely described the details of where the intervention took place (Item 7), whereas all educational and protective measure interventions completely described this item.

4. Discussion

This study reports on the incomplete reporting of BESSI for COVID-19 and provides details of the interventions to facilitate their replication and implementation. Most interventions evaluated related to educational interventions followed by protective measures (facemasks/shields and handwashing) then social distancing. At the time of analysis, just over half of the planned BESSI trials had completed and reported findings (as a preprint, report, or journal publication). For most of the 30 trials available as a publication, protocol, preprint, or report, intervention reporting is insufficient to allow replication or application. Some items are infrequently reported, such as training of the intervention provider (Item 5B). Contact with investigators led to more complete description of interventions, but about half of the investigators did not respond, or the response was insufficient to consider the intervention completely described.

As far as we are aware, this is the only study to have examined intervention descriptions in trials of BESSI that might be used to reduce COVID-19 transmission. Duplicate rating of the completeness of intervention descriptions and extraction of intervention details is the strength of this study. A limitation is that some trials which were only registered at the time of analysis may have since completed and additional intervention details made available in preprints, publications, or reports. Further new trials of BESSI may have commenced. The most recent version of trial documents and new BESSI trials can be accessed at https://www.bessi-collab.net/. We did not assess the reporting of control interventions in the BESSI trials though these should also be completely described in order to aid the interpretation of trial findings and allow meaningful conclusions to be drawn.

The rates of intervention description completeness found in this study are comparable to studies of other nondrug interventions. For example, an analysis of 173 randomized trials in ear, nose, and throat surgery found 60% had less than 60% adherence to the TIDieR checklist items [8]. A review of 23 exercise trials for diabetes found that the replication of each exercise was not possible in 52% of the interventions [9]. Similar to other studies, completeness of intervention descriptions was improved by contacting investigators for additional information [[10], [11], [12]]. In contrast to studies contacting investigators of trials published over a wider period of time, we were able to deliver our request for information to all trial authors. However, the problem of investigator unresponsiveness to requests for information despite reminders, and inability of investigators to provide sufficient information to make the description complete is consistent with other studies. Though the trials assessed in this study were planned or conducted within the last 2 years, recency of publication does not seem to improve responses to requests [10,13].

Given the need for rapid implementation of effective nonpharmaceutical interventions to reduce the transmission of COVID-19, complete intervention descriptions are a vital public health resource. Regardless of whether the intervention is effective, complete descriptions are crucial for the interpretation of the results, and necessary to allow replication of such research and if applicable, implementation into practice. Incomplete intervention descriptions in trials also significantly limits the usability and reproducibility of systematic reviews that include them and downstream evidence sources such as guidelines, that base recommendations on systematic review evidence [14]. The poor reporting of interventions is an important but correctable source of research waste [15].

Improvements in the reporting of intervention descriptions in clinical trials could occur at the design, registration, and reporting of findings stage. Tools such as the TIDieR author tool (http://www.tidierguide.org/#/author-tool) which is designed to assist those writing intervention descriptions are useful for protocols as well as later reporting of results. However, the effect of reporting guidelines such as TIDieR depends in part on authors being aware of them and following them carefully. We therefore urge trialists and research educators to provide training in how to better describe interventions using tools such as TIDieR. The effect of reporting guidelines also depends on their use being enforced with checking of the completeness of reporting in articles prior to acceptance. We urge trial registries, preprint servers, and journals to adopt this approach. In addition, more journals should recognize that Item 5 of CONSORT (“The interventions for each group with sufficient details to allow replication, including how and when they were actually administered”), and item 17 of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 checklist for reporting systematic reviews (“Cite each included study and present its characteristics”)–often needs supplementing with guidance from TIDieR on the details needed [4,16]. While TIDieR may help to address some of the reasons for suboptimal intervention reporting, there can be other barriers in the evidence ecosystem that impede intervention reporting and our subsequent ability to replicate interventions beyond trials. These must also be addressed if the situation is to improve.

The need to improve the descriptions of nonpharmaceutical interventions has been clear for many years. However, urgency in the COVID-19 pandemic for rapid replication and application further highlights the importance of complete descriptions in published reports.