Perspective on reducing errors in research

Abstract

Efforts to ensure research integrity has mostly focused on research misconduct. However, the complexity of research operations and processes makes research work also prone to unintentional errors. To safeguard against errors and their consequences, strategies for error reduction, detection, and mitigation can be applied to research work. Nurturing a scientific culture that encourages error disclosure and rectification is essential to reduce the negative consequences of errors. Creating repositories where errors can be reported can enable learning from errors and creation of more robust research processes.

1. Background

Efforts to ensure research integrity has to date been mostly focused on research misconduct. Research misconduct involves intentional actions to produce false research findings. Less attention has been devoted, however, to unintentional errors that would similarly result in reporting of incorrect research findings. The magnitude and importance of this matter is unknown. Many of the errors may remain undetected by researchers, or discovered post publication and not reported. Research work typically involves multiple team members and the research process is complex with ample opportunities for errors. Thus, it is unrealistic to expect research work to be error-free without deliberate actions to reduce the likelihood of error occurrence and increase researchers’ ability to detect and mitigate them once they occur. To our knowledge, there are no guidelines that discuss how to approach unintentional errors in research.

We recently discovered a serious statistical programming error in one of our trials. This had a profound impact on the interpretation of our research findings and led us to initially report incorrect research conclusions [1]. Subsequently, we conducted a grounds-up reanalysis of the trial's data from which we gained insight about several aspects of the research process that may make it vulnerable to errors. In this brief paper, we share what we learned and describe error reduction, detection, and mitigation strategies that have been used in healthcare and can be applied to research work.

2. Findings from grounds-up reanalysis

The involved trial tested the impact of a support program for patients hospitalized for chronic obstructive pulmonary disease (COPD) on acute care use (number of visits to hospital and emergency department) and health-related quality of life (score on the St. George Respiratory Questionnaire [SGRQ]; score range 0 (best) −100 (worst)). The identified error was in a file used for preparing the datasets for statistical analysis, and occurred when the variable referring to the study group assignment was recoded to change the randomization assignment variable format of “1, 2” to “0, 1” for analysis. This was performed incorrectly and resulted in a reversed coding of the study groups leading us to interpret the differences between study groups in a direction opposite to what occurred. This did not raise suspicion of an error to our investigative team because the findings of this erroneous analysis supported our study hypotheses. The programming error was discovered during a secondary data analysis of the economic impact of the study outcomes. After discovering the programming error, we conducted a grounds-up reanalysis starting from re-exporting the data from the study's secure online database system where the data were directly entered by the research team via tablet devices, and then repeating all data preparation, programming, and analysis. During this process, we detected three other errors and one of those had an impact on interpretation of study findings. In this study, we imputed the worst possible SGRQ score (100) for participants who died. However, a few participants had missing SGRQ score at 6 months but died after that timepoint. A value of 100 was incorrectly imputed for their missing score instead of leaving it as missing. After correcting this error, the significant difference in health-related quality of life between the study groups was no longer present. The other two errors occurred in summarizing the baseline medication classes in the patient characteristics and a missed count of two hospitalizations among participants [1]. We subsequently retracted the original study publication and republished the results of the reanalysis [1,2].

3. Strategies to reduce errors and their consequences in healthcare

In 1999, the National Academy of Sciences in the United States published its landmark report “To Err is Human” which highlighted the fact that medical errors do occur and have severe consequences [3]. This report helped accomplish the essential first step towards solving any problem which is to acknowledge its presence no matter how unpleasant that acknowledgment might be. Since then steady efforts have been made to address medical errors in patient safety research, healthcare professionals’ education, clinical practice, health system design, and policy/regulation. Strategies to reduce errors and their consequences that have been used in the manufacturing, nuclear power, and aviation industries are being applied to healthcare with important impacts on the fields of anesthesia and pharmacy among others. The goal is to reduce the likelihood of error occurrence as much as possible, and maximize the chance of detecting and addressing it before it reaches the patient and results in possible harm. A hierarchy of strategies exists where the most effective is to prevent the error (eliminate possibility of its occurrence) whenever possible. Next is to detect the error and correct it before it reaches the patient. And, last is to mitigate the effects of the error which involves minimizing its negative consequences on the patient (e.g. providing prompt resuscitation treatment after a medication overdose is administered to a patient) [4]. Safety research has increasingly focused on the importance of maintaining a workplace culture where it is accepted that despite best efforts to prevent errors, some will still occur and systems must be in place to detect them and mitigate their effects [5]. Several safety practices have since been shown to reduce risk of adverse events in healthcare [6].

4. Reducing errors in research

In their 2017 editorial Correcting the Medical Literature: “To Err Is Human, to Correct Divine”, Christiansen and Flanagin urged researchers who discover errors to report them [7]. That step, though extremely important, will only help mitigate the effects of discovered errors on the published medical literature. But what about the errors that we do not discover and may lead us to wrong pathways of thinking, inquiry, and clinical practice? Calls have been made for replicating research, data sharing, and making statistical code available for review [8,9]. Though these are important approaches, they are not sufficient to safeguard research against errors and their consequences. Creating more robust and resilient research processes requires a close examination of work practices, identification of potential failure modes, and development of best practices that safeguard against them. Table 1 depicts strategies to reduce errors and their consequences, with examples of how they have been applied to healthcare and could be applied to research work.

Table 1.

Strategies to reduce errors and their consequences, with examples of application to healthcare and research work.

Strategy description	Example of how it has been applied to healthcare	Example of how it can be applied to research work
Prevent Error
Establish a reliable process (standardize whenever possible)	Consistent use of a checklist that details steps for safe insertion of central lines prevented catheter-related blood stream infections [10].	Create a study data management plan that details how data elements will be handled and adequately train research team members performing data handling tasks. For example, describe how missing values will be exported and coded distinguishing the handling of zero values, codes for missing values (like 999), and out of range/impossible values. Also, specify the type of data (dates, text, numbers) and pre-define a value range to identify out of range/impossible values.
Change process (or device) so that it is impossible to make the error anymore	A safety system was incorporated in the design of the anesthesia machine that safeguarded against the possibility of delivering the wrong gas supply. The system included a specific pin configuration for the Oxygen and Nitrous Oxide gas cylinders which made the user unable to connect the cylinder to the incorrect plug [11].	Use statistical software that allows for programming and direct export of tables and any associated text instead of copying/pasting values from analytic output. This eliminates possibility of errors from copying the wrong values or pasting them incorrectly into the table.
Eliminate unneeded tasks or parts	The concentrated injection solution of esmolol HCl (250 mg/mL) used to treat cardiac arrythmias was discontinued to prevent medication overdoses that resulted from failure to dilute it. Currently, this medication is available in ready-to-use 10-mg/mL vial (does not require dilution) [12].	Use direct data entry into computer devices (e.g use tablets or laptops to directly enter data as you collect it) rather than writing on paper forms and then reentering the data into computer. The data entry programs should include checks for inconsistencies or out-of-range responses. Avoid variable recoding as much as possible (if needed, clearly name and label the recoded variable for audit).
Facilitate the work, reducing complexity and ambiguity, so that it is less likely to make a mistake (e.g. use checklists and well documented procedure manuals)	Use of electronic medical record systems with built in algorithms facilitated decision-making and prescribing of venous thromboembolism (VTE) prophylaxis medications [13]. Tall-man lettering on medication vials is used to prevent mixing up look-alike drugs [14].	Create a process for data managers and analysts to become familiar with research study background, design, and all input forms and instruments, before proceeding to data preparation and analysis. ( E.g. hold dedicated meetings for this purpose prior to starting any data preparation for analysis) Maintain a single electronically-locked master data file from which data can then be exported for specific analytic purposes. Any version of a master data file should be annotated with a datetime stamp to confirm the latest version to be used. Documentation should note the reasons for subsequent master data files.
Create work environment that strives to prevent errors and supports teamwork	In response to high rates of preventable adverse events post hospital discharge, interventions have been implemented to facilitate coordination between inpatient and outpatient providers, including adding dedicated case managers and transition coaches on healthcare teams to help with discharge planning, and addressing patients' needs [15].	Consider handoffs of information and responsibilities (e.g. with team member turnover) as high risk/error-prone periods. Ensure sufficient communication and clarity on who is doing what at such times, and how any questions will be resolved.
Detect error
Make errors more visible/discoverable	Patient identification bands are used to avoid patient misidentification errors [16]. Electronic prescribing system alerts to prevent medication errors [17].	Use variable names that refer to specific forms so that they can be audited back to their source document, making errors more visible/discoverable. (Use industry standards and best practices as applicable) Run range checks and challenges for improbable and impossible values. Check consistency of values across study visits.
Create redundancy (i.e. multiple checks}	Independent double check of medication doses for high-alert medications [18].	Have critical and error-prone tasks performed by two independent individuals. This includes checks if summary tables and values have been copied/pasted. (Have this checking plan specified ahead of time so the two independent individuals apply the same rules).
Mitigate the effect of errors
Minimize direct effects of the errors	Rapid resuscitation measures for victims of medication overdose.	Report corrections for all published work that is affected by errors.
Learn from mistakes to prevent similar future events	Promoting safety culture and use of voluntary error- reporting systems to learn from errors and institute measures to prevent their recurrence [19].	Promote a culture that encourages admission of errors, discussion of underlying causes, and learning from them. This would require a collective ongoing effort from research leaders and funders to acknowledge that errors do occur in well conducted research, encourage reporting of those errors, and support those who report them.

5. Conclusions

Like any other field, research work is prone to errors. Despite researchers’ best efforts to be vigilant, given the complexity of research operations and processes, it is unrealistic to expect research work to be devoid of errors. Without deliberate actions to detect errors, they may go undiscovered and result in incorrect conclusions that can negatively impact clinical practice and contribute to the inconsistency in research findings across studies. Similar to healthcare, increasing awareness about errors in research, and applying systematic strategies to prevent and detect them is warranted. Nurturing a scientific culture that encourages error disclosure and rectification is essential to reduce the negative consequences of errors. Creating repositories where errors can be reported and later analyzed can enable learning from them, and inform best practices development and creation of more robust research processes.