CONCERN Factorial Design Survey (FDS) Methods Test: Using REDCap as a Survey Platform

Abstract

We assessed the feasibility of using REDCap as an FDS platform. REDCap lacks randomization and automation functionality, requiring the development of a workaround. A template survey was created containing all vignettes, copied for each survey instance and edited to hide unwanted content. REDCap configuration required three hours for forty-two surveys. The utilized “copy-and-hide” workaround was successful, providing quasi-automation and reasonable labor-time. Additional strategies are planned using REDCap’s Data Dictionary and other survey software.

Introduction

Previous research shows that the effectiveness of clinical decision support (CDS) systems decreases when clinicians are presented with a high frequency of alerts that are not perceived as clinically relevant, leading to overriding or ignoring alerts [1]. Factorial design is an experimental design that contains two or more factors, which consist of discrete levels, whose experimental conditions take on all possible combinations of levels across all factors, referred to as vignettes. For example, a study containing three factors, each with two levels, has a total number of six possible combinations (2 × 2 × 2 = 6), or six vignettes. A factorial design survey (FDS) does not study the effects of individual factors, but enables researchers to draw conclusions about the significance of the vignettes. It allows researchers to study clinician decision-making, such as which combination of statistically significant factors are perceived as clinically significant [2]. In turn, this helps yield more generalizable conclusions. In the context of our CONCERN study, vignettes provide scenarios that can be indicative of patient deterioration in clinical documentation, and we are performing FDS to determine the perceived clinical relevance of these scenarios for our CDS model.

REDCap is a survey generator and electronic data capture software, but it has limitations when serving as an FDS platform. Although REDCap contains subject randomization capability, it lacks survey field randomization and automated field population that can help relieve the burden of this complex survey design on the researcher. This case study outlines a workaround developed to circumvent REDCap’s limitations and to facilitate its use to run a factorial design survey.

Methods

A pilot survey trial was conducted using a workaround, referred to as “copy-and-hide”. Four factors, each with two levels, were used in the pilot (Table 1). We sought to test surveys containing four of the sixteen total vignettes (Table 2). To assign which vignettes were tested in each survey, block randomization was performed off-platform. Each vignette was numbered (1–16), and a function in Excel was created that randomly assigned four numbers out of the 16 respectively to four surveys. A template survey containing fields for all sixteen vignettes was created in REDCap’s Online Designer. Every field in the template survey containing a vignette had the action tag “@HIDDEN-SURVEY” applied, which hides fields from survey participants’ view. For every consecutive test survey created, the template survey was copied, and the fields containing the assigned four vignettes to be tested in that survey had their action tags removed. This allowed participants to see and score these four vignettes in their survey and left the remaining twelve hidden from view.

Table 1–

Pilot Factors and Levels

Factor	Levels (count)	Factor Level Values
Factor 1. Vital sign frequency	2	≤22 sets of vital signs in the last 48 hours
		> 22 sets of vital signs in the last 48 hours
Factor 2. Note / comment frequency	2	≤5 comments/notes in the last 48 hours
		> 5 comments/notes in the last 48 hours
Factor 3. Highlighted Oxygen saturation comment/note	2	Yes
		No
Factor 4. Highlighted blood pressure comment/note	2	Yes
		No

Table 2-.

Sample Vignettes

Vignette Number	Vignette Content
1	Patient chart shows the following: ≤22 sets of vital signs in the last 48 hours ≤5 comments/notes in the last 48 hours A highlighted Oxygen saturation comment/note A highlighted blood pressure comment/note
8	Patient chart shows the following: ≤22 sets of vital signs in the last 48 hours >5 comments/notes in the last 48 hours No highlighted Oxygen saturation comment/note No highlighted blood pressure comment/note
11	Patient chart shows the following: >22 sets of vital signs in the last 48 hours ≤5 comments/notes in the last 48 hours No highlighted Oxygen saturation comment/note A highlighted blood pressure comment/note

Results

Forty-two surveys were created and disseminated to forty-two participants in three hours. Thirty-one responses were returned. Extracted data table was sized 43 by 821 cells.

Conclusions

The significance of this work is that we identified limitations of available survey software for performing a factorial design survey. Factorial design is an experimental design that necessitates multiple iterations of a survey, where its components’ material and order change from iteration to iteration. REDCap’s lack of randomization and automation functions are limitations. The “copy-and-hide” method was a successful workaround in that it provided our team a reproducible method with practical labor time (~4.5 minutes/survey). However, the reliance on manual input in this method introduces increased potential for human error when scaled up. Additionally, REDCap cannot differentiate between hidden and unhidden cells when a user extracts data, which created a large spreadsheet with numerous empty cells for the hidden fields that were not scored during our pilot. Scaled up, utilizing this method for FDS with hundreds of surveys would yield a cumbersome data workbook for researchers. A second pilot test is planned utilizing REDCap’s Data Dictionary, which is a completely off-platform survey designer that we hope will decrease labor-time further by allowing us to rapidly create smaller, specialized surveys which we can upload onto the software and streamline data extraction by decreasing the number of empty cells in the spreadsheet. Additionally, we plan to run an FDS pilot on other popular survey software and compare results.