Designing a web-based factorial survey experiment on diagnostic evaluation of cardiovascular symptoms in emergency care

Abstract

Background

Guidelines describe evidence-based diagnostic testing strategies for patients with suspected pulmonary embolism (PE), yet there is sparse empirical literature on how clinicians decide whether to suspect PE.

Objectives

Our goal was to design a valid and reliable approach to assess how patients’ demographics, symptoms, and risk factors impact clinicians’ decisions to pursue PE testing.

Methods

We designed and conducted early-stage validation of an online factorial survey experiment featuring 7 vignettes of hypothetical adult emergency department (ED) patients with cardiopulmonary symptoms. Using a 2 × 2 × 2 factorial design, we generated 8 versions of each vignette that differed by patient gender (woman/man), race (Black/White), and a case-specific clinical factor. In each vignette, participants rated their suspicion for PE and other cardiovascular conditions using a visual analog scale and indicated diagnostic tests they would order. Participants were not informed of the study design. We measured test–retest reliability in 16 physicians from 4 EDs and conducted pilot testing at a large national conference of ED clinicians.

Results

Visual analog scale scores for perceived PE risk demonstrated high test–retest reliability (Pearson r = 0.85). Pilot testing by 61 ED physicians from 18 states had a response rate of 68%, a completion rate of 95%, and a mean completion time of 8.1 minutes (SD, 3.1 minutes). This cohort reported higher perceived PE risk in vignettes with intermediate-risk rather than low-risk revised Geneva scores, providing early support for this tool’s criterion validity. Cognitive debriefing interviews suggest good face validity and response process validity.

Conclusion

These pilot data provide preliminary support for the validity and reliability of using an online factorial survey with clinical vignettes to experimentally assess ED clinicians’ perceptions of PE risk and decisions for PE diagnostic testing.

Essentials

• •Pulmonary embolism (PE) is often misdiagnosed in the emergency department (ED).
• •We created a factorial vignette survey to examine ED clinician decisions on PE testing.
• •Pilot testing suggests appropriate validity and test–retest reliability (Pearson r = 0.85).
• •Future research can use this approach to identify factors that lead to PE diagnostic error.

1. Introduction

A central goal of emergency department (ED) care is the rapid evaluation of individuals with undifferentiated symptoms to identify and stabilize those with clinically important acute conditions. For most patient encounters, this process involves a clinician’s data collection, generation and refinement of a differential diagnosis, and decisions to pursue diagnostic laboratory or imaging tests. For certain life-threatening cardiovascular conditions, the threshold for ordering diagnostic tests is very low. Many EDs now use nurse-initiated protocols at triage to expedite low cost, rapid diagnostic tests such as electrocardiograms (ECGs) and chest radiographs (CXRs) to quickly identify life-threatening, time-sensitive conditions including ST-elevation myocardial infarction, pneumothorax, and pulmonary edema. However, for most conditions, decisions about diagnostic testing depend on a clinician’s assessment of disease likelihood (such as the degree of overlap between a patient’s presentation and conditions that share similar signs, symptoms, and test results), as well as the potential risks of diagnostic testing (including unnecessary radiation exposure; false-positive or false-negative results; and delays in ED throughput). For patients with classical or pathognomonic findings for serious conditions, the assessment is simple since these elements match well-known heuristics (learned mental models) of common disease presentations. However, many patients with serious illness do not have classical symptoms for their disease, and these individuals may experience delays in diagnostic testing.

Pulmonary embolism (PE) is the third leading cardiovascular cause of death in the United States and the most frequently misdiagnosed cardiovascular condition in acute care [[1], [2], [3], [4]]. Several factors complicate the diagnosis of PE. Disease severity spans a very broad spectrum, ranging from asymptomatic presentations to obstructive shock and cardiac arrest. Between 20% and 50% of patients with a first-time PE have no classical risk factors for venous thromboembolism disease at the time of diagnosis [[5], [6], [7], [8]]. Many of the signs and symptoms classically associated with PE (eg, pleuritic chest pain, tachycardia, and unilateral leg swelling) are absent in over half of patients with the disease, substantially limiting these elements’ negative predictive values. Lastly, guidelines provide evidence-based diagnostic pathways for patients with suspected PE, yet data are sparse on how physicians decide whether patients should be considered for PE testing.

While the postpartum period and the use of oral contraceptives are well-recognized risk factors for PE [9], male patients have a higher overall age-adjusted incidence of PE than female patients [[10], [11], [12], [13], [14], [15], [16]]. Despite this, many PE registries and diagnostic studies enrolled a substantially higher proportion of female patients [[17], [18], [19], [20]]. This higher testing rate in female patients was associated with a lower diagnostic yield for PE. In a secondary analysis of the PE rule out criteria validation trial, female patients in every age group had a higher frequency of computed tomography (CT) imaging for PE and a lower yield on those tests than age-matched male patients [21]. This testing pattern suggests potential harms at both extremes: over-testing in female patients (with the associated risks of breast radiation exposure, especially in young women) and undertesting and misdiagnosis in male patients. However, it is unclear whether physicians pursued CT imaging in female patients more often due to specific risk factors or whether physicians have an implicit perception that female patients have an intrinsically higher likelihood of PE.

We designed the Physician Estimation of Risk factors and Clinical hEuristics In diagnostic eValuation of Pulmonary Embolism (PERCEIVE-PE) study to identify factors associated with ED clinicians’ perceived risk of PE and their decisions to pursue PE diagnostic testing in patients with undifferentiated cardiopulmonary symptoms. This report describes the iterative process to develop and perform early validation of a novel web-based factorial survey instrument. Our long-term goal is to use this tool to examine factors associated with PE diagnosis, but this approach could be adapted to answer other questions related to diagnostic reasoning in the ED setting.

2. Instrument Design and Pretesting

Our study was designed to identify factors associated with ED clinicians’ diagnostic evaluation and testing decisions for patients presenting with cardiopulmonary symptoms. Eligible participants included emergency medicine (EM) physicians (faculty and residents) and advanced practice providers (APPs; physician assistants and nurse practitioners) who evaluate adult patients and order diagnostic tests in ED or urgent care settings in the United States or Canada. Exclusion criteria included clinicians who cared only for pediatric patients and those who had retired from clinical practice >6 months prior to enrollment. We excluded ED clinicians who exclusively care for pediatric patients because PE is predominantly a disease of adults and pediatric PE risk factors are less well characterized. The study was approved by the institutional review board at Oregon Health & Science University.

The survey instrument included 4 parts (Figure). Part 1 comprised a factorial survey experiment (FSE) featuring a series of 1-page clinical vignettes regarding ED patients with cardiopulmonary symptoms. Part 2 included questions about participants’ demographics, geography, clinical experience, and risk tolerance. Part 3 solicited participants’ opinions on how a wide range of clinical features affects their decisions to pursue testing for PE, and part 4 examined the diagnostic strategies that participants use to select and interpret diagnostic tests when evaluating patients for PE.

Figure.

Survey diagram. PE, pulmonary embolism.

Initial study design was informed by discussion with subject matter experts in PE, diagnostic reasoning, and survey design. We then conducted pretesting with a cohort of 12 ED physicians (7 attending physicians and 5 residents) with a similar range of age and experience as the intended study audience; this sample allowed thematic saturation. These 1-on-1 sessions were conducted using internet-based video conferences in which a participant completed the instrument while sharing their screen view so the principal investigator could assess task completion. We instructed participants to verbalize their interpretation of the survey layout, the questions, and any barriers they encountered. At the end of the session, a study team member conducted a brief semistructured cognitive interview with the pretesting physician to elicit feedback about the design and face validity of the survey.

2.1. Part 1: Vignette design

We designed the vignettes based on discussion with the above-referenced content experts, and each vignette element was subsequently revised based on iterative pretesting with the pretesting cohort. The design included 7 clinical vignettes of hypothetical ED patients with symptoms including chest pain, dyspnea, palpitations, or near syncope. Each vignette consisted of a 1-page summary that described the history of present illness, medical history, and physical examination of each patient. After reading the vignette, participants would answer 2 questions relating to their clinical suspicion for different cardiovascular diseases and the types of diagnostic tests they would obtain to evaluate these symptoms.

We used a factorial design to allow for experimental assessment of how patient characteristics (eg, gender; race) or clinical elements (eg, symptoms and risk factors) were associated with participants’ diagnostic evaluations. We dichotomized these 2 elements (female and male; Black and White) for the purposes of the factorial design. The third factorial in each case was a clinical element that could impact clinicians’ suspicion of PE through diagnostic anchoring or premature closure, such as the timing of symptom onset; presence of tachycardia; or a known chronic disease that could cause the patient’s symptoms (eg, chronic obstructive pulmonary disease). Participants were assigned vignettes in a random order, and each participant received only 1 version of each case.

2.1.1. Vignette content

Design Vignettes described adult patients (age range, 27-71 years) with a new symptom of dyspnea, chest pain, palpitations, or near syncope within the past 7 days (Supplementary Table 1). The history of present illness was described in a 3-sentence paragraph. The vignettes were designed to be realistically undifferentiated in a manner reflective of routine ED care. All cases had at least 1 associated nonspecific symptom such as nausea, subjective fever, cough, or fatigue. We scored each vignette using 3 PE risk prediction models (Wells score for PE; revised Geneva score [RGS]; and 4-Level PE Clinical Probability Score [4PEPS]) and revised pertinent clinical elements to ensure each vignette was in a low- or intermediate-probability category (eg, Wells Score ≤ 6, RGS ≤ 10, and 4PEPS ≤ 12) even if the physician considered PE the most likely diagnosis [[22], [23], [24]]. We generated a patient name for each vignette by randomly selecting given names and surnames from lists of the 10 most common gender-neutral given names and surnames in the United States [25,26].

2.1.2. Feedback and revision

We asked the pretesting cohort to review each vignette for realism, to assess whether vignettes had an appropriate level of diagnostic uncertainty, and to describe their level of engagement or mental fatigue as they completed each subsequent vignette. We also solicited feedback on the visual layout of vignette details. The vignettes had high face validity, with all participants indicating that the vignettes were realistic and had similar complexity to patients they might evaluate during a routine ED shift. When asked to indicate the most likely diagnosis for each of the 7 vignettes, the majority responses were heart failure (2 cases), PE (2 cases), ischemic heart disease (2 cases), and bronchitis (1 case). All respondents reported maintaining normal attentiveness through 5 vignettes; half of participants reported a subjectively important level of mental fatigue or disengagement by the start of the sixth vignette, and 11 of 12 respondents did so by the start of the seventh vignette. As a result, we reduced the number of vignettes provided to each participant to 5. We also revised the history of present illness section to use bullet points, as 7 respondents (63%) indicated this format would improve readability and comprehension.

2.1.3. Facial photograph stimuli

Design: Each vignette included a representative color facial photograph matched to a patient’s age, race, and gender. The photographs were placed in the upper left corner of the vignette page and measured 1 × 1.5 inches when viewed on a 24-inch monitor. Photographs were included to enhance vignette realism and to potentially evoke unconscious biases that may arise during real clinical encounters but might not be triggered by a written vignette alone. We searched for facial stimuli databases that had (1) high-resolution color photographs; (2) subjects who expressed a neutral expression; and (3) validated norming data for gender, age, and race. We also preferred photograph datasets that had standardized elements (eg, consistent neutral background colors and similar clothing on all participants) to reduce other sources of unconscious bias.

2.1.4. Photograph selection

We obtained photographs with approval from 3 facial stimuli databases, including the FACES database (Max Planck Institute for Human Development, Berlin, Germany) [27], the Face Research Lab London Set [28], and the Chicago Face database [29]. Using norming data provided from these data sets, we selected photographs that matched the age, gender, and race of each hypothetical patient case. For example, a vignette featuring a 71-year-old patient required 4 photographs (eg, Black female, Black male, White female, and White male) of individuals who could credibly appear to be 71 years old. We considered photographs to be age matched if the norming data reported an average estimated age within 3 years of the age indicated on the vignette (eg, a vignette of a 71-year-old patient could use a photograph for which norming data indicated a perceived age between 68 and 74 years). We were unable to find appropriate high-resolution facial stimuli for older Black patients for 14 cases across 5 vignettes. In these cases, we purchased stock images (Shutterstock). As in the facial stimuli datasets, we selected photographs of individuals with neutral facial expressions and similar nondescript clothing.

2.1.5. Feedback and revision

We surveyed the pretesting cohort and 6 additional research assistants to gather norming data on the age, gender, race, and facial expression of each stock photograph subject. We classified race and facial expression using schemas from the Chicago Face database [29]. We replaced 2 photographs based on age results, but the remaining photographs had appropriate criteria (eg, matched age, race, and gender; and neutral facial expression) for inclusion.

2.2. Question stems and answer scales

Each vignette included 2 questions about PE risk perception and diagnostic testing that we developed based on discussion with content experts in diagnostic reasoning and survey design.

2.2.1. Question 1 design

Based on the information provided in the case summary, participants were asked to rate their perceived risk of each of 5 common diseases among ED patients with cardiopulmonary symptoms: heart failure; obstructive lung disease such as asthma or chronic obstructive pulmonary disease; respiratory infection such as bronchitis or pneumonia; ischemic heart disease; or PE. Although PE evaluation was our primary focus, participants were not informed of this. We included other conditions to not only preserve uncertainty regarding the study purpose and to reduce demand characteristics but also allow the collection of secondary outcomes on evaluation of other cardiovascular conditions [30]. We excluded diagnoses that could be confirmed or excluded by basic laboratory tests or rapid noninvasive tests, such as a pneumothorax identified on CXR or an ST-elevation myocardial infarction on ECG. Participants indicated their perception of each diagnosis using a visual analog scale (VAS); our initial design used 2 response anchors of low risk and high risk on the left and right ends of the scale, respectively. Participants did not assign a diagnosis to the hypothetical patient or indicate the most likely diagnosis.

2.2.2. Question 1 feedback and revision

We gathered input from the pretesting cohort regarding content validity, response process validity, and response burden for question 1. Most respondents (n = 10) asked for clarification as to whether the answer scale represented the probability of disease (eg, as a percentage) or the representativeness of disease; our intent was the latter. For instance, a vignette could strongly align with a physician’s mental model of a disease (eg, highly representative), yet the overall likelihood of that disease could be low if there were other common causes of similar symptoms (eg, low probability). To clarify this question and improve response process validity, we solicited input on preferred terms for this concept. Most respondents disliked representativeness (Representative of what? I don’t know how to answer this if the patient has atypical symptoms) and clinical suspicion ([suspicion] has negative connotations). Fortunately, all respondents agreed that clinical concern was an appropriate and widely recognized phrase for this concept, so we changed the response anchors to low clinical concern and high clinical concern.

The 5 conditions in question 1 were initially presented in a random order to reduce order-effect bias. Respondents uniformly stated that this design element was disruptive and made it more difficult to answer questions, so we changed the list to alphabetical sorting. Respondents also recommended removing the default markers (0 and 100) on the VAS as these numerical markers implied that the answer should be a probability.

2.2.3. Question 2 design

Participants were next asked to indicate the type of diagnostic testing they would order for the patient in each vignette. Answers were selected from a set of common laboratory tests and diagnostic imaging studies performed in the ED. Participants were also given the opportunity to enter additional diagnostic tests they would order. A nuanced element of PE diagnostic testing is whether physicians use D-dimer screening, which is the guideline-recommended approach for patients who have low- or intermediate-risk pretest probability, as in these vignettes, or CT pulmonary angiography (CTPA). We pilot tested several approaches for presenting the D-dimer and CTPA options. Due to limitations of the survey software, respondents could either (1) be allowed to select zero, 1, or both the options or (2) be restricted to selecting only 1 option by placing the items in a separate category that permitted a single response.

2.2.4. Question 2 feedback and revision

Question 2 answer choices were revised following pretesting. EDs use nurse-delegated protocols to order basic tests (complete blood count, metabolic panel, ECG, and CXR) on patients with cardiopulmonary symptoms before they are seen by a clinician, and these 4 tests were selected in 93% of all pretesting vignettes (n = 78). Nine respondents recommended reducing the response burden by preordering these tests in the question stem. Six respondents indicated they typically wait for these results before ordering additional tests and suggested that question 2 would not accurately elicit PE testing patterns if participants did not have these test results. As a result, the stem in question 2 was revised to indicate that these 4 tests were obtained and were normal.

We also solicited feedback on participants’ interpretation of the D-dimer and CTPA choices. There was uniform agreement that placing D-dimer and CTPA in a separate category from other tests was awkward and drew unnecessary attention (It makes me think about ordering them). All respondents suggested we include D-dimer with the other laboratory tests and CTPA with the other imaging tests, so we modified the layout to incorporate these tests into their respective categories. One possible outcome of this arrangement was that study participants could potentially select both D-dimer and CTPA; all respondents agreed that this set of answers should be interpreted as the CTPA order being contingent on the D-dimer result.

2.3. Part 2: Participant information

Participants completed a brief set of questions on demographics (age, sex, gender, and race), clinical characteristics (clinical role and year of residency completion), and US state or Canadian province and type of institution (academic medical center; community hospital affiliated with EM residency; community hospital not affiliated with EM residency; other acute care hospital including Veterans Affairs; freestanding ED or urgent care; and other) at which they work primarily. We also assessed traits relating to risk tolerance using the 6-point risk tolerance scale of the Jackson Personality Inventory [31], which previous literature has associated with physician-level differences in diagnostic imaging for patients with abdominal pain [32]. These questions were placed after the vignettes to minimize attrition and prioritize the amount of clinical data obtained.

2.4. Part 3: PE diagnostic elements

After completing the first 2 sections, participants were informed of the study objective to examine factors that influenced PE diagnosis. Next, we introduced a freelisting exercise in which participants reported factors that they believed influenced diagnostic decisions for PE. Freelisting is a qualitative research method used to identify how a group of respondents defines and perceives a specific topic [33]. Our survey instructed participants to input up to 10 elements that influence their PE testing decisions.

The second component in this section asked participants to rate how different clinical elements impact their likelihood of ordering a diagnostic test for PE. These elements were clustered into 5 groups: recent medical events; symptom types; symptom duration and onset; the absence of classical symptoms; and factors relating to pregnancy or exogenous hormone use. These elements were answered using a VAS with response anchors of less likely to test and more likely to test and corresponding numerical answers (hidden to respondents) from −100 to 100.

2.4.1. Feedback and revision

Eight respondents recommended adding a no impact on testing response anchor in the middle of the VAS rating scale answer scale. We revised the instructions and anchors to incorporate this feedback. For the second exercise, participants expressed concerns about mental fatigue and the number of different categories. To reduce response burden, we changed the survey design to randomly assign participants only 4 of the 5 categories and to randomly assign only a subset of items from each category (range, 4-8 items for each category), for a maximum total of 23 items.

2.5. Part 4: PE diagnostic strategies

The final section of the survey assessed participants’ awareness and use of different diagnostic strategies for PE. These elements included pretest probability assessment tools (Wells score, RGS, and 4PEPS), as well as approaches to interpreting D-dimer results (conventional; age-adjusted; and YEARS) [[34], [35], [36], [37]]. Participants indicated which approaches they were aware of, followed by which approaches they had used clinically.

2.5.1. Feedback and revision

We asked respondents whether they used a consistent diagnostic strategy for PE or whether they sometimes used different approaches. More than half of respondents (n = 7) indicated that they used multiple pretest probability strategies or D-dimer interpretation methods. As a result, we opted to maintain a question format in which respondents could select all approaches they used, rather than asking respondents to select a single approach most often used.

3. Pilot Testing

3.1. Reliability

We assessed test–retest reliability in a cohort of 16 emergency medicine physicians at 4 emergency departments (1 academic ED and 3 community EDs) at 2 health systems in Oregon and California. This cohort included 6 physicians who had participated in pretesting and 10 physicians not previously involved in the study. Participants were invited to complete the study and were informed that our team would contact them in 3 weeks for follow-up questions, but participants were not informed of the retesting step at the time of the initial testing. Each participant received a randomly assigned set of 5 vignettes during the initial test, and this order was maintained during retesting.

Participants completed retesting an average of 25 days (range, 21-39 days) after the initial survey. We calculated Pearson coefficient to assess the pairwise correlation for each participant’s perceived clinical concern for all 5 clinical conditions assessed in question 1. Across all participants and conditions, we observed a Pearson coefficient (r) of 0.84. Participants’ clinical concern for PE had a pairwise correlation of 0.85, and we observed similar correlations for the other 4 conditions (range, 0.82-0.87). Reliability differed across individual pilot testers but was consistent (range, 0.73-0.91). Results were similar for individuals who participated in pretesting (r = 0.83; range, 0.73-0.90) and those who did not (r = 0.86; range, 0.76-0.91).

3.2. Validity

We conducted an internal pilot test among attendees at the Society for Academic Emergency Medicine 2023 Annual Meeting. The intent of this step was to assess criterion validity and response burden and to confirm the adequacy of the enrollment target. Participants completed the survey by opening a link in a web browser or scanning a QR code. After the survey, a subset of participants opted to participate in brief cognitive interviews using verbal comprehension probes to assess elements revised in pretesting, such as response process validity, face validity, and response burden. Participation incentives included a drawing performed at the end of the study for 1 of 8 $500 gift cards. We approached 94 potential participants, of whom 64 (68%) participated and 61 (95%) completed the survey. The pilot cohort included 19 faculty/staff physicians and 42 residents from 18 states. The mean age of respondents was 34 years (range, 25-70 years). The average time for survey completion was 8.1 minutes (SD, 3.1 minutes; range, 3.2-20.4 minutes).

Pilot data were normally distributed. All vignettes demonstrated question 1 scores for PE that were statistically insignificant on the Shapiro–Wilk test (P value ranged from 0.13 to 0.81 across cases) and had appropriate ranges of skewness (−0.53 to 0.51) and kurtosis (−0.96 to −0.52). We performed a preliminary assessment of criterion validity (eg, how well did question 1 compare with an established standard) by comparing vignettes with low-risk RGSs to vignettes with intermediate-risk scores. We found a significantly higher perceived risk of PE, as measured by a 0-to-100 VAS, in intermediate-risk vignettes (mean difference, 13.3 points; 95% CI, 12.4-14.3; P < .01).

Eleven respondents participated in brief cognitive interviews. These individuals interpreted question 1 as intended to assess a physician’s clinical suspicion for disease rather than probability of disease based on the description in the question stem (n = 9) or the formatting of the answer scale (n = 2). All participants agreed that the vignettes were realistic and representative of typical undifferentiated ED patients. No respondents indicated that the response burden was unacceptable.

4. Discussion

In this study, we designed and pilot tested a web-based FSE to evaluate clinical concerns and diagnostic testing decisions by ED physicians and APPs for patients with acute cardiopulmonary symptoms. Our initial results provide early support to indicate that this tool has appropriate face validity, construct validity, response process validity, and test–retest reliability for this purpose. We will describe our use of the factorial vignette design and its use in prior similar studies; compare this approach to other methods for assessing clinical decisions and implicit bias; and characterize the critical role of these studies in improving diagnosis of PE.

The FSE method was designed in the 1950s and more broadly disseminated in the 1980s to assess individuals’ attitudes and decisions in complex situations [38,39]. This approach benefits from the strengths of a classical experiment while using a simple survey–based tool that reduces barriers to participation, and it is widely used across multiple fields including education [[40], [41], [42], [43]], economics [[44], [45], [46], [47]], and marketing [[48], [49], [50], [51], [52]]. Studies using FSEs have examined decisions by emergency and critical care physicians regarding time-sensitive decisions such as intubation [53,54], and other FSE studies in health care have examined decisions regarding chronic disease management [55], use of low-value care [56], treatment of musculoskeletal pain [57], and shared decision making [58]. A 2008 systematic review identified 30 studies published between 1983 and 2005 that used clinical vignettes, of which the majority (73%) examined treatment decisions rather than diagnostic decisions [59]. We are aware of only 1 prior vignette study regarding diagnosis of PE published in 1986 [60]. This study found that (1) there was vast heterogeneity in diagnostic decisions across individuals at each level of experience (eg, medical students, interns, third-year residents, and faculty); (2) clinicians’ diagnostic decisions were most strongly influenced by vignette variables related to leg examination findings, arterial partial pressure of oxygen, and lung ventilation–perfusion scintigraphy results; and (3) vignette variables related to age and heart rate had virtually no impact on diagnostic decisions.

The FSE method is one of several methods for assessing clinical reasoning and implicit bias. The think aloud protocol (TAP) method is a concurrent qualitative method in which interviewers ask clinicians to verbalize their thoughts while performing a task. This approach gathers highly granular data that reveal the intermediate steps of reasoning (eg, hypothesis generation, data interpretation, and differential diagnosis refinement) and differentiate between knowledge errors and information processing errors (eg, premature closure) [[61], [62], [63], [64], [65]]. However, the TAP has resource-intensive data collection and analysis methods that limit feasible sample sizes and make it difficult to use in environments (eg, clinical settings) with high cognitive loads and frequent distractions. Video-stimulated (or video-cued) recall (VSR) is a similar approach in which participants describe their reasoning while watching a video of a task they recently performed [[66], [67], [68]]. VSR reduces the cognitive burden imposed during the task and allows for deep reflection after the event, but it is susceptible to post hoc rationalization in which participants may incorrectly attribute the causes of their decision during a task.

Individuals’ attitudes include both implicit (unconscious) and explicit (conscious) elements, and the TAP and VSR methods both focus purely on conscious elements. Implicit attitudes are more strongly associated with subsequent behaviors (including physician treatment decisions) than explicit attitudes [[69], [70], [71]]. The implicit association test (IAT) was designed to measure implicit bias by measuring participants’ reaction times when associating stereotypically similar or dissimilar terms (eg, associating strength with terms related to male or female gender). Literature using the IAT suggests the presence of gender and racial bias in physician evaluation of coronary disease [70,72]. However, several studies have demonstrated that the IAT has limited predictive validity for future behavior [[73], [74], [75]]. In this regard, an FSE approach offers the advantage of demonstrating a manifestation of bias (eg, disparities in decisions) among a group of respondents rather than the potential for bias.

PE diagnosis is a critically important area for evaluating physicians’ diagnostic decisions. Current guidelines provide recommendations for patients with suspected PE, yet the factors used by ED physicians to suspect PE in routine ED practice are unclear. PE is the most frequently misdiagnosed cardiovascular condition in emergency care, and understanding the factors that influence clinicians’ diagnostic evaluations for PE is a critical prerequisite for improving patients’ outcomes. Since the current results suggest that this factorial design survey is a reliable tool for measuring these evaluations, our team will use this approach to evaluate a larger cohort of clinicians in academic and community EDs in the United States and Canada. This future study will seek to identify signs of (1) implicit bias related to gender or race or (2) cognitive biases (eg, premature closure or anchoring) related to clinical elements in a patient’s presentation.

Our study has a few limitations. Although our pilot testing evaluated measures of validity and reliability in respondents from different clinical and geographic settings, this testing group was primarily composed of residents and attending physicians and does not represent the full heterogeneity of ED physicians. Our FSE used dichotomous categories for gender and race based on feasibility requirements, but we acknowledge that gender exists on a spectrum and race is a social construct that is significantly more nuanced than a Black–White dichotomy. The vignettes contain a restricted set of symptoms and do not contain all elements that clinicians may use in their diagnostic evaluation of cardiopulmonary symptoms. There is a possibility that our response format could bias participant responses. For example, it is possible that participants could express a lower concern (or no concern) for PE or other conditions if our question 1 did not require participants to rate their concern for each condition. Similarly, providing a set of common ED diagnostic tests in question 2 could induce participants to select tests that they would not indicate in the absence of such a list. However, any impact of this formatting would likely impact all vignettes and would not selectively bias the results for a certain experimental factor (eg, gender or race).

4. Conclusion

We designed and performed initial validation of a web-based factorial survey experiment to evaluate factors that influence diagnostic evaluations by emergency physicians and APPs for patients with cardiopulmonary symptoms. These initial results suggest that this method has appropriate face validity, construct validity, response process validity, and test–retest reliability. This tool will allow for experimental assessment of factors that influence clinicians’ diagnostic decisions for PE and other acute cardiopulmonary conditions. Future research should further validate this approach and use it to assess gaps between evidence-based PE testing recommendations and practice patterns observed in real clinical practice.