Patients’ Reactions to Letters Communicating Collateral Findings of Pragmatic Clinical Trials: a National Web-Based Survey

Abstract

Background

Collateral findings in pragmatic clinical trials are findings that may have implications for patients’ health but were not generated to address a trial’s primary research questions. It is uncertain how best to communicate these collateral findings to patients.

Objectives

To determine how reactions to a letter communicating collateral findings relate to who signed the letter, the type of finding, or whether the letter specified that the finding arose from a pragmatic clinical trial.

Research Design

Web-based survey experiment using a between-subjects design in which respondents were randomly assigned within education strata to view and respond to 1 of 16 hypothetical scenarios.

Subjects

Adults recruited from an online panel constructed from a probability sample of US-based postal addresses.

Measures

The primary outcomes were the action the respondent would take next (i.e., contact a doctor immediately or something else) and the respondent’s emotional reactions (i.e., all positive, all negative, mixed, or none).

Results

A total of 4080 respondents had analyzable data. Although some effects were statistically significant (P < .05), none exceeded a prespecified threshold for policy relevance (15 or more percentage points). Ratings of letter clarity and level of understanding were lower for letters that included a description of the clinical trial.

Conclusions

Signatory and level of detail about collateral findings did not substantially affect people’s intentions to take the recommended action of contacting their doctor. Deciding whether to include a description of the pragmatic clinical trial requires a trade-off between transparency and more difficulty understanding the contents of the letter.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11606-021-07087-8.

INTRODUCTION

Pragmatic clinical trials generate evidence of how health care interventions work in real-world settings. Many pragmatic clinical trials, such as those supported through the National Institutes of Health (NIH) Health Care Systems Research Collaboratory, are embedded within health care systems. That is, the identification and random assignment of participants and sites, intervention delivery, and outcome assessment occur in the context of usual clinical operations. Such trials raise a number of ethical issues that have been described previously.² One issue relates to what we have coined “pragmatic clinical trial collateral findings” (PCT-CFs)—findings, discovered intentionally or unintentionally, that may have implications for patients’ health but were not generated to address the trial’s primary research question.⁵ For example, in a pragmatic clinical trial comparing the effectiveness of 2 medications for hypertension, researchers collecting data from the electronic health record might discover that a patient appeared to be taking 2 contraindicated medications.

There are important questions concerning the identification, assessment, and management of PCT-CFs. What constitutes a PCT-CF that should be reported? What are the roles and responsibilities within a health care system for managing PCT-CFs? To whom and how should findings be reported? The NIH Collaboratory conducted a multiphase project to understand the challenges of PCT-CFs and suggest policy recommendations for handling them. The first step was a conceptual analysis of ethical issues associated with PCT-CFs.⁵ The second step was a qualitative study of the perspectives of stakeholders in pragmatic clinical trials, including principal investigators and health care system leaders.⁶ The third step was a qualitative study of the views of the general public on the disclosure of PCT-CFs.¹

Results of this previous work and input from a multistakeholder advisory panel revealed significant uncertainty about whether and how to communicate PCT-CFs to patients. Specifically, it is unclear who should notify the patient and whether to explain that the PCT-CF was discovered in the context of a pragmatic clinical trial. It is also unknown how the effects of these 2 factors might depend on the type of PCT-CF. Therefore, we conducted an experiment, using a nationally representative web-based survey, to determine how reactions to a letter communicating a PCT-CF relate to who signed the letter, the type of finding, and whether the letter specified that the finding arose from a pragmatic clinical trial.

METHODS

We conducted a web-based survey using a between-subjects design in which respondents were stratified by educational level (i.e., high school degree or less vs some college or more) and randomly assigned within strata to 1 of 16 hypothetical scenarios. The scenarios reflected all possible combinations of who signed a letter notifying the patient about a PCT-CF, whether the letter included a detailed description of the pragmatic clinical trial, and the type of collateral finding (Table 1). In each scenario, the respondent visits their doctor, Dr. Lee, at City Medical Center for 1 of 3 possible indications: treatment for high blood pressure, evaluation of back pain, or routine screening for colon cancer. Two years later, the respondent receives a letter in the mail from City Medical Center reporting a collateral finding that may be important to their health. The respondent then records their reactions and perceptions in response to a combination of open- and closed-ended questions.

Table 1.

Experimental Factors and Levels

Experimental factor	Level	Description
Signatory	Personal physician	Chris Lee, MD Internal Medicine Practice City Medical Center
	Quality assurance officer or research administrator	Chris Davis, MD Chief Quality Officer Senior Medical Director for Research City Medical Center
Trial description	No	Letter includes a description of the finding, the fact that the finding was detected during a review of their medical records, and instructions to contact their doctor for follow-up.
	Yes	Letter includes a description of the finding, the fact that the finding was detected by researchers reviewing their medical records as part of a study, a description of the study, and instructions to contact their doctor for follow-up.
Finding	A/B drug trial—contraindicated medications	Trial: Cluster randomized trial comparing 2 medications for blood pressure control. Collateral finding: Detection of contraindicated medications in some patients.
	A/B drug trial—hematuria	Trial: Cluster randomized trial comparing 2 medications for blood pressure control. Collateral finding: Lack of documentation of follow-up for trace hematuria in some patients.
	Multisite imaging study—contraindicated medications	Trial: Multisite trial to identify the incidence of osteoporosis among patients with a spinal fracture. Collateral finding: Detection of contraindicated medications in some patients.
	Colorectal cancer screening—underperforming test kit	Trial: A cluster randomized trial comparing screening rates among patients who received test kits at routine clinic visits vs those who received them in the mail. Collateral finding: Detection of an underperforming test kit used by some of the participating study sites.

Sample

Sample selection and administration of the survey were managed by Ipsos. During the field period from May 5 through May 14, 2020, 7635 English-speaking potential respondents 18 years and older were randomly sampled from the Ipsos KnowledgePanel of 55,000 US adults. The sample recruitment process uses an address-based sampling method from the latest Delivery Sequence File of the US Postal Service—a database with full coverage of all delivery points in the USA. Thus, samples from KnowledgePanel cover all households regardless of their access to a telephone, providing fully representative online samples. Adults recruited from households without internet access are provided a web-enabled device, such as a tablet computer, and free internet service. Once assigned to the survey, panel members received an email invitation to participate in the study.⁴ Nonresponders received an email reminder after 3 days. Respondents who completed the survey received $5 worth of points that could be redeemed for cash, gift cards, or other merchandise.

Development of Scenarios

The study team drafted the 16 scenarios and letters based on previous focus group findings.¹ Draft letters were tested and revised on the basis of small-group cognitive interviews with 16 individuals of a variety of ages, genders, and races and ethnicities. Additional changes to the letters were made in response to individual cognitive interviews with a separate sample of 20 participants. To ensure that the scenarios and letters were realistic in terms of how the collateral findings would be reported to patients, the letters were reviewed by 4 health care system administrators who also served as members of the project’s advisory committee.

Data Collection

The study team drafted 16 versions of a survey, each of which began with a scenario and letter describing the finding (see Appendix 1 for example). Common to all survey versions was a series of open-ended, multiple-choice, and closed-ended questions that focused on the following topics regarding the hypothetical scenario and letter²: emotional reactions to receiving the letter⁵; why respondents had these reactions⁶; questions they had after reading the letter¹; what their next steps would be⁴; perceptions of why they were being sent the letter; and³ thoughts about the communication itself (Appendix 2). Questions about the communication addressed the amount of information in the letter, how well respondents thought they understood the communication, the clarity of the letter, and the acceptability of receiving the communication by mailed letter rather than other modes. For some items, the survey provided open text fields to allow respondents to explain their answers. The survey also included questions about respondents’ perception of their health, whether they regularly see a health care provider, and whether they thought their survey answers were influenced by the ongoing COVID-19 pandemic. All survey items were evaluated and revised through 2 additional rounds of individual cognitive interviews with a total of 20 participants, who also provided feedback on the letters as noted earlier.

After the survey data were collected, pre-banked information that was previously collected by Ipsos on panel members’ demographic characteristics and backgrounds was added to the data set. Study consent and execution were conducted online by Ipsos. After agreeing to participate, respondents read the description of the scenario to which they were randomly assigned, including the letter reporting the PCT-CF, and answered the questions described above. The Institutional Review Boards of the Duke University Health System and Johns Hopkins Medicine approved this research.

Data Analysis

We prespecified 2 primary outcome variables: the action the respondent would take next and the respondent’s emotional reactions. The only action the letters asked the respondents to take was to contact their doctor immediately. Therefore, respondents’ answers to a question asking them what they would do next were coded as “yes” if they would contact their doctor immediately and “no” otherwise. For the questions about emotional reactions, exploratory analyses suggested that the best way to reflect these reactions was to categorize respondents as selecting all negative, all positive, mixed, or no feelings.

Respondents who sped through the survey (i.e., they took less than 2 min to complete it) or who were missing responses to more than one-third of the questions were omitted from the analytic sample. Patient characteristics were summarized using standard descriptive statistics on the unweighted sample. All remaining quantitative analyses used survey weights to allow inferences to the US English-speaking adult population. The primary outcome, whether the respondent would contact their doctor immediately, was analyzed using a multiple logistic regression model that included the role of the letter’s signatory, the presence of a trial description, the type of PCT-CF, and all 2-way and 3-way interactions among these variables. We used a threshold of P < .05 as a heuristic to derive a final model by first testing the 3-way interaction, then an omnibus term for all 2-way interactions, and then individual 2-way interactions, with testing of lower-order effects only if the value for the current step was P > .05. Results for all retained covariates are expressed in terms of odds ratios and corresponding 95% confidence intervals, along with model-predicted probabilities by the level of each covariate.

We prespecified a policy-relevant difference between groups in a dichotomous outcome as 15 percentage points. We conducted sensitivity analyses by estimating the same model for respondents who reported having someone (nurse practitioner, doctor, etc.) they see for regular health visits and separately for those in worse versus better self-reported overall health (“poor”/“fair”/“good” vs “very good”/“excellent”). Another sensitivity analysis counted respondents who indicated they would contact either their doctor or some “other health care professional” as taking the recommended action. We analyzed the secondary outcome of emotional reaction profile the same as for the primary outcome, but using a multinomial logistic regression instead of a binary model. We conducted similar sensitivity analyses for respondents who had their own doctor and based on self-reported health.

Two members of the research team (JB, EM) reviewed responses from the open text field for the survey item, “What questions, if any, would you have?” and developed a 14-item codebook based on commonly recurring themes. The codebook was iteratively revised by the coding team with input from other coauthors (GG, KW). Each survey response was independently coded by 2 members of the coding team, who subsequently discussed and resolved any coding discrepancies (JB, EM).

RESULTS

Of the 7635 people who were sampled and invited to participate, 4218 (55%) completed the study. After removing 138 respondents for speeding through the survey and/or missing more than one-third of responses, the final analytic data set included 4080 respondents. The median time for survey completion was 6 min. Table 2 shows the demographic characteristics of the unweighted sample. The sample reflects diversity with respect to all key demographic variables.

Table 2.

Characteristics of Sample (Unweighted; N = 4080)

Characteristic	Overall, No. (%)
Age
18–29 y	471 (11.5)
30–44 y	842 (20.6)
45–59 y	1085 (26.6)
≥ 60 y	1682 (41.2)
Education level
Less than high school	329 (8.1)
High school	1685 (41.3)
Some college	845 (20.7)
Bachelor’s degree or higher	1221 (29.9)
Race/ethnicity
Black, non-Hispanic	368 (9.0)
Hispanic	486 (11.9)
Two or more races, non-Hispanic	130 (3.2)
White, non-Hispanic	2930 (71.8)
Other, non-Hispanic	166 (4.1)
Sex
Female	2023 (49.6)
Male	2057 (50.4)
Current relationship status
Married	2460 (60.3)
Living with partner	235 (5.8)
Separated	58 (1.4)
Divorced	445 (10.9)
Widowed	208 (5.1)
Never married	674 (16.5)
Metropolitan statistical area status
Metro	3478 (85.3)
Nonmetro	602 (14.8)
Region—based on state of residence
Northeast	750 (18.4)
Midwest	933 (22.9)
South	1465 (36.9)
West	932 (22.8)
Current employment status
Working—as a paid employee	2074 (50.8)
Working—self-employed	347 (8.5)
Not working—on temporary layoff from a job	9 (0.2)
Not working—looking for work	157 (3.9)
Not working—retired	1097 (26.9)
Not working—disabled	193 (4.7)
Not working—other	203 (5.0)

In response to the question “Do you think you would have answered any of these questions differently if we weren’t in the middle of a coronavirus/COVID-19 pandemic?” 85% of respondents answered “no,” 3% answered “yes,” and 12% answered “maybe.”

Contacting a Doctor Immediately

Respondents’ intention to contact a doctor immediately or not did not vary by the role of the letter’s signatory (P = .80), but there was an interaction between whether the letter included a description of the trial and the type of PCT-CF (χ²₃ = 21.8; P < .001). Inclusion of a trial description was associated with a lower likelihood of contacting a doctor among respondents assigned to the A/B drug trial with a PCT-CF of contraindicated medications, and a higher likelihood among respondents in the colorectal cancer screening trial with a PCT-CF of an underperforming test kit (Supplemental Figure 1). There was no relationship between trial description and intention to contact a doctor for the other 2 types of findings (contraindicated medications and hematuria). However, as shown in Table 3, neither of the effects observed for trial description resulted in differences in predicted probabilities that exceeded our prespecified threshold for a policy-relevant difference.

Table 3.

Predicted Probabilities of Intending to Contact a Doctor Immediately by Type of Finding and Level of Detail^a

Type of finding	Trial description
	No		Yes
A/B drug trial (BP)–contraindicated medications	0.64	>	0.52
A/B drug trial (BP)—hematuria	0.56	≈	0.54
Underperforming colon cancer test kit	0.49	<	0.55
Multisite imaging study—contraindicated medications	0.64	≈	0.52

^aNo differences exceeded the prespecified cutoff for a policy-relevant difference. Values represent predicted probabilities from a multiple logistic regression model that includes signatory, finding, detail, and the finding × detail interaction. Inequality signs (<, >) denote an effect of detail at P < .05. Approximate equality signs (≈) indicate P ≥ .05

In sensitivity analyses, the absence of effects exceeding the predefined policy-relevant threshold was consistent for respondents who had their own doctor and for those who self-reported worse and better overall health (data not shown). Broadening the correct answer to include immediately contacting another health professional or doctor also did not produce effects that exceeded the threshold.

Emotional Reactions

Respondents endorsed a range of emotional reactions after reading the letter, with “concerned” being the most frequent reaction (Table 4). In terms of the combination of emotions endorsed, 2869 (70%) respondents endorsed all negative emotions, 219 (5%) endorsed all positive emotions, 561 (14%) endorsed mixed emotions, and 423 (10%) endorsed no emotions. Membership in each of these categories was related to letter signatory (χ²₃ = 12.2; P < .007), trial description (χ²₃ = 9.9; P = .02), and type of PCT-CF (χ²₉ = 31.7; P < .001). There were no interactions among these factors. Supplemental Figure 2 shows the distribution of respondents in each emotional category by level of each experimental factor derived from the multinomial logistic regression model. For all 3 experimental factors, none of the differences in the percentage of respondents exceeded the predefined threshold for policy relevance. This lack of differences was consistent for respondents who had their own doctor and for those who self-reported worse and better overall health.

Table 4.

Endorsement of Emotional Reactions After Reading Hypothetical Letter

Emotion	No. (%)
Concerned	2436 (59.7)
Worried	1546 (37.9)
Irritated/annoyed	1121 (27.5)
Surprised	1018 (25.0)
Confused	958 (23.5)
Fearful	787 (19.3)
Angry	685 (16.8)
Grateful	544 (13.3)
No feeling	408 (10.0)
Overwhelmed	373 (9.1)
Hopeful	174 (4.3)
In control	151 (3.7)
Relieved	132 (3.2)
Other feeling	122 (3.0)

Questions Asked by Respondents

The number of respondents who asked a question was similar between those whose letters included a trial description (57%) and those whose letters did not (52%). Table 5 shows the distribution of question codes by whether the letter included a description of the trial. The distribution of codes was consistent across the 2 groups, with the largest difference not exceeding our predefined threshold of policy relevance of 15 percentage points. The most frequently asked questions in both groups addressed next steps that should be taken, health concerns, and a negative impact on trust or confidence.

Table 5.

Questions Asked by the Pragmatic Clinical Trial Description (Yes/No)^a

Code	Code description	PCT description, No. (%)
		No	Yes
Health concern	Any question or concern directly related to health and wellbeing, e.g., What are the long-term effects? Will I have an irregular heartbeat now?	351 (17.0)	227 (11.2)
Time delay	Any mention about the time delay, e.g., What took so long? Why was I not informed sooner?	154 (7.5)	74 (3.7)
Cost/time burden	Any question asking about cost or time, e.g., Do I have to take time off work to go get retested? How much is this going to cost?	58 (2.8)	40 (2.0)
Privacy	Any question or statement about privacy, e.g., Who was in my medical records? Is this HIPAA compliant?	22 (1.1)	71 (3.5)
Negative impact on trust/confidence	Any questions or statement signaling decreased trust/confidence in their doctor, health care system, the care provided, etc., e.g., Shouldn’t my doctor have checked this out before giving it to me? Will the next one be accurate or fail too?	230 (11.2)	169 (8.4)
Positive statements	Any positive statement, regardless of theme, e.g., I am grateful they wrote me the letter.	3 (0.1)	7 (0.3)
Misunderstanding/irrelevant	Any questions that did not pertain to the scenario or indicated that the respondent did not understand the scenario, e.g., I don’t take these medications.	110 (5.3)	103 (5.1)
Next steps	Any questions that asked for next steps or asked questions about the logistics of next steps, e.g., When can I get rescreened?	381 (18.5)	261 (12.9)
Legal liability	Any questions or comments regarding legal actions, e.g., Who do I sue? Who is liable?	24 (1.2)	22 (1.1)
Technical questions	Examples: How did they know the test kit did not work? What is the rate of false positives?	42 (2.0)	30 (1.5)
Concern about enrollment/consent	Any questions asking about how/why they were included in the study, e.g., Why wasn’t I asked for my consent? Did I give my permission to be in a study?	0	59 (2.9)
Defer questions to doctor	Any responses indicating they would defer to their doctor, e.g., I would just call my doctor. I would ask my doctor what he recommends.	32 (1.6)	18 (0.9)
Views about the letter	Any questions or statements about the communication, e.g., Why was a letter was sent instead of a phone call? Why would I call about research and quality control? Why is that section in the letter?	55 (2.67)	51 (2.5)
No questions	No response entered or responses stating the respondent had no questions.	894 (43.4)	962 (47.6)

^aNo differences exceeded the prespecified threshold of policy relevance (15 percentage points)

Other Reactions

When asked about the main purpose of the letter, the 2 most frequently endorsed answers were “to give you information about your health” (57%) and “to tell you to contact your doctor” (46%). Almost one-third of respondents (31%) endorsed “to protect people at the medical center from legal trouble.”

Most respondents rated the letters as “very clear” (45%) or “somewhat clear” (47%). Similarly, 51% said they understood the information “completely,” and 46% understood “somewhat.” In 7 of 8 cases, inclusion of a description of the pragmatic clinical trial resulted in lower ratings of letter clarity and subjective understanding (data not shown). When describing the amount of information contained in the letter, half of respondents (54%) felt it was the “right amount of information,” whereas 38% felt it was “not enough” and 7% felt it was “too much.”

DISCUSSION

As more pragmatic clinical trials are embedded within health care systems, there is a need to anticipate and respond to PCT-CFs. This study was conducted to provide evidence to inform the most effective strategies for communicating PCT-CFs to patients. The results demonstrate that, while there were subtle influences on responses, people’s initial emotional reactions to a letter reporting collateral findings and their intention to contact their doctor immediately were not affected to a noteworthy degree by the role of the letter’s signatory or whether the pragmatic clinical trial was described in the letter. People’s subjective impressions of the letter’s clarity and of their own level of understanding were generally lower for versions of the letter that included a description of the trial.

Our findings have several implications for crafting effective communications to patients about PCT-CFs. The likelihood that patients would follow the recommendation to contact their doctor was generally consistent across scenarios—around 50–60%. Thus, with respect to patient welfare, we would not expect any impact related to who sends the letter (such as a personal physician rather than a researcher or administrator) or whether the letter describes the pragmatic clinical trial that generated the collateral finding. With respect to patient rights, it could be argued that transparency in how the PCT-CF was identified is important. Yet, our findings are less clear on this point. When a trial description was not included, we did not observe any examples of respondents asking questions that would have been answered by a trial description. Respondents were more focused on next steps and what the collateral finding meant for their health. Similarly, inclusion of a trial description did not appear to increase the number of questions or change the nature of the questions, though it did adversely affect clarity and subjective understanding. More work would be needed to develop clearer communication about pragmatic clinical trials, if a trial description were to be included in such a letter.

Our findings support our prior observation from our qualitative work¹ that health care system leaders and clinicians should be prepared for negative reactions from patients who receive information about PCT-CFs. They should also be prepared for questions that focus on the range of issues described in Table 5, including questions about health concerns and next steps. The number of questions the health care system will need to field does not seem to be affected by who sends the letter or whether the pragmatic clinical trial was described. Also relevant for health care systems is the finding that one-third of respondents believed the primary purpose of the letter was to protect the health care institution from liability. These perceptions might stem from a broader mistrust of health care institutions. Future work could examine whether such reactions differ between traditional health care systems and those that are more integrated and structured as learning health systems.

Our study has several strengths. The large, representative sample allowed for statistically precise characterizations of experimental effects. To ensure the experiences of people with lower levels of education were well represented, we explicitly sampled and randomized within 2 levels of education. The letters we used were developed with qualitative input from members of the public and were vetted by health care system administrators for verisimilitude to current practice.

The study also has limitations. First, while responses to hypothetical vignettes can be informative,^3,7 it is unknown how well the findings translate to real-world situations. Our expectation is that the relative effects we observed in terms of differences between types of letters are representative of the effects one would observe with real letters sent to real patients. We expect the absolute response levels (e.g., the percentage of people who would contact their doctor immediately) may not generalize as well as the relative effects. Finally, it was challenging to capture and report the initial emotional reactions of the participants. Emotional reactions are often complex and dynamic. For example, a person might report feeling concerned and then explain their feeling in terms of both being worried about a potential health problem and being angry about their inclusion in a study without their awareness. Our strategy for querying and analyzing these reactions was informed by our prior qualitative work with the public,^1,6 but it is likely that significant aspects of these experiences are not adequately captured in web-based surveys.

Conclusion

In hypothetical scenarios in which patients were informed of collateral findings from a pragmatic clinical trial, the role of the letter’s signatory and the type of collateral finding did not substantially affect people’s intentions to take the recommended action of contacting their doctor immediately. Deciding whether to include a description of the pragmatic clinical trial that led to the collateral finding requires a trade-off between transparency and greater difficulty understanding the contents of the letter. Health care systems should prepare for negative reactions and questions from patients in response to such letters. Some questions might be avoided by developing better ways to describe pragmatic clinical trials in communications to patients. Future work should evaluate patients’ reactions to disclosures of PCT-CFs in real-world settings.

Funding

This work is supported within the National Institutes of Health (NIH) Health Care Systems Research Collaboratory by the NIH Common Fund through cooperative agreement U24AT009676 from the Office of Strategic Coordination within the Office of the NIH Director. Supplemental funding for this work was provided by the National Center for Complementary and Integrative Health of the National Institutes of Health under award number U24AT009676-02S2.

Declarations

Conflict of Interest

Dr. Sugarman is a member of Merck KGaA’s Bioethics Advisory Panel and Stem Cell Research Oversight Committee; a member of IQVIA’s Ethics Advisory Panel; a member of Aspen Neurosciences’ Scientific Advisory Board; a consultant for Biogen; and a consultant for Portola Pharmaceuticals, Inc. None of these activities is related to the material discussed in this manuscript. No other authors reported conflicts of interest.

Disclaimer

The views presented here are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health.