Patient and family contributions to improve the diagnostic process through the OurDX electronic health record tool: a mixed method analysis

Abstract

Background:

Accurate and timely diagnosis relies on sharing perspectives among team members and avoiding information asymmetries. Patients/families hold unique diagnostic process (DxP) information, including knowledge of diagnostic safety blindspots - information that patients/families know, but may be invisible to clinicians. To improve information sharing, we co-developed with patients/families an online tool called “OurDX.” We aimed to characterize patient/family contributions in OurDX and how they differed between individuals with and without diagnostic concerns.

Method:

We implemented OurDX in two academic organizations serving patients/families living with chronic conditions in three subspecialty clinics, and one primary care clinic. Prior to each visit, patients/families were invited to contribute visit priorities, recent histories, and potential diagnostic concerns. Responses were available in the electronic health record and could be incorporated by clinicians into visit notes. We randomly sampled OurDX reports with and without diagnostic concerns for chart review and used inductive and deductive qualitative analysis to assess patient/family contributions.

Results:

7075 (39%) OurDX reports were submitted at 18,129 pediatric subspecialty clinic visits and 460 (65%) reports were submitted among 706 eligible adult primary care visits. Qualitative analysis of OurDX reports in the chart review sample (n=450) revealed that participants contributed DxP information across 10 categories, most commonly: clinical symptoms/medical history (82%), tests/referrals (54%), and diagnosis/next steps (51%). Participants with diagnostic concerns were more likely to contribute information on DxP risks including access barriers, recent visits for the same problem, problems with tests/referrals or care coordination, and communication breakdowns, some of which may represent diagnostic blindspots.

Conclusion:

Partnering with patients and families living with chronic conditions through OurDX may help clinicians gain a broader perspective of the DxP, including unique information to co-produce diagnostic safety.

INTRODUCTION

Prevention of diagnostic errors is a global priority. Diagnostic errors are estimated to affect 12 million Americans each year, cause significant harm to patients and families, and are the leading cause of malpractice claims in ambulatory care.[1] The landmark National Academy of Medicine (NAM) report on improving diagnosis emphasized the importance of engaging patients and families in diagnosis,[1] but to date proven mechanisms are lacking.

While patients/families are central to the diagnostic process (DxP), [2–6] they lack a systematic mechanism to notify clinicians of DxP concerns. This is a missed opportunity because patient/family insights are vital, especially when there is increased clinical complexity, patients are experts of their own illness, or care involves multiple visits with different providers or healthcare centers.[7] In addition, when patients/families share information, they do not always feel heard.[2–4] In an analysis of 596 patient-reported diagnostic errors in a nationally representative survey, not feeling heard by providers was the single most common patient-reported contributing factor to diagnostic error, described by nearly 70% of respondents.[8] Yet clinicians and organizations do not routinely assess whether patients have diagnostic concerns or whether they feel heard in real time and therefore lack the opportunity to identify, at the point of care, DxPs that are at risk.[9]

Diagnostic safety experts increasingly recognize the concept of “situativity” as central to improving diagnostic safety.[10,11] They underscore that diagnosis relies on distributed cognition – unique knowledge held by patients and clinicians – and is also influenced by environmental factors that may be beyond the view of any single individual.[10–13] For example, diagnostic delay or error may occur when clinicians miss or misunderstand key patient symptoms, or are unaware of test results or visits occurring outside the organization. They may also transpire when patients misunderstand or do not remember clinician-recommended next steps, such as diagnostic tests or referrals.[2,3,14] Improving diagnostic safety outcomes therefore relies on bidirectional sharing of critical information used for decision-making among team members, including patients and families.[15–17]

Health information transparency serves as a new platform to systematically engage patients and families in the DxP.[2,17–19] With implementation of the U.S. Cures Act Final Rule in April 2021, easy access to electronic health information is now federally mandated, and similar efforts are growing internationally.[20–22] In the U.S. and several other countries[21,22], patients can read visit notes to better understand the clinician’s perspective and determine whether their story was accurately captured.[3,19] In addition, some patients who read their visit notes identified important “blindspots” – safety issues that are known to them but may not be apparent to clinicians or organizations.[17] Patients who read notes also report using them to better remember diagnostic tests and referrals, make informed care decisions, and develop greater trust in their providers.[18] Similar benefits were observed for patients with chronic illness or those who may be at risk of healthcare disparities.[18,23–26] Evolving use of patient portals and health information transparency means patients can share their uniquely held information in new ways.

Based on these foundational principles, we co-developed an online tool called “OurDX (Our Diagnosis)” with patients and families.[2,19,27–29] In prior research, we found that participants used the tool, identified diagnostic concerns, and that the majority of these were verified on clinician review. In this paper, we aimed to: 1) qualitatively characterize patient/family contributions to the DxP; 2) assess whether patient/family contributions were actionable to clinicians; and 3) compare DxP contributions between participants who reported diagnostic concerns vs those who did not.

METHODS

OurDX development

OurDX is an online pre-visit survey co-designed by patients; family members; clinicians; and experts in user-centered design, diagnostic error, and patient experience. Our goal was to develop a streamlined DxP engagement tool that captures relevant and actionable patient/family-reported information while minimizing the burden on both patients and providers.[30] We adapted items from previously tested pre-visit surveys and a patient-centered diagnostic engagement framework, focusing on information to help providers at the point of care.[2,27] The tool was tested with additional patients and clinicians outside the design team prior to implementation, with integration of feedback. Further details of OurDX design have been reported elsewhere.[31,32] In summary, the tool focuses on 3 domains: 1) visit priorities; 2) patient/family-reported history; and 3) potential concerns related to the DxP (Supplement 1). The tool is an academic non-commercial product that is freely available for use. Patient contributions through OurDX were imported into the medical record and could be used by clinicians to co-create the visit note.

OurDX implementation

We tested the tool in two healthcare organizations between December 2020 - March 2022; three medical and surgical subspecialty clinics in an academic pediatric center serving urban patients and families (site 1), and one primary care general medicine clinic in an academic center serving rural adult patients (site 2). Participants were recruited to complete OurDX as a pre-visit survey via email. Site 1 used Tonic Health software (Murray, UT) that did not require a portal account, potentially reaching a broader population. Site 2 used an Epic patient portal. We tolerated site-specific differences to prioritize existing information technology and clinical workflows, as a way to optimize sustainability of tool use after the study period.

Study population

Eligible participants included all patients/families with new and returning visits in the 3 participating clinics at site 1, and patients aged 18–99 in a general medicine clinic with at least one health condition who had ≥2 visits/year and were registered for the patient portal at site 2. In order to test the tools in patients with active symptoms, we excluded annual wellness and preventative health visits at site 2. Consent was implied by voluntary response to the clinical pre-visit survey. We defined patients/families with “diagnostic concerns” as any participant noting ≥1 concern in any of 3 closed-ended OurDX questions including: not feeling heard, a problem or delay with tests/referrals, or any other problem or delay related to their main health concern at the visit (Supplement 1).

Chart review sample

We randomly sampled OurDX reports among those with and without diagnostic concerns from each participating clinic. We anticipated that reports submitted by patients with diagnostic concerns would be most instructive from a safety standpoint than those without concerns. Based on the number of anticipated visits and sample size calculations for our planned analyses, we targeted 1000 OurDX reports at site 1 and 500 reports at site 2. Because the total number of OurDX reports far exceeded our target at site 1 (>7000 vs 1000), we randomly sampled 30% of OurDX reports with a patient-reported concern, and also randomly sampled half this total number of reports among those that did not have a patient-reported concern. At site 2 where the total number of reports approximated the target number (500), we planned to sample all reports with patient concerns and the remainder randomly selected among those without concerns for a total of 30% of all OurDX reports. We then conducted subgroup analyses to compare results between patients with and without diagnostic concerns. As a result, we used a subset (n=450) of all (n=6079) patients/families submitting OurDX reports as our study sample for chart review and qualitative analysis of OurDX reports. Among the 450 patients, 22 (4.9%) participants had more than one visit (Table 1), and one OurDX report was randomly selected for analysis. We used a standardized data extraction form in REDcap to complete chart reviews at each site, including the number of documented chronic conditions.[33,34]

Table 1:

Patient characteristics

Patient characteristics	All participants pediatric subspecialty clinics (site 1) N=5731	Chart review site 1 N=320	All participants adult primary care clinic (site 2) N=348	Chart review site 2 N=130
Age (mean,(SD))	7.14(7.56)	7.96(8.34)	69.79(12.32)	69.92(11.50)
Gender
Female	2497(43.57%)	136(42.50%)	180(51.72%)	73(56.15%)
Male	3234(56.43%)	184(57.50%)	168(48.28%)	57(43.85%)
Race
Asian	223(3.89%)	11(3.44%)	Race other than white (combined): 6(1.72%)	2(1.54%)
Black	262(4.57%)	17(5.31%)
Other	496(8.65%)	33(10.31%)
Unknown	944(16.47%)	57(17.81%)
White	3806(66.41%)	202(63.13%)	342(98.28%)	128(98.46%)
Ethnicity
Hispanic	349(6.09%)	25(7.81%)	1(0.29%)	0
Non-Hispanic	4168(72.73%)	225(70.31%)	347(99.71%)	130(100%)
Unknown	1214(21.18%)	70(21.88%)
Preferred language
Another language	213(3.72%)	16(5.00%)	0	1(0.77%)
English	5518(96.28%)	304(95.00%)	348(100%)	129(99.23%)
Total number of submitted OurDX reports
1	4634(80.86%)	316(98.75%)	266(76.44%)	112(86.15%)
2	907(15.83%)	4(1.25%)	60(17.24%)	16(12.31%)
3	149(2.60%)	0	15(4.31%)	2(1.54%)
>/=4	41(0.72%)	0	7(2.01%)	0
Did the visit involve an active problem or new diagnosis?*		318/320 (99.37%)		124/130 (95.38%)
Proportion of patients with at least one chronic illness?*		293/320 (91.56%)		127/130 (97.69%)

Adapted from xxx (anonymized) et al JAMIA 2023

Note: Confirmation of active problem or new diagnosis and number of chronic conditions was assessed by chart review. These data are therefore not available for the full study population

Patient/family contributions

We evaluated type of patient contributions through qualitative analysis, using both an inductive and deductive approach. To develop our codebook, two physician-researchers (FB and SB) reviewed a subset (n=30) OurDX reports, using the framework for patient-reported diagnostic breakdowns² to code patient contributions. Through our review of patient reports and iterative discussion, we identified new areas of patient contributions emerging from the data and added these to the coding scheme. We then coded another 30 reports to test for any additional emerging categories, achieving thematic saturation. Through this process we finalized 10 categories of patient contributions in OurDX reports (Table 2), establishing our codebook. We focused our qualitative analysis of OurDX reports on the chart review sample. To test intercoder reliability, the two physician-researchers coded 20% of OurDX reports in the study sample from each site.

Table 2:

Types and frequencies of patient and family contributions in reviewed OurDX reports

	Total participants with actionable information(N=441)		Participants with diagnostic concerns (N=274)		Participants without diagnostic concerns(N=167)		P value
Contribution category	N	%	N	%	N	%
Access barriers	17	3.85%	16	5.84%	1	0.60%	0.006
Clinical symptoms/History	361	81.86%	213	77.74%	148	88.62%	0.004
Information about medications	106	24.04%	66	24.09%	40	23.95%	0.974
Recent visits for the same problem/concern	77	17.46%	60	21.9%	17	10.18%	0.002
Multidisciplinary clinical information	78	17.69%	54	19.71%	24	14.37%	0.154
Test/Referrals	236	53.51%	174	63.50%	62	37.13%	<0.001
Diagnosis/next steps	227	51.47%	144	52.55%	83	49.70%	0.561
Care Coordination; Confusing/conflicting information	23	5.22%	22	8.03%	1	0.60%	0.0007
Communication issue	100	22.68%	94	34.31%	6	3.59%	<0.0001
Other problem/delay	42	9.52%	42	15.33%	0	0%	<0.0001

Note: Bolded items include contribution categories more frequently reported by participants with diagnostic concerns (p<0.05)

We assessed whether the information was “actionable” for clinicians (using yes/no response categories), adapting the definition for actionable information from the National Quality Forum (NQF) metrics for diagnostic measures: “information that aid[s providers] in decision-making and management;” including “data that [assist] a provider to diagnose and treat the patient, as well as provide any needed follow-up care.”[35] Next we focused on the types of patient contributions made through OurDX. We described these using 10 categories in the codebook described above. We coded as many contributions as present in each patient report, and only considered complete agreement as a match.

We used Gwet’s Agreement Coefficient 1 (AC1) and Cohen’s kappa statistic to test inter-rater reliability. As previously described, AC1 was the most appropriate choice since some patient categories were used more frequently than others, but the kappa statistic is more conservative and more commonly used; therefore we report both.[36] We considered agreement coefficients 0.61–0.8 as good agreement and 0.81–1.00 as excellent agreement. The inter-rater reliability was excellent, with AC1 1.0, kappa 1.0 for determination of whether patient reports provided actionable information, and AC1 0.89; 95% CI [0.86, 0.92] and kappa 0.84; 95% CI [0.80 ,0.88] for categorization of patient contributions. As a result, one researcher coded the remaining reports.

Finally, in order to learn more about how OurDX might contribute to shared understanding of the DxP, we explored whether participants provided information they wanted their providers to know and/or sought answers to their own questions. One researcher coded information in each OurDX report in “you need to know” and/or “I need to know” categories, to further understand patient/family perspectives and roles in diagnosis-related information exchange.[37,38]

Clinician feedback

We surveyed participating clinicians after OurDX was in the field for 12 months. The survey addressed risks and benefits of using OurDX and recommendations to improve the tool, using items adapted from a previously published instrument.[27] Because our study was implemented during multiple Covid-19 surges, we anticipated low clinician response rates.[39] We therefore pursued additional exploratory feedback in existing staff meetings to learn more about clinician experience.

Analysis

In addition to qualitative analysis, we used descriptive statistics to report patient contributions in OurDX and clinician feedback. We used the chi squared test for independence to compare DxP contributions from participants with and without diagnostic concerns.

Ethics

The study was approved through a single Institutional Review Board process (protocol IRB-P00034869) and Data Use Agreements were established between participating organizations.

RESULTS

Study population

Among 18,129 visits in participating pediatric subspecialty clinics at site 1, 7075 (39%) OurDX reports were submitted by 5741 patients and families. At site 2, among 706 eligible adult primary care visits, 460 (65%) OurDX reports were submitted by 348 patients. In total, 6079 participants submitted 7535 OurDX reports. Among these, 682/6079 (11.2%) of unique participants reported at least one diagnostic concern: 214/6079 (3.5%) reported they did not feel heard, 360/6079 (5.9%) reported a problem or delay with tests or referrals, and 260/6079 (4.3%) reported another problem or delay related to their main concern.

Characteristics of the whole study population and the chart review participants are shown in Table 1, and were similar between these groups. Patients were predominantly white and non-Hispanic, reflective of organization-wide pre-visit survey data or other online surveys at these sites.[40] Among 450 individuals with chart review, 320 participated in the pediatric subspecialty clinics (site 1) and 130 participated in the adult primary care clinic (site 2). Among these, 92% of patients at site 1 and 98% at site 2 had ≥1 chronic conditions documented in the medical record, and >95% had visits for an active problem.

Patient contributions

The average word count in each section of OurDX was 12–18 words and similar at both sites (Supplement 2). Of 450 participants, 441 (98%) provided actionable information in OurDX. The mean number of contribution categories in OurDX reports was 2.9 (SD 1.5). Below, we detail the most common types of contributions, additional examples are listed in Table 3.

Table 3:

Examples of patient and family contributions to the diagnostic process

Category	Example of patient/family contributions
Access to care	Trying to treat the [temporomandibular joint] TMJ is impossible. Insurance will not cover most treatments being recommended because dental will not cover it but health insurance is saying that it [is a dental issue]
	Lack of resource to find feeding therapists as well as the lack of therapist that provide the treatment
	Did not go for a(n) [Magnetic Resonance Imaging] MRI today as while going to the [appointment]. my wife who is also my driver became sick [and] we had to return home.
Clinical symptoms/history	I feel off balance and dizzy and I have had a hard cough for about 2 weeks now and it does not seem to be going away. My legs and feet have been swelling up just about every day on and off for the past 3 weeks.
	Chronic cough...vomiting has slowed since starting cyproheptadine but still happens at least 1–3 times a week...[patient has had] 7 ear infections since [date]; 4 Hospitalizations.
Information about medications	I have had constant chest tightness since [date] and have been seeing multiple doctors to try to figure it out. For years, I was managing it with albuterol, [tiotropium], and [fluticasone/vilanterol] but never really saw a difference.
	[Name] complains that she can’t always hear everything. She feels like she’s missing out on things the teacher says in school as well as at home. Her ears and throat hurt quite often. We have treated with allergy medication which does not seem to work.
Recent visits for the same problem/concern	I continue to have tightness in my chest when stressed or walking uphill. I [had a cardiac catheterization] at [another medical center recently].
Multidisciplinary clinical information	My breathing is a little improved but still not good. I went to the allergist but she said... she wouldn’t recommend allergy shots... I went to the Emergency Room at [Hospital] on [date] for palpitations. The doctor there said I should ask you to set me up with a cardiac event monitor. Chest X-ray showed platelike atelectasis in the left lung base.
Test/Referrals	I believe he is overdue for a swallow study
	We have not yet received lab results indicating if the infection is [methicillin-resistant Staphylococcus Aureus] MRSA or not
	We had a referral to your clinic in 2019, but we were told to keep calling because there were no appointments open. Then, I, her mother, became ill in the fall..., COVID occurred and we are only now getting this appointment.
Diagnosis/next steps	Is there genetic testing I should have done ([for] me, [my family]?) Will this progress into worsening hearing loss? Does [Name] need hearing aids? And what is the difference between cochlear implants vs hearing aids?
	The misdiagnosis of asthma in her chart that affected her Covid [Intensive Care Unit] ICU care. They discharged her while very sick with breathing issues but said it was in line with her existing asthma. [This worried me because] they didn’t [know that she doesn’t have asthma].
	[Doctor] in [another state] missed opportunities to investigate the issue with [Ear- Nose-Throat] ENT and failed to examine the bronchial tubes during her bronchoscopy though we were told ENT would be there. Also failed to give us images or video as requested.
Care Coordination; Confusing/conflicting information	His progress seems to be delayed because we seem to have to keep jumping through hoops to get any kind of plan or guidance, despite my multiple efforts. It also appears that there are too many people involved at this time, causing this to be more of a barrier than support to us.
	No team approach with my [providers]. It shouldn’t have taken a persistent mother to start finding answers on her own.
	Our hope is that there would be a medical professional that could help us figure out [Name] as whole. If any of her conditions are overlapping. Right now, she has many doctors but not much communication between them. Each person treats their own issue but we are often left as parents trying to figure out the puzzle.
	Making sense of his hearing loss; Unclear/conflicting diagnosis.
Communication issue; not feeling heard	I am very concerned that the problems... seem to have continued for the last five years, yet his [doctor] has not taken these life altering symptoms seriously.
	I have not received the answers to questions in regards to problems with my sons hearing and choking
	Dismissive [doctor], no clear rationale or risk/benefit ratio for endoscopy.
	I think that no one has underst[ood] our situation. I'm afraid of not being clear enough.
Other	Difficult to navigate the 504 process [school support plan for child with a disability] because felt I had gotten different advice in provider visits than [the hospital gave the] school

Clinical symptoms/history

The majority (82%) of participants contributed information related to clinical symptoms and concerns (Table 2). Contributions ranged from a word or few words to descriptions with multiple details including location, duration, frequency, and severity of symptoms and/or the impact of symptoms on patients and families:

“She snores and holds her breath multiple times in the night and at naps. She is up scared and startled from holding her breath at least 15–20 times a night she has yet to sleep through the night. I am very concerned and scared for her. I myself do not sleep listening to her struggle to breathe all night.”

Some reports provided the opportunity for accelerated diagnosis/testing and subsequent treatment at or before the visit. For example, one patient wrote: “I would like my urine tested. It seems a strange a dark yellowish green color and seems to have a smell.”

About one-quarter (24%) of contributions further enhanced the history with information about medications and their respective effectiveness. Others noted treatment limitations with healthcare utilization implications; for example: “Cyclical vomiting syndrome with abdominal migraines and neurological migraines every 90 days... Typically at home medications do not work and we end up in the ER.”

In 17% of reports, medical history was also enhanced by information about recent visits for the same problem, including those at other healthcare centers. These contributions often include information the provider may not have otherwise been aware of, such as test results, medication changes, or new diagnoses (potential blindspots). For example:

“I had a right knee replacement on [date] and a second surgery on [date] to deal with an infection at the site of the replacement. This caused a stay of several days in the ICU at [hospital] that was extended to 6+ days while the infection was being cultured. I have been under the care of [doctor] who is the contagious disease specialist at the [hospital] who put me on a course of IV antibiotic for six weeks concurrently and six months on oral antibiotics.”

Patients and families who reviewed their electronic health information made contributions regarding important missing information in the medical record, such as: “Can we add the blood clotting disorder to his chart.” Some participants provided history that helped prime providers to consider tests or medication management that may have been overlooked, such as: “…Osteoporosis, for which I have taken [alendronate, teriparatide] and most recently [abaloparatide]. The last two [were] discontinued in [date]. I now do nothing other than Calcium and vitamin D. I feel I should be doing something more proactive. Is new scan warranted?”

Tests or referrals

Over half (54%) of reviewed OurDX reports involved patient comments on tests/referrals. Many patients/families sought results from completed tests such as colonoscopies, radiology studies, or other results. Others reported delays in scheduling appointments and tests, some related to Covid-19, and often months in duration. Similar to patient-reported histories, some test/referral comments highlighted potential diagnostic safety blindspots, that may be inapparent to providers. One participant wrote:

“The referral to [clinic] wasn’t sent timely by the pediatrics office. After months of waiting, I checked and they said they sent it in April. After more months of waiting, I called [clinic] to find out they ‘never received it.’”

Some participants noted concerning findings associated with delays in testing, such as: “[Patient’s] right hand is often very cold in comparison to his left. An ultrasound was recently ordered at [hospital] but it is not scheduled until July.”

Respondents pointed out clinical delays due to system issues or administrative problems, leading to inefficiencies in care: “Had to reschedule appointment due to not having the sleep study results when needed.”

Questions, concerns, or delays related to diagnosis or next steps

Half (51%) of participants included information or questions on diagnosis or next steps, such as: “What is triggering her respiratory distress? Does she have a hemangioma in her airway? Only steroids seem to get her better. What is a better option?” Another participant asked: “What determines a ‗strep carrier’? Is it dangerous to his health to have strep in his system this long? How do I know when he needs to be tested for strep when he will always test positive…? Is he contagious? Will this go away?”

Participants sought better communication about diagnosis: “No one has explained Cystic Fibrosis (CF) to me. It was minimalized while inpatient and now that the second test came back the same, we are told he is being treated as CF but does not technically have it.” Some reports reflected angst experienced by patients/families with uncertain diagnoses: “I’m just desperate to know what is happening and why, hoping for testing to be done as quickly as possible due to the high risk of choking.” Others needed “someone to take the time to solve this puzzle,” or additional context to understand the implications of a new diagnosis: “What the hearing diagnosis means for [Name], both clinically and practically? What are the next steps?” In some instances, participants reported prior diagnostic errors, seeking better care (Table 3).

Exploratory review of patient reports for “tell” vs “ask” information revealed that 91% of participants provided “you need to know” content. In addition, 61% of participants included “I need to know” content. Participants commonly sought test results, diagnoses or revised diagnoses for unexplained symptoms, or answers to questions related to clinical uncertainty (such as what to do if symptoms worsen, tests are inconclusive, or treatment isn’t working). About half (57%) of participants included both “you need to know” and “I need to know” information.

Subgroup analyses of patient contributions by diagnostic concern and site

We did not observe clinically meaningful differences in actionable information, mean number of contribution categories, or “I need to know” vs “you need to know” content between participants with or without diagnostic concerns (data not shown). The most common contribution categories did not vary by diagnostic concern or sites, and the relative proportions and directionality of differences across all categories were similar overall between the two sites, with some loss of statistical power due to smaller sample sizes in some comparisons (Supplement 3a/3b). At both sites, participants with diagnostic concerns more frequently reported access barriers, recent visits for the same problem, information on tests/referrals, coordination of care problems, communication issues, and other problems or delays (Table 2, Supplement 3a/3b). At the pediatric subspecialty clinics, patients/families with diagnostic concerns were also more likely to contribute multidisciplinary information. About one-third (34%) of patients/families with diagnostic concerns reported communication issues such as not feeling heard (Table 2).

Regarding coordination of care, respondents noted inadequate communication between providers - including those at different sites, occasional disagreement between providers, and compartmentalized care that missed “the big picture” (Table 3).

Clinician feedback

A total of 16/52 (31%) of clinicians responded to the survey. Results are described briefly, but should be viewed as exploratory (Supplement 4). Overall, all surveyed clinicians reported that OurDX is a good idea and most found patient priorities and history somewhat or very useful. The majority reported no change in time spent writing or dictating notes, the same or greater visit efficiency, and same or more meaningful time spent with patients/families during visits with OurDX. A clinician commented, “Patient contributions helped me to best understand the patient’s concerns/priorities and adjust the appointment to that.” Several noted that OurDX helped clinicians plan for the visit and involve the appropriate providers, especially when there wasn’t information from the referring physician. A few clinicians did not feel they learned anything new from OurDX reports. In staff meetings, clinicians overall supported use of the tool and were interested in approaches to further enhance patient/family participation. Table 4 summarizes clinician recommendations and future research considerations.

Table 4:

Clinician recommendations and considerations in future innovation and research

Clinician recommendations/future considerations	Examples	Potential benefit
Support patients/families to complete OurDX	Improved user interfaces Digital health navigators	Reach broader patient populations, including patients with limited English-language health literacy
Notifiy clinicians about submitted OurDX reports and integrate workflow for easy access to reports	Indication on clinic schedule of patients with completed reports[29]	Increase effectiveness of pre-visit interventions by making it easier for clinicians to know about, read, and respond to patient contributions[68–71]
	Couple notification with one-click access to patient/family reports	Decrease patient frustration resulting from clinicians who weren’t aware of, or didn’t read, their input [72]
Include an item on any prior treatment(s) and impact(s) for active symptoms	Information on prior attempted treatments can help narrow the differential diagnosis and/or bring data from other centers to the clinician’s attention (for example, in the case of second opinion visits)	Potentially improve visit efficiency and timely/accurate diagnosis
Integrate OurDX questions into existing clinical pre-surveys Avoid duplication and critically evaluate the specific goals for each survey item and overall length of survey	OurDX items can be added to clinic-specific pre-visit surveys inquiring about medications, review of systems or other clinical data	Limit number of survey requests and streamline patient work, which may already be substantial, especially for individuals living with chronic conditions[30]
Test and adapt OurDX for use by broader patient populations and/or their care partners, and in other care settings with additional testing	Further testing of OurDX with more diverse patients Translation of OurDX to other languages Opportunity for care partners to add critical information, especially if they cannot attend the visit	Expand potential use of OurDX to promote safety equity. Expand use of OurDX to urgent care or other clinical settings. Leverage the unique knowledge of care partners in the DxP
Include option for time-restricted video upload to OurDX	Patients or care partners may choose to include a video of patient function at home	Deepen clinicians’ understanding of patients’ lived experience of illness ourside the clinic visit[73–76]
Use participatory design by multiple stakeholders, and a human factors lens to better understand the interfaces and boundaries of work systems in future research of OurDX and the DxP.[77]	Each clinician may focus on their perceived “slice” of the DxP, requiring integration of stakeholder input, along with patients’ view the diagnostic journey across visits and providers.[2,31,59,77,78] The ambulatory diagnostic team has shifting membership, errors may develop over time and space, and breakdowns may occur at the “boundaries” of different work systems.	Broaden opportunities to improve human factors and engineering in the DxP to prevent missed information transfer, role ambiguity, faulty care coordination, confusion, and delay.[59,78] Patient input across the DxP may can help close gaps in provider knowledge of new tests, visits, or care plans, and identify potential boundary-spanning safety concerns.

DxP=diagnostic process

OurDX=OurDiagnosis tool

DISCUSSION

This mixed method analysis of diagnostic contributions submitted by 450 patients/families living with chronic conditions prior to primary care and subspecialty visits at 2 healthcare organizations revealed several insights. First, nearly all patient/family reports contained actionable information, even though responses ranged from a few words to detailed narratives. Due to the design of the survey (Supplement 1), even reports with few words held important information that could potentially help clinicians quickly identify DxPs at higher risk of error or delay, such as recent visits for the same problem, delays with tests or referrals, and patient experience of not feeling heard.[1,8,41]

Second, OurDX potentially added value to patients, clinicians, and the DxP in several ways. Many patients/families made important contributions to the DxP, including both imparting and seeking information from clinicians with the capacity to strengthen a shared understanding of the DxP.[42,43] Patients and families detailed information about histories, tests/referrals, diagnosis/next steps and communication issues. In some cases, participants reported diagnostic safety blindspots, such as referral delays, health insurance coverage issues, overdue tests, missing or missed test results, diagnostic delays or confusing/conflicting information.[17] OurDX reports from patients with diagnostic concerns may be especially helpful since they described several known DxP risks including access barriers, recent visits to other centers or providers, problems with tests/referrals, and coordination of care problems. Among patients/families with diagnostic concerns, roughly 1 in 5 provided “boundary spanning” information on recent visits related to the same problem with other (often multidisciplinary) providers or in other healthcare centers, that may be otherwise harder for the clinician to gather. Even in reports without diagnostic concerns, some patients/families provided information to prevent blindspots, such as notifying the clinician of a recommended cardiac event monitor after an ED visit. Some patients added critical context for the provider, integrating the DxP to date across visits and healthcare centers – especially in situations where information from the referring provider was not available.

Participant contributions also added value by enriching provider understanding of the patient experience during the DxP, including the emotional impact of long or complicated diagnostic evaluations. For example, in addition to providing usual symptom descriptors (such as location, duration, frequency), many participants commented on their fears and the impact on their lives of not only the condition but also not knowing its cause. Some participants described a sense of despair in difficult diagnostic journeys, including not feeling heard. These contributions can help providers get a more complete picture of patients’ lived experience of the DxP and important ways in which they may not “feel safe.”[44] Although such factors are central to a shared understanding of the DxP, they are not otherwise routinely elicited (Box 1).

Box 1: Views on sharing diagnostic experiences from patient participant in OurDX design[31].

“As a mother of three children, I…feel responsible for the health and wellbeing of our family in ways that are visible and invisible to providers. For our family, the process of reaching a diagnosis has on occasions been a complicated, confusing, and emotionally exhausting journey. But this experience is rarely discussed in clinic visits, nor is it documented in our notes—even though it becomes a core piece of who we are, how we move through the world, and how we interact with healthcare professionals. It also affects how we live with our diagnoses, our trust in our providers, and our willingness to engage in treatment. I have often wondered how and when to share these parts of our story.”

--Patient and family member

Co-production of the DxP through OurDX may also add value by creating a shared space to hold and address uncertainty, an important contributing factor to diagnostic error.[2,45–50] While the >90% of patient/family reports elicited “you need to know” information for providers, >60% also included “I need to know” content – such as questions pertaining to contingency planning for ongoing or worsening symptoms, confusing/conflicting test interpretation, or other aspects of clinical and diagnostic uncertainty. This was especially pronounced for parents of pediatric subspecialty patients, who at times described angst as a result of diagnostic uncertainty. By inviting patients and families to outline their priorities (framed as “What matters most to you” in the OurDX tool) several days before the visit and at a time and place most convenient for them, patients and families may have felt more comfortable reflecting on and raising such uncertainties for discussion. Participants commonly asked targeted diagnostic or management questions, which may help clinicians prepare for visits too.

Our study expands the field of shared diagnostic decision-making[50]. Other studies have surveyed patients about diagnostic concerns[19], but focused primarily on whether patients at high risk of diagnostic error thought the diagnosis was correct retrospectively. Our study is unique in that our tool was implemented with a broader patient population, invited patient/family contributions just prior to an office visit so that their input could then be incorporated into diagnostic decision-making during the office visit, and solicited both potential concerns and positive contributions to the DxP, including knowledge that may be unique to patients/families.

Finally, although limited in scope, clinician feedback suggested no major change in time demands and possibly more meaningful quality of time spent face-to-face with patients/families, similar to other pre-visit survey studies.[28,51] This may be because patients/families specified what matters most to them, all parties were more prepared, and the visit agenda was more defined. This preliminary finding is important because relationships are at the center of health care. Technologic innovations should not replace face-to-face time, but rather provide opportunities to enhance it.[52] Larger clinician studies should further pursue this inquiry.

Strengths and Limitations

Although this study included a relatively large sample of patient/family reports for in-depth review, and pediatric and adult patients in primary care and subspecialty visits, it was limited to 4 clinics in 2 U.S. academic healthcare organizations, and therefore may not reflect the views of all patients/families. Patient response rates were modest, but similar to those of other online or pre-visit surveys.[53–55] At one site, completion of the pre-visit survey relied on the patient portal, and known barriers limit registration for the portal among some sociodemographic groups.[56–58] In addition, respondents were predominantly white and English-preferring, especially at site 2. However, this site enabled study of older (mean age 70 years) and rural patients, themselves priority populations.[59,60] Studies with broader diversity are needed.[61]

We used standard EHR and portal functionality, potentially enabling broad patient reach in the future, alongside systematic efforts to engage diverse patients electronically.[62,63] Although we verified patient diagnostic concerns with clinician review in a separate analysis,[32] we did not focus on clinician verification for this study, since we were interested in characterizing the contributions made by participants who perceived concerns vs those who didn’t.

Finally, clinician response rate was limited due to testing in 4 clinics, implementation during Covid-19 surges, and potential effects of specialty;[39,64,65] but similar to other surveys.[39,66,67] Our clinician results should be considered exploratory and subject to bias, in that responses were likely skewed by clinicians with either strongly positive or strongly negative views. Studies larger than our pilot implementation are needed, perhaps with in-depth interviews to further characterize clinician experience. Our study focused on pragmatic implementation to assess the use of OurDX in real world workflows and resources, and to optimize sustainability after the study period. However, several clinician recommendations point to future innovation and additional research considerations (Table 4).

CONCLUSIONS

Soliciting contributions before an ambulatory visit provided patients/families living with chronic conditions an opportunity to help providers understand patient/family priorities, align visit agendas, and appreciate the “bigger picture” of the patient/family diagnostic journey. Patient/family input through OurDX may also help clinicians to recognize safety blindspots, identify diagnostic concerns and address uncertainty – each critical factors in co-producing diagnostic safety. While these results help operationalize the NAM mandate to engage patients and families in diagnosis through a scalable mechanism, larger studies with more diverse patient populations are needed to build on OurDX and further optimize its use for both patients and clinicians.

What is already known on this topic

Patients and families are key partners in the diagnostic process but systematic ways to elicit their unique knowledge are lacking, despite expert urging to engage them in diagnosis.

What this study adds

Patients and families living with chronic conditions who used OurDX (an online pre-visit tool eliciting priorities, history, and diagnostic concerns) provided actionable information at the point of care including patients’ lived experience of unexplained symptoms, test results or visits at other centers, communication concerns such as not feeling heard, and targeted questions about next steps. These contributions may help patients and clinicians to align priorities, address diagnostic uncertainty, and strengthen a shared understanding of the diagnostic process.

How this study might affect research, practice or policy

Routinely soliciting patient/family contributions and concerns before ambulatory visits through OurDX has the potential to systematically engage patients and families to coproduce diagnostic safety. Further study with more diverse patient populations and integration with organizational equity efforts are needed.