Challenges of Making a Diagnosis in the Outpatient Setting: A Multi-Site Survey of Primary Care Physicians

Abstract

Background

Although misdiagnosis in the outpatient setting leads to significant patient harm and wasted resources, it is not well studied. We surveyed primary care physicians (PCPs) about barriers to timely diagnosis in the outpatient setting and assessed their perceptions of diagnostic difficulty.

Methods

We conducted a survey of general internists and family physicians practicing in an integrated health system across 10 geographically dispersed states in 2005. The survey elicited information on key cognitive failures (such as in clinical knowledge or judgment) for a specific case, and solicited strategies for reducing diagnostic delays. Content analysis was used to categorize cognitive failures and strategies for improvement. We examined the extent and predictors of diagnostic difficulty, defined as reporting >5% patients difficult to diagnose.

Results

Of 1817 physicians surveyed, 1054 (58%) responded; 848 (80%) respondents primarily practiced in outpatient settings and had an assigned patient panel (inclusion sample). Inadequate knowledge (19.9%) was the most commonly reported cognitive factor. Half reported >5% of their patients were difficult to diagnose; more experienced physicians reported less diagnostic difficulty. In adjusted analyses, problems with information processing (information availability and time to review it) and the referral process, were associated with greater diagnostic difficulty. Strategies for improvement most commonly involved workload issues (panel size, non-visit tasks).

Conclusions

PCPs report a variety of reasons for diagnostic difficulties in primary care practice. In our study, knowledge gaps appear to be a prominent concern. Interventions that address these gaps as well as practice level issues such as time to process diagnostic information and better subspecialty input may reduce diagnostic difficulties in primary care.

Introduction

Missed, delayed and incorrect diagnoses (also commonly labeled as misdiagnosis) lead to significant patient harm and wasted medical resources but remain under-studied, especially in the ambulatory setting.^1-6 The diagnostic process is complex and involves multiple providers and settings over a period of time, and thus comprehensive investigation of this issue remains elusive. Prior work examining closed outpatient malpractice claims suggests that failures of clinical judgment, process breakdowns, and patient factors commonly contributed to missed and delayed diagnoses.^7,8 However, claims-based studies are not always generalizable to primary care practice.⁹

Physician perspectives are essential in enhancing understanding of missed and delayed diagnosis. For instance, a recent physician survey solicited specific cases of diagnostic errors from internists, revealing that, across multiple diseases, process breakdowns in the testing phase were commonly implicated.⁴ In another study on outpatient safety, primary care physicians (PCPs) believed that poor processes of care involving test results contributed significantly to medical errors.¹⁰ While these physician perspectives provide useful information about the types of missed and delayed diagnosis and establish the relationship of errors with process breakdowns, workflow and process issues that lead to diagnostic difficulty are not well understood. For instance, difficulty in achieving timely and correct diagnosis might relate to factors that are disease-specific, such as atypical disease presentation,¹¹ but may also result from factors related to availability and interpretation of diagnostic information, support from specialty consultants, and the PCP's work environment.

Physicians are often unaware of the diagnostic errors they make and lack systematic feedback systems to provide them with information about which of their patients experienced missed or delayed diagnosis.¹² However, they can readily report difficulties they experience in the diagnostic processes,¹⁰ and this knowledge could further advance the understanding of missed and delayed diagnosis. Therefore, our objective was to survey PCPs practicing in an integrated health system across multiple U.S. regions about potential barriers to timely diagnosis in the outpatient setting and the diagnostic difficulty they faced in their practices. By reflecting on their own specific case of missed or delayed diagnosis, physicians reported key cognitive failures (such as in clinical knowledge or judgment) that occurred in the case. We also solicited their opinions on strategies to reduce diagnostic delays.

Methods

This survey was conducted as part of an internal quality improvement process within a large, integrated health care system. The goal of this program was to characterize the vulnerabilities in the outpatient diagnostic process. In addition to the survey discussed in this report, the project included focus groups of physicians and, separately, health plan member patients, discussing the diagnostic process. The information was used by the organizational leadership to attempt to improve the outpatient diagnostic process.

Study population

The target population included salaried, adult PCPs (family medicine or internal medicine) practicing within five distinct geographic regions of a large integrated health system. The survey was done as part of an internal quality improvement initiative in 2005. The research team identified potential participants through credentialing departments, continuing medical education departments, and published physician lists. Sub-specialists were not included.

Survey development and outcome measures

Because validated measures about diagnostic difficulty and barriers to timely diagnosis are underdeveloped, we developed questions based on a comprehensive literature review, and analyses of medical-legal claims. The survey instrument was pilot-tested through cognitive interviews with 15 physicians and modified accordingly.^13-15

To determine the extent of perceived diagnostic difficulty, we used the item, “In the past year, about what percentage of your patients did you consider difficult to diagnose?” with 5 ordered responses of 0%, 1%-5%, 6%-10%, 11% to15%, and > 15%. This was our primary outcome of interest.

Respondents reported age, gender, region, and year in practice as well as years in the health system. Because we were mostly interested in PCP perspectives, we obtained information on time practiced in the outpatient vs. other (hospital, urgent care, nursing home) setting and whether participants managed an outpatient panel. The survey evaluated potential barriers to timely diagnosis related to three conceptual domains: (1) processing of diagnostic information such as from symptoms, signs and tests, (2) referral processes and (3) patient characteristics. We selected these domains because of their association with the diagnostic process and because several aspects of these domains were measurable using a PCP-focused structured survey instrument.

The survey also included two open-ended items that (1) elicited key failures in clinical knowledge or judgment that occurred in a specific case of delayed diagnosis recalled by the participant; and (2) requested suggestions to reduce delays in diagnosis, respectively. For the knowledge and judgment failures, the research team (with expertise in cognitive psychology and diagnostic error) used an established taxonomy of cognitive errors^5,16 and used content analysis to categorize the described failures into one or more cognitive errors. Some respondents did not provide sufficient information for categorization and their responses were excluded. For the suggestions to reduce delays, a trained qualitative analyst used content analysis with open coding to describe the free-text responses, and the research team categorized them into several categories of interventions targeting systems and process issues to improve the diagnostic process.

Survey administration

An independent research firm administered the final 54-item confidential survey via mail to participants in March 2005. Physicians were offered a $10 incentive to participate, and non-responders received mailed reminders 7 and 14 days after the survey was distributed. The research firm performed all of the data collection and entry and de-identified results for analysis.

Measure Testing

We performed an explanatory factor analysis to ensure a single underlying construct for each scale related to the three conceptual domains (processing of diagnostic information, referral processes and patient characteristics). We calculated Cronbach's alpha for each measure to determine reliability levels. Based on the results, items mapped to 3 underlying factors, corresponding to three domains, as follows: (1) information processing, including availability of information and time to review it (5 items, range 1-5, 1 factor, eigenvalue=1.8, Cronbach alpha=0.70); (2) the subspecialty referral process, encompassing communication between physicians as well as logistical problems and delays in referrals (7 items, range 1-5, 1 factor, eigenvalue=2.7, Cronbach alpha=0.81); (3) patient characteristics, such as non-adherence to recommended follow-up (7 items, range 1-5,1 factor, eigenvalue=2.0, Cronbach alpha=0.72). For each domain, a higher score indicates a more desirable outcome.

Analysis

We tabulated physician characteristics and the frequency with which they reported diagnostic difficulty (our primary outcome). We then investigated whether physician characteristics and their ratings of potential barriers were associated with perceived diagnostic difficulty. We created a logistic regression model for the dependent variable of diagnostic difficulty, defined by perception that > 5% of cases in the prior year were difficult to diagnose. We chose this cut-off for frequent diagnostic difficulty based on the distribution of responses in our data (Figure 1). Variables significant at the level of p=0.05 in univariate models were retained in the multivariate model, and geographic region was included as pre-specified. We suspected that those reporting more frequent diagnostic difficulty would respond differently to questions about which specific factors served as barriers to timely and accurate diagnosis. Therefore, we also assessed how the overall pattern of responses to individual items in each domain differed between those reporting more versus less frequent diagnostic difficulty. We obtained IRB approval to analyze this data for publication.

Figure 1. Diagnostic difficulty: distribution of perceived frequency of difficult-to-diagnose cases.

Results

Of 1817 surveys mailed, 1054 were completed (response rate 58%). We used American Association for Public Opinion Research response rate definition 6¹⁷. For subsequent analysis, we restricted our study sample to 848 (80%) respondents who reported primarily practicing in the outpatient setting and had a patient panel. Respondents had been in practice for a mean of 13 years, and 32% reported prior work experience in private practice/non-group-model health maintenance organization. Their mean age was 43, and 62% were male. Overall, 50% of respondents reported that >5% of their cases were difficult to diagnose (Figure 1).

Regarding specific knowledge and judgment issues contributing to missed and delayed diagnoses, less than half of the respondents to this question (198 of 453) reported data that involved any cognitive factors; others reported system and patient factors instead. We report the most 10 common cognitive factors and the 10 most common systems factors described by participants, in Table 1.

Table 1. Cognitive factors related to missed and delayed diagnosis (N=251 responses from 198 respondents).

Cognitive contributory factor	Examples*	N (%)**
Knowledge base inadequate or defective	Insufficient knowledge of relevant condition	50 (19.9)
Faulty detection or perception	Symptom, sign, or finding should be noticeable, but clinician misses it	30 (12.7)
Failure to order or follow up on appropriate test	Clinician does not use an appropriate test to confirm a diagnosis or does not take appropriate next step after test	24 (9.6)
Overestimating or underestimating usefulness or salience of a finding	Clinician is aware of symptom but either focuses too closely on it to the exclusion of others or fails to appreciate its relevance	23 (9.2)
Failure to act sooner	Delay in appropriate data-analysis activity	17 (6.8)
Misidentification of a symptom or sign	One symptom is mistaken for another	17 (6.8)
Ineffective, incomplete, or faulty history and physical examination	Failure to collect appropriate information from the initial interview and examination	16 (6.4)
Premature closure	Failure to consider other possibilities once an initial diagnosis has been reached	13 (5.2)
Distraction by other goals or issues	Other aspects of patient treatment are allowed to obscure diagnostic process for current condition	10 (4.0)
Failure to consult	Appropriate expert is not contacted	10 (4.0)
System Factors (N=201)	Examples	N (%)
Related to diagnostic tests		35 (17%)
Wait time for ordered test	Procedures not available in a timely manner	19
Wait time for results	Delay in receiving test results	14
Related to follow-up and tracking issues		39 (19%)
Physician not alerted to abnormal results	Radiologist did not call about abnormal x-ray, Unreported abnormal lab result, buried in chart	7
Physician did not follow-up or delayed follow-up with patient about results	Physician believed abnormal result notification was computerized, Physician did not follow-up on negative mammogram with patient and missed patient's continued symptoms	7
Physician missed test results	Physician missed results due to information overload (excessive emails), Did not see the results in a timely manner, Physician was out on disability leave	7
Wait time issues	Long wait for procedures and consultations	40
Communication and/or coordination issues between PCP and specialist	Lack of communication between specialist and primary care physician to complete work up.	11
Lack of appropriate action on requested consultation	The referral was sent, but the specialty department didn't schedule patient	11
General System Issues		65 (32%)
Workload issues	Lack of time with patient, lack of time review documentation at visit, high number of patients, too few PCPs, nursing staff turnover, resource issues, Reverse incentives for specialty code in HMO	31
Coordination issues	Multiple providers	10
Chart availability issues	Documentation not available at the time of the visit or the information is incorrect	8

^*Adapted from Graber et al, Archives of Internal Medicine 2005.

^**Percentages do not add to 100% because we report only the 10 most common cognitive and non-cognitive factors.

The most common cognitive issues reported were inadequate knowledge base and inadequate detection or perception of a clinical problem. Problems with interpretation of clinical information were also frequently reported, such as mis-identification of a symptom or sign or over-estimation of the reliability or saliency of a finding. Systems issues related to information availability and the negative impact of a heavy workload on the diagnostic process. Two patient-related factors emerged consistently from the responses: (1) lack of adherence to physician recommendations and diagnostic evaluation, and (2) failure to provide information about symptoms.

A variety of themes emerged in open-ended suggestions to reduce diagnostic delays, as shown in Table 2. The most common suggestions related to physician workload, which included lengthening patient visit times, reducing the size of each physician's panel, and off-loading non-clinical work to non-physicians. Respondents cited wait times for diagnostic tests, and for results to be conveyed to physicians and patients, as an area for improvement, mirroring results from other clinical settings.¹⁸ Respondents also reported that improved access to specialists would reduce diagnostic delays.

Table 2. Open-ended responses to improve diagnosis (Total responses=384).

Category	Examples	N (%).
Address workload issues (including panel size, non-clinical tasks, time constraints)	Primary doctors need help. Many “burned out.. Physicians overworked because we have too many responsibilities other than caring for patients. Too many phone messages, clerical work, and too 2 any patients per physician. Greater than half my time is spent on clerical matters or matters that a reliable intelligent teenager could perform. More time per encounter would allow a more thought full and deliberate assessment and plan.	81 (21)
Reduce wait times for diagnostic tests and results	Make tests available in a timely manner.	48 (13)
Improve access to specialist and procedures	Increase access to specialists. Increase number of specialists in order to increase access. Patients must be seen within two weeks of any referral submitted, not longer.	40 (10)
Ensure results are received and followed-up	More fail-safe systems to track what happened with abnormal results - did they get addressed? EHR will hopefully help with both. Should set up a system where abnormal test result/studies flagged + make sure there is follow-up.	37 (10)
Address communication/collaboration between providers	Better communication when scheduling with our own providers when seeing each other's patients. Improved communication between out pt MD and ER MD, so that report given by out pt MD actually gets to ER MD caring for the patient - perhaps on line or via fax - when I give report to triage nurse to give to ER MD, it often doesn't get relayed correctly. Consultants need to respond to e-referrals via e-referrals or e-mails so PCP can communicate and follow up with patients; which happens only a 5 % of the time now.	27 (7)
Address staffing issues	Reduce wait times => Hire more physicians and give them more time. More support staff RNIS,MA'S, LPN., can help with work flow to give me more time to concentrate on my patients and diagnosis.	25 (7)
Address information maintenance/accuracy/access (including chart access and resources)	Would certainly stress the need for faster transcription turnaround time for ER notes and discharge summaries. It is frustrating; time consuming and poor medicine to see a patient for follow-up hospitalization and have no idea what happened	20 (5)
Address responsibility issues related to tracking and follow-up	If a patient is seen by a consultant and that consultant feels that additional input is warranted from a 3 second consultant. then that 1st consultant should make a referral as opposed to sending patient back to PMD which may delay diagnosis 1-3 months. Avoid the policy of “PCP default” regarding abnormal test Flu.	19 (5)
Time to review results	Allow primary care provider's paid time to review growing amounts of data pertaining to patient care, labs/x-ray/test outside records.	17 (4)
Training opportunities(education -including better history taking and diagnostic decision making and policy/procedures)	Education that is case-based. Take examples such as those in Question 6 and educate MDs build diagnostic skills and thinking. Having protocols for most common missed or delayed diagnoses problems, ie, breast cancer/mast protocol in place now due to case here five years ago.	12 (3)
Improve computer system (design and configuration)	In changing to EMR, has been more difficult to follow lab results. Configuration of reports is harder to read. Hard to get full picture of patient with having to look up each visit on the computer separately. Time consuming us flipping through chart. Tests, labs, radiology notes all in separate areas of computer!!	11 (3)
Address patient factors (education, communication, outreach)	Encourage patient for better follow-up By far biggest cause of delay in diagnosis is patients' reluctance to come in, usually out of denial and fear. We need to encourage patients not to delay, especially regarding skin and breast lesions.	11 (3)
Minimize denial/refusal/cancellation of consults	Specialty consultations should accommodate appoints especially if cancer is being considered without having to resort to admitting the patient in the hospital. It is unbelievable that the specialists “close” a referral when a patient hasn't called within a few weeks rather than phoning that patient by the department. Also, when a patient does call a days after “closed” referral, they are denied an appointment. Specialty referrals should not be denied.	9 (2)
Curbside Consults	Encourage collaboration between primary care and specialists. On call Specialist carries a cell phone for “curbside” consults. All specialties need to have an on-call phone/pager to easily access a quick consult - some of the specialties are good, some are terrible and essentially unavailable.	8 (2)
Address cross-coverage policy and procedures	When a provider is gone, often the labs and test results are not given to the relieving doctor. They just wait for the provider to get back. Covering physicians should be just as responsible as physician responsible for test.	5 (1)
Miscellaneous comments (unable to classify)	Good to do periodic surveys like this. If implementable changes Reward the MDs for accomplishing high quality care and service per unit	14 (4)

As shown in Table 3, the only physician characteristic associated with reporting more frequent diagnostic difficulty was time since completion of residency (those with more experience reported less frequent diagnostic difficulty). In multivariate analyses, diagnostic difficulty was associated with worse ratings of information processing and more problems with the referral process.

Table 3. Factors associated with Higher Frequency of Diagnostic Difficulty.

	Diagnostic Difficulty OR(CI)*
Gender	1.09 (0.82-1.43)
Formal Consultation
1 to 2 times per week	0.91 (0.39-2.09)
3 to 4 times per week	0.80 (0.35-1.84)
5 or more times per week	0.75 (0.33-1.70)
Experience in private practice/non-group model HMO	0.85 (0.63-1.13)
Fewer years since residency	1.04 (1.02-1.06)
Worse information processing	1.29 (1.04-1.60)
Referral process problems	1.43 (1.16-1.78)
Patient-related barriers	1.07(0.89-1.28)

^*Odds ratio for each 1-unit change, adjusted for all listed factors and geographic region.

Within the domain of information processing, those who reported a higher frequency of diagnostic difficulty reported that clinically relevant information was not always available. In addition, they reported more frequent concerns about the adequacy of time to integrate clinical information and to think carefully about diagnostic issues (Table 4).

Table 4. Items associated with reporting more diagnostic difficulty.

Information Processing Item

Incomplete, missing, or illegible documentation*

Test result tracking system that fails to let me know when test results are available

Not receiving ED discharge summaries or consult notes when my patients have been seen in the ED or by another service

Insufficient time for me to integrate the volume of clinical information (e.g., labs, test results) I receive*

Insufficient time for me to think carefully about diagnostic issues*

Referral Scale Item

Specialists return referrals with requests for additional information*

Test results or diagnostic information but no recommendation for next steps from the specialist who has done the consult*

Unclear understanding between specialists and me concerning who is responsible for informing and treating patients when abnormal test results are reported*

Unclear understanding when a specialist should refer a patient to another specialist rather than back to me*

Long wait times for specialty care consult appointments*

Specialists deny referrals*

Specialists not responding to my calls or e-mails in a timely manner*

Responses: 1=All of the time, 2=Most of the time, 3=Some of the time, 4=A little of the time, 5=None of the time

^*P for difference in response for those reporting more frequent vs. less frequent diagnostic difficulty<0.01

In terms of the referral process, those reporting diagnostic difficulty reported less subspecialty access, inadequate communication between PCPs and subspecialists, and insufficient guidance from subspecialists back to the referring provider (Table 4). In addition, those reporting difficulty were significantly more likely to report an unclear understanding between specialists and themselves regarding who is responsible for informing and treating patients when abnormal test results are reported (Table 4).

Conclusions

We report findings from a multi-center survey conducted to support quality improvement efforts to address ambulatory diagnostic problems faced by primary care physicians. We elicited several cognitive factors PCPs believed to be associated with diagnostic delays, as well as strategies for improvement. We found that half of respondents reporting that more than 5% of their cases were difficult to diagnose. Though this is not a high proportion for any individual clinician, it represents many difficult diagnoses overall. In our analysis, diagnostic difficulty appears to correlate with inadequate time to process diagnostic information and insufficient guidance from subspecialists.

Our study provides rich data regarding processes of care that make diagnosis vulnerable to errors in the primary care setting. Cognitive factors including inadequate knowledge base and faulty perception/detection were noted, but even when specifically asked about cognitive factors, respondents referenced system and patient factors, underscoring their importance. The most frequent cognitive factor reported was related to inadequate knowledge, in contrast to a previous study where knowledge problems were only infrequently related to diagnostic errors¹⁶. This was followed by faulty detection or perception of diagnostic information (such as from history/exam/record review), a factor also prominent in a recent survey of pediatricians¹⁹. In addition to improving information synthesis and reducing errors from biases and heuristics, efforts to reduce diagnostic errors should also focus on these two important factors. As with prior studies^4,7, lack of timely information review was commonly cited as a contributor to missed and delayed diagnosis.

We also report specific improvement strategies that primary care physicians believe will improve missed and delayed diagnosis, for which empiric literature is generally lacking. Respondents' strategies for improvement often centered on physician workload; they identified several distinct links between physician workload and delays in diagnosis. First, they perceived longer visit time with patients, which may allow for more thorough history and physical examination, as important to the diagnostic process. Second, they specifically cited reducing workload in order to have protected time to review patient results, to ensure follow-up of clinical abnormalities. Other recent studies also demonstrate this to be a high-risk area despite improved information availability.^20,21 For instance, studies reveal sub-optimal follow-up of abnormal test results, even in the presence of electronic health record-based clinician notification, suggesting that that missed or delayed diagnoses are not always related to lack of access to information. Other system and cognitive factors including time and workload are likely in play and need to be targeted for improvement.

In addition, participants suggested more time for physicians to think carefully about difficult cases, and to perform “cognitive work”.²² Moreover, they cited the burden associated with non-visit-related tasks. These tasks, such as following up patient test results, are increasingly becoming recognized as a significant contributor to physician workload,²³ and will require specific, workflow targeted efforts to address. It is possible that the patient-centered medical home model, ²⁴ with panel-based rather than visit-based reimbursement, ²⁵ will allow for clinicians to allocate their time in a way that better supports timely, correct diagnoses. As an example, in the integrated health system in which this study took place, assessment of PCP performance is shifting to quality, service, and access metrics across a PCP's patient population, rather than the measurement of the number of visits per day provided by the PCP.

Despite the presence of an integrated health care system where subspecialists were readily accessible, we found that the referral process is perceived as a barrier to timely and complete diagnosis. The PCPs in this study who reported diagnostic difficulty acknowledged problems accessing and communicating with subspecialists. This problem is likely be worse in individual practices, where PCPs interact with subspecialists from several other health care systems using many types of communication methods.²⁶ In addition, the issue of shared responsibility for a patient between a subspecialist and a PCP merits further inquiry. For instance, there is no clear consensus among clinicians about who should refer patients to a second specialist if the first specialty referral does not result in a diagnosis and/or treatment plan.²⁷ Similarly, physician perceptions vary about who should inform patients about abnormal test results; studies suggest that usual clinic practice leaves many patients uninformed.¹⁸ Some degree of standardization of referral expectations and practices is likely to enhance the diagnostic process.²⁶

Our study has several limitations. First, although conducted across five distinct U.S. geographic regions with significant variability in practice settings, it was originally designed as a quality improvement project within a single health system with salaried, staff physicians and thus, its generalizability might be limited. For instance, it might not be representative of communication between PCPs and subspecialists in U.S. ambulatory health care in general where per-consultation financial incentives for subspecialists are different.²⁸ Second, our study was performed in 2005. Findings remain relevant despite the age of the data because: (a, cognitive factors described here are unlikely to have substantively altered; (b) system factors such as short visit length and problems processing information from multiple inputs are likely to persist. However, this study was conducted prior to comprehensive electronic health record (EHR) implementation at this health system. Barriers we found associated with perceived difficulty might not necessarily be mitigated by EHRs²⁹ and most U.S. primary care practices do not currently have a comprehensive EHR. Third, this study was conducted prior to many of the landmark studies on diagnostic error and diagnostic processes; thus, the phrasing of the items might not at times follow taxonomies laid out in these papers^4,7,16,30. However, the issues described in the survey are interpretable, have been described in the literature, and emanate from primary care workflow. Fourth, the patient population cared for by the PCPs, although demographically diverse,³¹ represents a group with health insurance, mostly provided by employers. As such, patients in this system may have fewer barriers than in other ambulatory health care settings. Fifth, our results reflect only the missed diagnoses that providers are aware of. There are likely other missed diagnosis that remain unknown to primary care providers, and our study cannot shed light on this type of missed diagnosis. Finally, as with all survey research, subjects who chose to respond to the survey are likely to differ from those who do not. We did not have sufficient demographic information on non-responders to make inferences about how non-response affected the reported results.

In summary, a significant proportion of primary care physicians report diagnostic difficulty involving at least 5% of their patients. Misdiagnosis relates to several system and cognitive factors but knowledge gaps might be more often responsible than previously estimated. Interventions that address practice level issues such as time to process diagnostic information and better subspecialty input might potentially reduce diagnostic safety concerns in primary care.