Skip to main content
PMC home page
PLOS ONE logoLink to PLOS ONE
. 2021 Sep 27;16(9):e0257677. doi: 10.1371/journal.pone.0257677

Computerized history-taking improves data quality for clinical decision-making—Comparison of EHR and computer-acquired history data in patients with chest pain

David Zakim 1,*,#, Helge Brandberg 2,#, Sami El Amrani 1,‡, Andreas Hultgren 1,‡, Natalia Stathakarou 1,‡, Sokratis Nifakos 1,‡, Thomas Kahan 2,#, Jonas Spaak 2,#, Sabine Koch 1,#, Carl Johan Sundberg 1,3,#
Editor: Amit Bahl4
PMCID: PMC8476015  PMID: 34570811

Abstract

Patients’ medical histories are the salient dataset for diagnosis. Prior work shows consistently, however, that medical history-taking by physicians generally is incomplete and not accurate. Such findings suggest that methods to improve the completeness and accuracy of medical history data could have clinical value. We address this issue with expert system software to enable automated history-taking by computers interacting directly with patients, i.e. computerized history-taking (CHT). Here we compare the completeness and accuracy of medical history data collected and recorded by physicians in electronic health records (EHR) with data collected by CHT for patients presenting to an emergency room with acute chest pain. Physician history-taking and CHT occurred at the same ED visit for all patients. CHT almost always preceded examination by a physician. Data fields analyzed were relevant to the differential diagnosis of chest pain and comprised information obtainable only by interviewing patients. Measures of data quality were completeness and consistency of negative and positive findings in EHR as compared with CHT datasets. Data significant for the differential of chest pain was missing randomly in all EHRs across all data items analyzed so that the dimensionality of EHR data was limited. CHT files were near complete for all data elements reviewed. Separate from the incompleteness of EHR data, there were frequent factual inconsistencies between EHR and CHT data across all data elements. EHR data did not contain representations of symptoms that were consistent with those reported by patients during CHT.

Trial registration: This study is registered at https://www.clinicaltrials.gov (unique identifier: NCT03439449).

Introduction

Medical histories are the salient datasets for characterizing clinical states and informing diagnostic and treatment decisions [1–5]. Whenever studied, however, medical history data recorded by physicians is found consistently to be incomplete, inaccurate, biased, and not always fact-based [6–16]. These inadequacies may reflect, in part, poor documentation of what the physician knows about the patient. But real-time observations reveal too that there are deficiencies in the history-taking process as such [17, 18]. And since inadequate history data can lead to diagnostic errors [1, 19–21], it appears that we need better methods for history-taking than physicians interviewing patients.

The key barriers to effective history-taking by physicians are insufficient time with patients and the enormity of the knowledge for history-taking [22, 23]. It was realized even at the beginning of the computer-age that expert system software could solve these problems [24, 25]. Yet today, there is scant academic interest in developing expert systems for computerized history taking (CHT), i.e. automated, dynamic history taking as a patient interacts with a computer [26]. We bring attention to the possible value of CHT in the present work. We compare medical history data collected and entered into EHRs by physicians with history data collected by CHT at the same ED visit from 410 patients presenting to an emergency department (ED) with acute chest pain.

Materials and methods

Experimental design

The results we report are from a within person study in which the experimental intervention was deployment of two different methods for collecting and storing medical history data. The first of these was physicians providing routine ED care and entering their findings into an electronic health record (EHR). The second was expert system software trained to interview patients with acute chest pain, i.e. CHT. Data entries were stored automatically by the CHT program. Every enrolled patient was interviewed by a physician and the expert system software during the same ED visit. Data collected by CHT was not available to ED staff.

Study setting and recruitment of patients

All patients in this study presented to the ED of Danderyd University Hospital, Stockholm, Sweden with a recorded chief complaint of "chest pain" at entry to the ED. Patients with chest pain as presenting complaint were recruited consecutively if they were 18 years or older, had a non-diagnostic ECG for acute myocardial infarction or angina, had native fluency in Swedish or English, and had an equivalent Manchester triage score of 3, 4 or 5. Some eligible patients were excluded because they arrived at the ED without eyeglasses and were unable to read the screen of a tablet computing device. There were no other criteria for inclusion or exclusion. Patients were recruited between October 2017 and November 2018. Participation was skewed toward male patients (57% of 410 patients) as compared with a distribution of 50.4% male patients in the Danderyd ED population of patients with chest pain [27]. The age distributions of male and female patients, categorized by cut points in the HEART score [28], were essentially identical (Table 1). Swedish law does not allow questioning patients about ethnic origin.

Table 1. Gender and age distributions of enrolled patients.

Gender Male Female
Number enrolled 236 174
Fraction ages 18–44 0.311 0.302
Fraction ages 45–64 0.408 0.409
Fraction age ≥ 65 0.281 0.298
Open in a new tab

Patients were recruited by term 9 Karolinska Institutet medical students (SEA and AH), who were familiar with the CLEOS program and the purposes of the research plan. SEA and AH explained to patients the purpose of the research program and demonstrated how to interact with the CHT program running on an iPad® (Apple Inc, Cupertino, CA, USA). Patients were told that agreeing or declining to participate would not affect care, would not affect wait time in the emergency department, would not prolong the duration of their stay in the emergency department, and would not affect eventual discharge to home or admission. Patients were informed they were free to end the computerized interview at their discretion and that the computerized interview would end at the time a decision was made to discharge or admit whether or not the computerized interview was complete. Patients were informed that the data collected by the computerized interview would be unavailable to their providers of care in the ED and that the data collected would be used exclusively for clinical research. Patients also were informed that the identifiers of the data collected by the CHT program were stored in a locked safe at Danderyd University Hospital and that the only people with access to the safe would be Karolinska Institute faculty participating in the research program. SEA and AH revisited participating patients during CHT sessions to insure patients were able to navigate the program on the iPAD.

CHT software

The CHT software (CLEOS) is described in detail elsewhere [23, 26, 29]. In brief, CLEOS is a dynamic expert system for history-taking from adults with acute and long-term complications from chronic disease. The knowledge base emulates the thinking of expert physicians to pose questions that are relevant to the program’s working differential diagnosis, which is formulated first from the patient’s chief complaint and demographic profile. Data collected is interpreted continuously in the context of applicable pathophysiology to direct questioning to resolving the working differential diagnosis. Review of systems data is collected by the same scheme. The knowledge base of questions and answers is developed in English comprehensible to a native English-speaker with a U.S. sixth grade education. Textual questions have answers sets of Yes/No, Yes/No/Uncertain, or multiple choice answers. Questions are graphic whenever this is possible, as for example when the site of primary or radiated pain is queried. Each answer is coded to indicate the question asked and the answer entered. Patients’ answers are stored as codes. The English language version of CLEOS was translated to Swedish by professional translators experienced with technical translation. The Swedish translation was tested for clinical accuracy by Karolinska Institute physician and nursing staffs for comprehension by a representative, demographic of the Swedish population. Problematic language was corrected before deployment in the work described.

This study used a version of CLEOS with 17,500 decision nodes, more than 10,000 pages of queries and several thousand rules that continuously determine the next most appropriate question, patient-by-patient, by interpreting the clinical significance of data as it is collected. The CHT interview ran on a server at Karolinska Institutet connected via VPN to the ED. CLEOS is owned fully by Karolinska Institutet, a public university. The standard CLEOS interview, which begins by querying the chief complaint, was modified for the present work. Instead of asking for the chief complaint, patients were asked by the question in Fig 1 to confirm or deny that chest pain was their chief complaint. All patients were interviewed by the Swedish language version of CLEOS.

Fig 1. Image presented to patients to confirm that chest pain was the chief complaint.

Fig 1

Open in a new tab

Patients were asked to confirm with a "yes" or "no" that they were seeking medical care because of pain somewhere in the blue region of the image.

Time-relationship between physician history-taking and CHT interview

Patients interacted with the CHT software during wait times. CHT was interrupted at patient request or for care. Almost all patients began CHT prior to examination by their ED physician. We have exact time-stamps for the start of a CHT session and for interruptions during a session. There was no reliable data for the exact time of physician examinations, however. Ten of the 410 patients for whom data is reviewed arrived at the ED in early hours of the morning when medical students were not present to recruit patients. These 10 patients were examined by a physician prior to the start of a CHT interview. Search of time stamps in CHT records revealed that 20 of 400 patients arriving in the ED during hours of active recruitment had at least one interruption of 10 min. or longer. A first history-taking session by a physician might enhance patient recall of information during a subsequent CHT session and might have occurred or began during a CHT interview with a long delay between answers for successive questions. We thus compared data in EHRs and CHT files for those CHTs that began after a physician’s examination (10 patients) and those with a greater than 10 min interruption (20 patients). Differences between EHR and CHT data for these 30 patients were not different qualitatively or quantitatively from comparisons across the other 380 patients. Data analyses are presented for a single group of 410 patients.

Data extraction

EHR records corresponding to personal national identity numbers were accessed at the Danderyd University Hospital ED. Personal national identity numbers were not copied or stored. The date of signed patient consent was used to verify that EHR and CHT data were for the same ED visit. Data elements analyzed in EHR and CHT files were clinical attributes relevant to the differential diagnosis of chest pain and known only to the patient, e.g., site(s) of chest pain, time of onset, setting for onset, frequency of episodic pain, constancy of pain, and so on. Data was extracted from EHRs by SEA and AH. Extracted datasets were identified only by an accession number corresponding to a CLEOS file. CLEOS codes representing specific answers to specific questions were extracted by DZ, who had no knowledge of patient identifiers. The chi-square statistic for categorical variables was used to determine the significance of differences between data elements in EHR and CHT data sets. To calculate X21, the number of patients in EHR data with a given finding was the observed value; the number in CHT data with the given finding was the expected value. P-values for level of significance were determined for X2 for 1 degree of freedom.

Exclusion of selected data sets

We found that repetitive tapping by patients on the button sending answers from iPAD to server and requesting a next question corrupted the pathway of knowledge graphs. The problem was corrected during the course of this study. Interviews corrupted in this way were excluded from analysis. We excluded data for four patients, who did not confirm chest pain as chief complaint. We excluded data for two additional patients, who appeared to make deliberately false entries during CHT. The evidence for this was large discrepancies between year of birth in the CHT record (selected from a drop-down menu) and age recorded in the patient’s EHR, apparent indiscriminate selection of all answers for sites of pain and sites of radiated pain and answers to questions that were not consistent with each other. The remaining 410 patients were included in the study.

The protocol was approved by the Stockholm Regional Ethical Committee (now Swedish Ethical Review Authority) (No 2015/1955-31).

Results

Data for location of primary pain

Of the 410 EHRs, 213 had no entry for a specific site of chest pain or an entry too imprecise to assign pain to an anatomic location in the chest, shoulders or epigastrium. All 410 CHT records had a primary site of pain entered by patients interacting with a version of Fig 2 without numbering of the anatomic areas.

Fig 2. Image used by patients to enter sites of chest pain.

Fig 2

Open in a new tab

Tapping on any region of the image added blue color to it. The patient could deselect a blue area with a second tap. The areas in the image presented to patients were not numbered. We show numbers here because they indicate the areas of pain in S1 Fig.

S1 Fig displays EHR and CHT data for location(s) of primary pain for all 410 patients. Each line in S1 Fig shows locations of pain for a single patient with EHR findings in the upper triangle and CHT findings in the lower triangle of each numbered rectangle. S1 Fig shows that CHT datasets had more precise locations of pain and a far greater heterogeneity of pain patterns. S2 Fig displays examples of EHR and CHT data for locations of pain projected onto the chest.

We analyzed EHRs and CHTs specifically for entries for central chest pain (Table 2). Congruence of EHR and CHT data for central chest pain was limited. Of 256 patients with central pain during CHT, EHR data for pain location was missing in 120. CHT reports for the presence and absence of central pain did not confirm frequent EHR entries for the presence of central pain or the absence of central pain.

Table 2. Findings for central chest pain in EHR and CHT records.

Site of Chest Pain EHR Data Patients CHT Data Patients p-Value EHR vs CHT Congruence EHR and CHT Findings Patients (%)
Central Chest Pain 94 256 < 0.001 77 (30.1)
Central Chest Pain Exclusively 88 62 < 0.001 30 (48.4)
No Central Chest Pain 92 154 < 0.001 41 (26.6)
Open in a new tab

EHR and CHT data was reviewed for 410 patients. An EHR and CHT record was available for every patient. Congruence between EHR and CHT findings is the instances in which the same patient was identified by EHR and CHT data. P-values are from a X2 distribution table for one degree of freedom and X2 calculated as in Methods. The denominator for percent congruence was the number of patients identified by CHT data.

We searched for evidence that discrepancies between EHRs and CHTs for locations of pain reflected inaccurate recall of pain that remitted. But discrepancies between EHR and CHT data for pain location were equally frequent for patients with or without pain during CHT. CHT data showed too that pain patterns were stable over time. Patients with a first onset of pain days to weeks before presentation and negative self-reported histories for angina, myocardial infarction and revascularization reported that sites of pain did not change in the intervals between a first occurrence and the presenting event. CHT data indicated that pain at presentation was the same as in prior episodes of angina or infarction for all but one patient with a CHT reported history of angina or infarction.

Location of radiated pain

Positive or negative entries for radiated pain were missing in 224 of 410 EHRs. An entry for radiated pain was missing in 2 of 410 CHT files because of early termination of the interview. CHT data for radiated pain was missing for 6 patients with no entries of anterior chest pain. This occurred because CLEOS was programmed in error to not ask about radiated pain in the absence of anterior chest pain. We searched EHR and CHT data specifically for patterns of radiated pain relevant to the differential of chest pain (Table 3) [28]. CHT as compared with EHR data identified significantly more patients with relevant radiation patterns (Table 3). CHT data did not confirm most positive EHR entries for these patterns of pain. And as noted in Table 3, 80% of patients with jaw/neck radiation in EHR data denied this finding during CHT. We also found that 50% of explicitly negative EHR entries for radiation to the neck/jaw were for patients reporting radiation of pain to the jaw and/or neck during CHT.

Table 3. Patterns of radiated pain.

Site Radiated Pain EHR Data Patients CHT Data Patients p-Value EHR vs CHT Congruence EHR and CHT Data Patients (%)
R + L shoulder or R + L arm 8 63 < 0.001 3 (4.8%)
R shoulder, arm or hand; no left-sided radiation 1 24 < 0.001 1 (4.1%)
    Jaw and/or neck 30 48 < 0.001 6 (12.%)
Open in a new tab

EHR and CHT data was reviewed for 410 patients. Patients with radiation to R and L shoulders and radiated pain to an arm are not identified separately. P-values are from a X2 distribution table for one degree of freedom and X2 calculated as in Methods. The denominator for percent congruence was the number of patients identified in CHT files.

Setting for the onset of pain

Table 4 summarizes findings for onset of pain during physical exercise or emotional upset. EHR data was absent or not interpretable for the setting of onset of pain for 30 of 44 patients with CHT data for onset of pain with exercise. Eighty percent of patients identified in EHR data with exercise-induced onset of pain denied during CHT that this was a correct representation of the setting in which their pain began. EHR data was even less consistent with CHT data for patients reporting onset of pain during emotional upset. Premature termination of CHT led to missing CHT data for one patient with EHR data for onset of pain during physical activity.

Table 4. EHR and CHT data for settings for onset of pain.

Setting for Onset of Pain EHR Data Patients CHT Data Patients p-Value EHR vs CHT Congruence EHR and CHT Data Patients (%)
Physical activity 68 44 < 0.001 14 (31.8)
Not physical activity 153 360 < 0.001 142 (39.4)
Emotional upset 7 36 < 0.001 6 (16.0)
Not physical activity; not emotional upset 3 324 < 0.001 3 (< 1)
Open in a new tab

EHR and CHT data was reviewed for 410 patients.

P-values are from a X2 distribution table for one degree of freedom and X2 calculated as in Methods. The denominator for percent congruence was the number of patients identified in CHT files.

Frequency and duration of pain

EHR data was missing for 81 patients reporting two or more episodes of pain in the 24 hours before presentation, during CHT; 8 of 53 patients cited in EHRs with two or more episodes of pain reported constant pain during CHT (Table 5). There was no significant difference for EHR and CHT data for < 2 episodes of pain. But lack of congruence between EHR and CHT data for < 2 episodes of pain was significant with a p-value of <0.001, Data for the frequency of pain was missing in 7 CHTs that ended prematurely for uncertain reasons. CHT data for frequency of pain was missing in one patient with epigastric and right posterior chest pain at the level of T7-12. The CHT program excluded acute coronary syndrome from the working differential for this patient.

Table 5. Frequency and duration of pain.

Attribute EHR Data Patients CHT Data Patients p-Value EHR vs CHT Congruence EHR and CHT Data Patients (%)
2 or More Episodes of pain in prior 24h 53 123 < 0.001 28 (22.7)
Less than 2 Episodes of Pain in prior 24h 106 102 NS 27 (26.5)
Constant Pain from Onset or Pain Became Constant 47 119 < 0.001 24 (20.1)
Open in a new tab

P-values are from a X2 distribution table for one degree of freedom and X2 calculated as in Methods. NS means not significant. The denominator for percent congruence was the number of patients identified by CHT data.

EHR data was missing for 95 of 119 patients with CHT findings of constant pain since onset or pain that became constant. Time domain data was missing in EHRs for 22 patients reporting constant pain for 12–24 hrs during CHT and for 49 patients reporting constant pain for > 24 hrs during CHT.

Data for associated symptoms

EHR data was missing for a significant proportion of patients reporting sweating during CHT; and there was poor congruence between EHRs and CHTs for explicitly positive and negative findings for sweating with chest pain (Table 6). There was no significant difference between EHR and CHT data for the number of patients with no sweating.

Table 6. Associated symptoms of sweating and pleuritic pain.

Attribute EHR Data Patients CHT Data Patients p-Value EHR vs CHT Congruence EHR and CHT Data Patients (%)
Sweating by history 23 106 < 0.001 14 (13%)
Negative sweating by history 243 263 NS 187 (71%)
Pain worse with breathing 64 60 NS 35 (58%)
Pain only with inspiration 0 9 < 0.001 0
Open in a new tab

EHR data for patients with pain caused by inspiration did not specify relief by breath-holding. CHT data for pain only on inspiration means pain was absent during breath-holding. CHT data for pain only on inspiration is included in patients with pain exacerbated by breathing. Ten percent of CHT interviews ended before associated symptoms were queried. P-values are from a X2 distribution table for one degree of freedom and X2 calculated as in Methods. NS means not significant. The denominator for percent congruence was the numbers of patients identified by CHT data.

But the lack of congruence between EHRs and CHTs for absence of sweating was significant at p <0.001. The lack of congruence between EHR and CHT data for pain exacerbated by respiration also was significant (p < 0.001). EHRs cited that some patients had pain with breathing and others had pain with inspiration. CLEOS asked patients with a pleuritic component of pain whether pain was exacerbated by breathing or caused by inspiration and absent during breath-holding. CHT data identified 9 patients with pain relieved by breath- holding. None of these patients was identified by EHR data. Nine patients cited in EHRs with pain during inspiration had a pleural component of pain in CHT data but indicated that breathing exacerbated pain that was not relieved by breath-holding.

Use of nitroglycerin

CHT interview identified 26 patients, who self-administered nitroglycerin prior to care in an ambulance or ED (Table 7). EHRs indicated that 6 of these patients were treated with nitroglycerin without specifying self-administration. Two patients reported during CHT that pain relief occurred more than 15 min after self-administration of nitroglycerin. These patients are included in Table 7 in the set without complete relief of pain.

Table 7. Use and effect of self-administered nitroglycerin.

Attribute CHT Data Patients EHR Data Patients p-Value EHR vs CHT Congruence EHR and CHT Data Patients (%)
Self-administered nitroglycerin 26 16 0.05 NA
Complete pain relief by nitroglycerin 10 6 0.2 6 (60%)
No pain relief after nitroglycerin 16 10 0.2 8 (50%)
Open in a new tab

EHR and CHT data was reviewed for 410 patients. Data is keyed to the CHT finding of self-administered nitroglycerin prior to medical care. EHRs were searched for an EHR entry of nitroglycerin use only for patients with a CHT entry for self-administered nitroglycerin. Patients in the row Self-administered nitro-glycerin and column EHR Data are the number of EHRs with entries for use of nitroglycerin. CHT data indicated that 2 patients had complete pain-relief only between 15 and 30 minutes after self-administration of nitroglycerin. These patients are listed as no relief from nitroglycerin. NA indicates not applicable. P-values are from a X2 distribution table for one degree of freedom and X2 calculated as in Methods.

Dimensionality of EHR and CHT datasets

We quantified the dimensionality of EHR data by searching EHRs for those with entries for a site of primary pain, a positive or negative finding for radiation to jaw or neck, a positive or negative finding for onset during physical activity, and a positive or negative finding for use of nitroglycerin. We included the criteria of a positive or negative finding because our data shows that missing EHR data may not reflect omission of negative findings. We found only 3 of 410 EHRs with explicit entries for the 4 attributes searched. Reducing the dimensions searched to any set of 3 of the above 4 expanded the number of complete EHRs only from 3 to 16. Four hundred nine CHT datasets included a positive or negative for all 4 attributes in the dimensionality analysis; 90% of CHT datasets had explicit positive or negative findings for all the data elements analyzed in all Tables and Figures in the preceding sections; and missing CHT data was related predictably to the question at which a CHT session was aborted.

Discussion

We made a patient-by-patient comparison of EHR and CHT files for a set of data elements significant for the differential diagnosis of chest pain that are available only by history-taking from affected patients. We found, as compared with CHT files, that significant amounts of data were missing in EHRs; missing EHR data was not limited to absence of negative findings; and missing EHR data was distributed randomly across all EHRs and all data elements, which limited severely the dimensionality of EHR data. We also found frequent discrepancies of fact between EHR and CHT data for specific positive and negative findings. These findings are not unique to the environment in which our study was conducted or to the clinical problem patients presented; for histories in paper and EHR charts are known generally to have the same deficiencies as our analysis found [6–18]. There is, however, an important difference between the present and most prior analyses of data quality in physicians’ histories. Thus, our work shows that CHT can address these issues in the context of a specific clinical problem.

The advantages of CHT for history-taking are general

CHT has multiple, general advantages for history-taking as compared with physicians (Table 8). Computers running a CHT program have unlimited cognitive capacity and short-term memory [30]. Computers interacting with patients are not time-constrained and will not limit patient response times to seconds [16]. Expert system software is not programmed to use heuristics to collect and process history data [31, 32]. And, as compared with knowledge- and time-constrained physicians at the “bedside,” expert system software for CHT is developed by panels of clinical experts working deliberatively, without time constraints, using System 2 thinking exclusively [33], and with immediate access to each other and the literature. The clinical experts formalizing their medical knowledge in the form of an expert CHT program also can test and edit their work because CHT leaves an exact record of how data was collected and what might have been missed in any interview. It appears, therefore, that the advantages of CHT technology for acquiring history data directly from patients can be adapted to address a large range of clinical problems [9, 10, 24–26].

Table 8. Comparison of physician and CHT attributes affecting the completeness, accuracy and bias of data collected by each method for history-taking.

Attribute Physician CHT
Limited short term memory Yes No
Non-expandable cognitive capacity for learning impacted negatively by time-pressure and distraction Yes No
Use heuristics to collect and interpret data Yes No
Limited time with patient Yes No
Limit patient’s answer-times to seconds Yes No
Leverage expertise and scale resources to need No Yes
Complete control of what is asked and what is not asked to standardize data collection No Yes
Automated data-entry into a structured, electronic database No Yes
History-taking in the patient’s preferred language Maybe Always
Open in a new tab

Indeed, review of the CHT data in the current work identified an error in the CHT knowledge base, e.g., patients with pain restricted to the posterior chest were not asked about radiation. EHR records are too noisy to allow detection of such discrete errors (6–21), not to mention the near impossibility of detecting and correcting such errors in the midst of history-taking by physicians. Still further advantages of CHT are standardization of history-taking by a single controlled protocol, leveraging of expert knowledge, and scaling of resources. CHT technology also is a mechanism for interviewing patients in their preferred languages while presenting findings and decision support in physicians’ preferred languages. Of course, the value of CHT will have to be established on a problem-by-problem basis across the scope of medical practice. It will have to be shown too that CHT is acceptable to patients and that the reporting of CHT findings is acceptable to physicians. Considerably more academic, clinical research appears needed therefore for developing and testing CHT software.

Limitations of the present study

Data are from a single institution

EHR data was extracted from records at a single hospital. Our results for physician histories may not be general; and the enrolled patients may not represent patients outside the capture area for Danderyd University Hospital.

Data were collected only in the ED and addressed a single clinical problem

Data in this study was collected exclusively in a busy ED. There is no certainty that the inadequacy of EHR data in an ED apply equally to ambulatory care. There is no certainty that the differences found between EHR and CHT data will apply to all other clinical problems.

Incomplete CHT sessions

Relevant data was missing from some CHT files because some patients were discharged to home or admitted to hospital before completing CHT sessions. Also, some patients ended interviews prematurely at their discretion. No data was collected to determine factors influencing patients’ decisions to enroll, continue with or discontinue a CHT interview. These issues need targeted study.

Supporting information

S1 Fig. Sites of primary pain recorded in EHRs and reported by patients during CHT.

Each line are sites of pain for the same patient. Column numbers refer to regions of the chest displayed in Fig 1. Blue-colored upper triangles for each location of pain are data from the patient’s EHR. Blue-colored lower triangles for each location of pain are data from the patient’s CHT interview. Empty upper triangles indicate that the region was not mentioned as affected by pain in EHR data. Empty lower triangles indicate that the region was not selected by the patient interacting with the image in S1 Fig. Absence of EHR data for location of chest pain indicates that the EHR did not record a specific site of chest pain. Blue-colored upper triangles in Column 0 reflect that the description of chest pain in EHR narratives was too imprecise to be associated with a specific anatomic region of the chest. Projected areas of pain on each line are data for the same patient.

(PDF)

Click here for additional data file. (1.1MB, pdf)
S2 Fig

Regions of primary pain projected onto an image of the chest for selected patients recorded in EHR data (left hand panels) and CHT data (right hand panels).

(PDF)

Click here for additional data file. (2.6MB, pdf)

Acknowledgments

The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Data Availability

Data cannot be shared publicly because of ethical restrictions from the Swedish authorities as the data contain potentially identifying and sensitive patient information. Data could however be available for researchers who meet the criteria for access to confidential data, upon reasonable request to the authors and with permission of the Swedish Ethical Review Authority (https://etikprovningsmyndigheten.se registrator@etikprovning.se).

Funding Statement

This work was funded by the Robert Bosch Stiftung (https://www.bosch-stiftung.de/de, Stuttgart, Germany), grant number 11.5.1000.0258.0 to DZ. Region Stockholm (ALF project; https://www.vr.se/english/about-us/organisation/advisory-groups-and-administrative-offices/office-for-alf.html, Stockholm, Sweden), grant number 20190593 to TK. Karolinska Institutet Research Foundation (https://staff.ki.se/ki-research-foundation-grants-2020-2021, Stockholm, Sweden) and Stiftelsen Hjärtat (http://www.stiftelsenhjartat.se, Stockholm, Sweden) to TK. Funders had no role or influence on the design and conduct of the research, including software development, and were not involved in data analysis, conclusions drawn from the data, and drafting or editing the manuscript.

References

  • 1.Kirch W, Schafii WC. Misdiagnosis at a University Hospital in 4 Medical Eras. Medicine 1996; 75: 29–40. doi: 10.1097/00005792-199601000-00004 [DOI] [PubMed] [Google Scholar]
  • 2.Clarke AJ. Musings on genome medicine: the value of family history. Genome Med 2009; 1: 75.1–75.3. doi: 10.1186/gm75 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Wang CS, Fitzgerald JM, Schulzer M, et al. Does this dyspneic patient in the emergency department have congestive heart failure? JAMA. 2005;c294:1944–56. Clarke AJ. Musings on genome medicine: the value of family history. Genome Med 2009, 1: 75.1–75.3 [DOI] [PubMed] [Google Scholar]
  • 4.Paley L, Zornitzki T, Cohen J, et al. Utility of the clinical examination in the diagnosis of acute patients admitted to the department of medicine of an academic hospital. Arch Intern Med 2011; 171: 1394–6. doi: 10.1001/archinternmed.2011.340 [DOI] [PubMed] [Google Scholar]
  • 5.Howells JW, Short PA. The Value of the History and Physical Examination—Sailing Through Medicine With Modern Tools A Teachable Moment. JAMA Internal Medicine 2015175: 1901–2. doi: 10.1001/jamainternmed.2015.5768 [DOI] [PubMed] [Google Scholar]
  • 6.Bentsen BG. The Accuracy of recording patient problems in family practice. j med educ 1976; 51: 311–6. doi: 10.1097/00001888-197604000-00006 [DOI] [PubMed] [Google Scholar]
  • 7.Platt FW, Mcmath JC. Clinical Hypocompetence: The Interview. Ann Intern Med 1979; 91:898–902. doi: 10.7326/0003-4819-91-6-898 [DOI] [PubMed] [Google Scholar]
  • 8.Romm FJ, Putnam SM. The Validity of The Medical Record. Med Care 1981; 19: 310–5. doi: 10.1097/00005650-198103000-00006 [DOI] [PubMed] [Google Scholar]
  • 9.Zakim D, Braun N, Fritz P, Alscher MD. Underutilization Of Information And Knowledge In Everyday Medical Practice: Evaluation Of A Computer-Based Solution. BMC Medical Informatics and Decision Making, 2008; 8: 50. doi: 10.1186/1472-6947-8-50 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Zakim D, Fritz C, Braun N, Frtiz P, Alscher MD. Computerized History-Taking As A Tool To Manage LDL-Cholesterol. Vasc Health Risk Manage 2010; 6: 1039–46. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Burnett SJ, Deelchand V, Franklin BD, Moorthy K, Vincent C. Missing clinical information in NHS hospital outpatient clinics: prevalence, causes and effects on patient care. BMC Health Services Res 2011; 11: 14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Eze-Nliam C, Cain K, Bond K, Forlenz K, Jankowski R, Magyar-Russel G, et al. Discrepancies between the medical record and the reports of patients with acute coronary syndrome regarding important aspects of the medical history. BMC Health Services Res 2012; 12: 78. doi: 10.1186/1472-6963-12-78 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Hartzband PA, Groopman J. Off the Record—Avoiding the Pitfalls of Going Electronic. NEJM 2008; 358:1656–58. doi: 10.1056/NEJMp0802221 [DOI] [PubMed] [Google Scholar]
  • 14.Cohen GR, Friedman CP, Ryan AM, Richardson CR, Adler-Milstein J. Variation in Physicians’ Electronic Health Record Documentation and Potential Patient Harm from That Variation. J Gen Int Med 2019; 34: 2355–67. doi: 10.1007/s11606-019-05025-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Ospina NS, Phillips KA, Rodriguez-Guiterrez R, CastenadaGuarderas A, Gionfriddo MR, Branda MF, et al. Eliciting the Patient’s Agenda- Secondary Analysis of Recorded Clinical Encounters. J Gen Int Med 2019; 34: 36–40. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Mamykina L, Vawdrey DK, Stetson P. Clinical documentation: composition or synthesis? J Am Med Inform Assoc 2012; 19:1025–31. doi: 10.1136/amiajnl-2012-000901 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Farmer SA, Higginson IJ. Chest Pain: Physician Perceptions and Decision Making in a London Emergency Department. Ann Emerg Med 2006; 48: 78–85. [DOI] [PubMed] [Google Scholar]
  • 18.Berdahl CT, Moran GJ, McBride O, Santini AM, Verzhbinsky IA, Schrigor DL. Concordance Between Electronic Clinical Documentation and Physicians’ Observed Behavior. JAMA Network Open. 2019; 2(9):e1911390. doi: 10.1001/jamanetworkopen.2019.11390 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Zwaan L, Thijs A, Wagner C et al. Does inappropriate selectivity in information use relate to diagnostic errors and patient harm? The diagnosis of patients with dyspnea. Soc Sci Med 2013; 91: 32e–8e. [DOI] [PubMed] [Google Scholar]
  • 20.Clark BW, Derakhshan A, Desai SV. Diagnostic Errors and the Bedside Clinical Examination. Med Clin N Am 2018; 102: 453–64. doi: 10.1016/j.mcna.2017.12.007 [DOI] [PubMed] [Google Scholar]
  • 21.Berner ES, Kasiraman RK, Yu F, Ray MN, Houston TK. Data quality in the outpatient setting: impact on clinical decision support systems. AMIA Annu Symp Proc 2005; 41–5. [PMC free article] [PubMed] [Google Scholar]
  • 22.Bastian H, Glasziou P Chalmers I. Seventy-five Trials and Eleven Systemic Reviews a Day: How Will We Ever Keep Up? PlosMed 2010; 7: e1000326. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Obermeyer Z, Lee TH. Lost in Thought—The Limits of the Human Mind and the Future of Medicine. NEJM 2107; 377: 1209–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Slack WV, Hicks GP, Reed CE, Van Cura LJ. A computer- based medical-history system. NEJM 1966; 274: 194–8. doi: 10.1056/NEJM196601272740406 [DOI] [PubMed] [Google Scholar]
  • 25.Mayne JG, Weksel W, Sholtz PN. Toward automating the medical history. Mayo Clin Proc 1968; 43: 1–25. [PubMed] [Google Scholar]
  • 26.Zakim D. Development and Significance of Automated History-Taking Software for Clinical Medicine, Clinical Research and Basic Medical Science. J Int Med 2016; 280: 287–99. doi: 10.1111/joim.12509 [DOI] [PubMed] [Google Scholar]
  • 27.Brandberg H, Sundberg CJ, Spaak J, et al. Use of Self-Reported Computerized Medical History Taking for Acute Chest Pain in the Emergency Department–the Clinical Expert Operating System Chest Pain Danderyd Study (CLEOS-CPDS): Prospective Cohort Study. J Med Internet Res 2021; 23: e25493. doi: 10.2196/25493 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Six AJ, Backus BE and Kelder JC. Chest pain in the emergency room: value of the HEART score. Neth Heart J 2008; 16: 191–196. doi: 10.1007/BF03086144 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Zakim D, Alscher MD. A Comprehensive Approach To The IT-Clinical Practice Interface In Critical Issues For Sustainable E Health Solutions Wickramasinghe N, et al. (Eds) 2012, pp. 353–374, Springer, NY, NY. [Google Scholar]
  • 30.Halford GS, Baker R, McCredden JE, Bain JD. How many variables can humans process? Psychol Sci 2005; 16: 70–6. doi: 10.1111/j.0956-7976.2005.00782.x [DOI] [PubMed] [Google Scholar]
  • 31.Bordage G. Why did I miss the diagnosis? Some cognitive explanations and educational implications. Acad Med. 1999;74(10 Suppl): S138–43. doi: 10.1097/00001888-199910000-00065 [DOI] [PubMed] [Google Scholar]
  • 32.LeBlanc VR, Brooks LR, Norman GR. Believing is seeing: the influence of a diagnostic hypothesis on the interpretation of clinical features. Acad Med 2002; 77: S67–9. doi: 10.1097/00001888-200210001-00022 [DOI] [PubMed] [Google Scholar]
  • 33.Kahneman D. Thinking Fast and Slow. Farrar, Straus Giroux, NY, NY. 2011. [Google Scholar]

Decision Letter 0

Amit Bahl

27 Apr 2021

PONE-D-21-11324

Computerized History-Taking Improves Data Quality for Clinical Decision-Making. Comparison of EHR and Computer-Acquired History Data in Patients with Chest Pain.

PLOS ONE

Dear Dr. Zakim,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Jun 11 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Amit Bahl

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. In your Methods section, please provide additional information about the participant recruitment method and the demographic details of your participants. If materials, methods, and protocols are well established, authors may cite articles where those protocols are described in detail, but the submission should include sufficient information to be understood independent of these references (https://journals.plos.org/plosone/s/submission-guidelines#loc-materials-and-methods).

Thus, please ensure you have provided sufficient details to replicate the analyses such as:

a) the recruitment date range (month and year),

b) a description of any inclusion/exclusion criteria that were applied to participant recruitment,

c) a table of relevant demographic details,

d) a statement as to whether your sample can be considered representative of a larger population,

e) a description of how participants were recruited, and

f) descriptions of where participants were recruited and where the research took place. Moreover, please ensure that the tool used has been described in sufficient detail.

3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

4. We note that you have a patent relating to material pertinent to this article.

a. Please provide an amended statement of Competing Interests to declare this patent (with details including name and number), along with any other relevant declarations relating to employment, consultancy, patents, products in development or modified products etc.

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” If there are restrictions on sharing of data and/or materials, please state these.

Please note that we cannot proceed with consideration of your article until this information has been declared.

b. This information should be included in your cover letter; we will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: No

Reviewer #3: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors conducted a prospective observational study where they compared the data collected by physicians during an ED encounter for chest pain vs the data collected by a computerized history taking software. They enrolled 410 patients and concluded that computerized data was more complete and more representative of patient perceptions. They also suggest that computer generated histories will be more beneficial for aggregated data bases to be used in research. The authors continue to mention that physician histories are inaccurate and incomplete to an extent that clinical outcomes and patient care are effected by this. However, they present no data here to suggest that this is the case. Overall, the manuscript is well written. However, there are issues with broad overarching conclusions based off of likely clinically irrelevant differences between CHT and EMR data. This study is very relevant to EMR based population studies and shows how CHT can significantly increase the value of research using this methodology. If the language and conclusions are paired down to only include what was evaluated within this study, then this manuscript should be re-considered.

Points

1. In methods, need to describe the clinical setting and selection criteria rather than referencing previous work without any description.

2. Need to include the explicit question of pain that was shown in CHT. Looking at figure 1 this is quite different then what is described in the body of your paper.

3. Multiple mentions of improving patient outcomes. Authors seem to suggest that CHT, given more available data, will lead to better patient care. However, there is no room for these conclusions to be made from this paper. They have no data on accuracy of diagnosis, patient outcomes, or patient satisfaction. Therefore it seems premature to make any comment on CHT improving patient care.

4. Further analysis of the data from the 20 interrupted CHT sessions should be examined for level of agreement between EMR and CHT. It would be interesting to see if after talking to a physician there was more agreement between the 2 history taking methods. It is possible that the physician could clarify points that the CHT was unable too leading to differences in CHT answers after interview.

5. If you are going to excluded data from 2 patients who appeared to make deliberately false entries during CHT, this data should be made available to the reader so that they can understand your conclusion.

6. The description of how patient’s selected CP location that is currently in the results section and should be moved into methods. Additionally, the description currently seems overly complicated and verbose.

7. A major sticking point of the authors is specific location of chest pain. They note that EHR data was documented as left, right, or central but rarely specified upper or lower. The main issue I take here with calling this “incomplete” EHR data is that when given more options to specify upper or lower, the patient likely will, as we see with the CHT data. However, the clinical relevance to such specificity is likely little to none. Looking at figure 2 the CHT system of location classification is extremely over complicated from a clinical perspective. I would recommend re-evaluating your data based on more reasonable expectations from a clinical standpoint. Consider dividing the locations into L and R anterior chest, L and R arm, and back. 15 locations to describe chest/torso pain would never be something reasonable or clinically relevant to document from a physicians perspective. Another option would to be rather than comparing a clinically meaningless variable to a clinical one (EHR hx), just present how much more granular data given in CHT can be. In this way you could make the argument that this level of granularity may at some point show clinical relevance and therefore must be studied. Currently I do not know of any literature that would suggest this level of granularity is of any clinical value.

8. This theme is repeated in the discussion regarding pain radiation to the arm. Authors note that EHR indicated isolated radiation to the arm, while CHT indicated arm + shoulder. This information again seems clinically irrelevant and likely highlights the point that CHT was collecting unnecessary data that would have slowed down physician history taking. Additionally, even if mentioned by the patient, this likely something that would have been documented in a truncated fashion by the physician in order to save time during EMR note composition. If the goal was to demonstrate fallibility of the EMR physician history, they you would need to specifically analyze what was said by the patient and then what was documented by the physician. In clinical encounters it is not uncommon to exclude portions of the history that you feel are irrelevant or combine portions during authoring of your HPI in order to save time.

9. Authors note that CLEOS was programmed to ask about radiation only when patient had anterior chest pain therefore 6 patient’s did not have radiation data in this group. This concept perfectly illustrates the issue with concluding that CHT is more complete and beneficial to patient care than physician history taking. Many experienced clinicians have the same “programming” when it comes to history taking and only include or exclude questions based on overall gestalt and patient complaint. When presented with a limited amount of time for history taking the physician must rely on their ability to expand or contract their history taking in order to ask the most relevant and pertinent questions.

10. Sup Figure 1 and Figure 3 are very complicated and difficult to understand easily. Since this is a key component of your argument would recommend rethinking this figure.

11. Figures from CHT are poor resolution making them difficult to read.

12. Overall, the strongest point of this paper is that EHR data is likely inadequate for any population-based studies and that CHT could improve this process. It is overreaching and likely inaccurate that CHT improves any sort of patient outcomes or that physician history taking is deficient. There is certainly no data within this manuscript to suggest that CHT may have lead to any improvement in quality of patient care.

Typos: Page 5 “The problem was corrected during the course of this stud.”

Reviewer #2: The authors have developed a tool for patients to use to enter their clinical history electronically, when they attend an emergency department with chest pain. They have tested this by enrolling patients at a single site, university hospital in Sweden, who attended their ED with chest pain, who didn't have a diagnostic first ECG on arrival. Patients were asked to enter clinical data into the tool, which was compared to the medical notes made by their attending physician after they had seen the patient. They enrolled a convenience sample of 410 patients. They then compared the patient authored data to the physician medical notes in the Electronic Health Record. They conclude that there is a lot of variation between the patient and physician notes, then state that their tool is better than the physician evaluation of the patient.

Major issues:

The manuscript has multiple major issues. Some are domain issues, others relate to the write-up.

1. The structure of the manuscript - I strongly suggest that the authors review the appropriate equator guideline and rewrite their manuscript accordingly. This is a comparative observational study, so STROBE would be appropriate. If this is done, the authors will realise that multiple mandatory areas of information are absent from each section. They should end with re-writing the abstract into a structured format. Given that there is machine learning involved in the development of the tool, the authors should also consider reporting elements of the TRIPOD guidelines.

2. There seems to be a major misunderstanding by the author group about the role of medical notes. This has made their work fundamentally flawed, despite the valid work that has gone into their research. Medical notes are not a verbatim record of what the patient said, nor are they usually intended to be a complete record - physicians don't have time and would record medical interviews if this was required. Medical notes are a synthesis or interpretation of information provided from various sources, aiming at being concise and useful in explaining ongoing medical decision making. Physicians deliberately exclude some history features from their notes, when the patient thinks they have a symptom, but when clarified, the symptom is different to the one sought by the physician - for example pleuritic chest pain. Comparing patient reported symptoms to a medical note is not a reasonable comparison. Whilst it is very possible that physicians have missed important aspects of the history during their consultations, this hasn't been proven by this research.

The authors might be better off asking a different research question: If the physician is presented with the patient reported information to read prior to/during/after the consultation, does this change the medical management or decision making or differential diagnosis at all? It is possible that it does change management and that physicians don't always ask enough detail about patient symptoms, but this hasn't been investigated so far and would be much more important to physicians.

3. The length of the manuscript and the discussion is overwhelming. A more concise version may be better received by readers. In contrast, the limitations section is too short, seemingly failing to understand the limitations of the study.

4. If readers are to understand the research, some illustrative examples should be provided of the typical outputs obtained by the tool, compared to the medical notes.

Suggested write up improvements:

Abstract: Many important details are missing from the abstract. This includes a description of the aims, population, setting, intervention (and that you developed it and own the patent), demographics of participants, methods of comparison, and so on. You seem to have simply written up your conclusions. You have no evidence that your tool outperformed clinicians, simply that the data you compared were different.

Introduction: A well structured, medical manuscript introduction should be about 4 paragraphs, broken up into the initial description of the problem and its magnitude; what is currently known; what is the gap in the literature and why does it matter and then a short goals of the investigation section.

Methods:

Each section needs improvement (or creation). It is not enough to state that you have published a study protocol, and not to describe your setting and population. Readers shouldn't have to look up another study for such basic information. You need a statistical plan - sample size calculation, statistical methods etc. Please refer to the equator guidelines +/- TRIPOD for more information

Results:

This should start with a flow diagram or statement, beginning with how many patients attended your ED with chest problem (either at triage or as a discharge diagnosis), how many patients were screened, how many were excluded and why, how many were enrolled, how many complete datasets (per participant) you obtained, how many were analysed, how many weren't and why etc. (see CONSORT guidelines for examples)

Next you require a section on your participant demographics, compared to those you didn't enrol.

Finally the main results should be presented. I would move much of your description to an appendix, the length is overwhelming to read. I would suggest that the narrative text that remains is more concise and less judgemental - present that something was present or absent from one set of notes or another. You have some odd groupings to your section sub-headings. Why for example is there a heading about both sweating and pleuritic pain, this should be broken up as the symptoms are completely unrelated clinically.

Discussion

Much of the discussion reads as though there is a perception that the patient notes are better (more accurate) and more useful. At best the authors can state that the data was different. Heuristics are extremely important in clinical medicine, to effectively write this off is deeply flawed. The medical evaluation and the medical notes are intended to be a synthesis and the way the manuscript is written seems to have completely misunderstood this point.

Much of the discussion is giving unproven opinions. I would suggest that as this manuscript is intended for a medical audience (I assume) that the structure is 1. a short summary of important results 2. A comparison to previous literature (does it confirm or refute previous work) 3. external generalisability (or lack of same) 4. limitations and future work

It would be important to discuss multiple limitations to this work in the discussion. One very important point is that patients rarely present to the ED with differentiated, well defined symptoms - such as chest pain. Many patients, even those with chest pain, wouldn't have only chest pain. To only obtain a history regarding their chest symptoms, would be a mistake that will lead to medical errors.

Reviewer #3: The manuscipt addresses an important question in this era of computerized medicine - does the traditional EHR record compare favorably to a patient-entered computerized history taking program. However the manuscript in the current form has several critical issues that must be addressed in order to be publishable.

Abstract:

The abstract has no background or context for the study. It dives right into methods. What do we know about computerized history taking? Why is it important to compare computerized history taking to the standard EHR history? What were the primary and secondary aims of the study?

Introduction:

This needs some major editing. There are many unclear statements such as: "Moreover, objective data cannot yet supplant medical history data". The introduction includes results and interpretation which are inappropriate for this part of the manuscript. This needs to be restructured: what is the background, what is unknown about the problem, what knowledge holes are you hoping to fill in, what are your primary and secondary aims?

Materials and Methods:

Also needs major edits. Clinical setting, selection criteria, and recruitment need to be spelled out here, not listed in a separte document. There needs to be an explanation of how the EHR history was taken by the physician and entered into the computer as well. There needs to be a discussion of the statistical analysis. Finally, the last sentence (Four patients did not...) should be in the results section.

Results:

This is far too long. Please summarize the major findings and point to the specifics in your tables. Also, the tables need improvement. What are the Ns for each grouping? Are the congruences reported statistically significant? Also there is occasionally author interpretation in these results which should be in the discussion

Discussion:

This discussion veers way off topic and needs improvement. This isn't a review article about the challenges of history taking. This is a study comparing EHR vs computerized history taking and the discussion should reflect that. How do these findings fit in with what is known? What remains to be discovered? Is this clinically relevent? Maybe physicians obtain what they need to in order to make the correct diagnosis and the rest is not needed. Maybe the physicians do ask the questions but don't waste their time on entering it into the EHR. The statement "It is fair to argue that history-taking by physicians has become an inefficient use of the physician's time with the patient" is not supported - what about building rapport? What about the nuance that comes from non-verbal or non-written conversation?

Finally:

There are multiple spelling and typographical errors that need to be addressed.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2021 Sep 27;16(9):e0257677. doi: 10.1371/journal.pone.0257677.r002

Author response to Decision Letter 0

19 May 2021

Reply to Reviewer #1. Reviewer’s words are in Italics.

Summary statement.

a. The reviewer concludes that we describe results from an observational study. This is correct only in the sense that all experiments involve "observations." An "observational study" in the clinical context refers, however, to a study in which clinical outcomes are measured in the absence of matched experimental and control cohorts. Our work is a "within person" comparison of CHT and EHR chest pain data. Outcomes for enrolled patients were not captured. We clarify the design of the study in the revised manuscript by adding an explicit statement about the study design as the first paragraph of the Methods section.

b. The reviewer states: The authors continue to mention that physician histories are inaccurate and incomplete to an extent that clinical outcomes and patient care are effected[sic]by this.

This comment misstates what the text includes and ignores the evidence cited to support this text. We do not aver but cite the evidence in the Introduction that medical history data is the salient data for diagnostic decisions, that history-taking by physicians is inadequate and that clinical errors are frequent. We assumed that the connection between these factually correct statements i.e., citations of the evidence in the literature, is obvious to the biomedical community and did not explicitly connect poor history-taking with clinical errors in the original text. We do so in the edited text and cite the literature supporting this connection.

c. We did not elaborate in the original text that poor history-taking, including the falsification of findings, is documented by post hoc review of records physicians generate and by real-time observation of physicians interviewing patients. This work was cited in the original text but not mentioned separately. The edited text elaborates that there are two sets of observations documenting poor history-taking by physicians: (i) post hoc analysis of medical records and (ii) real-time observation of physicians interviewing patients.

d. The reviewer states: ... there are issues with broad overarching conclusions based off of likely clinically irrelevant differences between CHT and EMR data.

The original text compared EHR and CHT files for a range of data elements that are relevant now to the differential diagnosis and triage of patients with chest pain. Our original text also described the full range of differences between EHR data and CHT data for precise locations of primary and radiated chest pain. The reviewer seems only to raise a question about differences between EHR and CHT data for elements that appear to have no clinical relevance today. We have addressed this criticism in the following way. i. . Results for primary and radiated pain in the edited text are restricted, nearly exclusively, to data elements relevant to this differential and to triage algorithms for patients with chest pain. ii. We have removed almost all text for primary and radiated pain that does not include areas and patterns of pain applicable to guideline-based management of chest pain. iii. We have removed one table and one figure to focus the data for locations of pain on guideline-based patterns. iv. We mention only in passing that CHT data is more precise as compared with EHR data and refer the reader to Supplementary Fig 1.

Numbered items.

In methods, need to describe the clinical setting and selection criteria rather than referencing previous work without any description.

Response: These details have been added to Methods and not simply referenced.

2. Need to include the explicit question of pain that was shown in CHT. Looking at figure 1 this is quite different then what is described in the body of your paper.

Response: We do not understand the problem the reviewer mentions. It seems the reviewer might have confused Figs. 1 and 2. We have reviewed the text citing these Figs and their legends. Their meaning seems clear.

3. Multiple mentions of improving patient outcomes. Authors seem to suggest that CHT, given more available data, will lead to better patient care. However, there is no room for these conclusions to be made from this paper. They have no data on accuracy of diagnosis, patient outcomes, or patient satisfaction. Therefore it seems premature to make any comment on CHT improving patient care.

Response: The word "outcomes" appears only once in the original text (in the Introduction). We have removed the word "outcomes" from the edited Introduction. The word "outcomes" appears otherwise in the titles of 4 journal articles in the original reference list. These references were to studies of the effectiveness of EHR-based automated clinic al decision support. The edits to the text no longer refer to any of these citations, which are absent from the edited reference list.

Moreover, we did not in the original and do not in the revised text state or suggest that CHT will improve diagnostic accuracy or outcomes.

However, on the basis of substantial literature supporting it, we make this argument in the original and edited Introduction: Medical history data is essential for correct diagnostic and treatment decisions. Poor medical history data is associated with diagnostic errors. Thus, we need to develop better methods for history-taking. 4.

4. Further analysis of the data from the 20 interrupted CHT sessions…

Response: We have added the relevant details to Methods in the revised manuscript. These details, i.e., discrepancies between EHR and CHT data for 20 records with a long interruption during CHT as frequent as in 390 other files, belong in Methods because they explain why data for all 410 patients was combined.

5. If you are going to excluded [sic]data from 2 patients …

Response: We have added to Methods the specific data elements in the interviews of two patients that led us to exclude their data, which appeared to be corrupted deliberately.

6. The description of how patient’s selected CP location that is currently in the results section and should be moved into methods. Additionally, the description currently seems overly complicated and verbose.

Response: We believe the manuscript reads more clearly with the Fig and its description in Results. We decided not to move Fig 2 and the description of how patients indicated areas of primary pain to Methods.

We note, however, that DZ misread the author instructions for placement of figure legends and tables in the main text. He placed these immediately after the first mention of a figure or table and not, as instructed, in the first paragraph after the first mention. DZ's error undoubtedly contributed to the reviewer's interpretations of the text and legend for Fig 2 as ... overly complicated and verbose.

DZ's errors in placing figure legends and Tables in the text are corrected in the revised manuscript. We did not edit the text referring to Fig 2, which is a simple sentence. "Patients reported pain during CHT using a version of Fig 2 without numbered anatomic areas." We did not edit the legend for Fig 2, which seems clear.

7 and 8. A major sticking point of the authors is specific location of chest pain.

Response: Our response to the criticism that sites of pain were not clinically relevant and the edits made to the text are given in detail in our response to Summary statement, paragraph d. The fact of the matter is the original text described important differences between EHR and CHT data for sites of primary and radiated pain that are relevant to managing chest pain. As mentioned above, however, we have removed detailed comparisons between EHR and CHT data for sites of pain that are not known to be clinically relevant.

This theme is repeated in the discussion regarding pain radiation to the arm.

Looking at figure 2 the CHT system of location classification is extremely over complicated from a clinical perspective.

This is true if the data has to be communicated orally to a physician, who has to remember the details of complex pain locations. It is not true in a CHT system.

Our results show in fact that the physician probably does not collect and or does not recall simple pain locations accurately, e.g., 50% of EHRs did not cite a location for primary chest pain. Half the EHRs did not have an entry for radiated pain. The half without a site for primary pain was not the half without an entry for radiation. These points are emphasized in the edited text that compares EHR and CHT data, as regards primary and radiated pain, for presence/absence of central chest pain and citation of bilateral pain radiation or radiation to the right side or no radiated pain.

In clinical encounters it is not uncommon to exclude portions of the history that you feel are irrelevant or combine portions during authoring of your HPI in order to save time.

This criticism of our view of the value of an accurate clinical record is precisely why we need programs like CHT to collect and record medical histories.

For as we cite in the original and make more emphatic in the edited text, failure to enter negative findings does not explain the high frequency of the absence of clinically relevant positive findings in EHRs and does not account for the apparently false negative and false positive findings in the EHRs examined, e.g., no entry for site of pain, incorrect entries for which patients did or did not have central chest pain, false positive and false negative findings for pain beginning during physical activity, and so on.

DZ, TK, JS, HB and CJS are experienced clinicians. We have all saved time by shaving details from our clinical notes. But we acknowledge that electronic medical records are intended to serve as a mechanism to generate patient-specific clinical decision support and as databases for clinical research and that these uses imply the expectation that physicians will record all relevant data in patients' EHRs.

We all know, however, that we don't and in fact cannot meet this standard. So we need a method like CHT that accurately collects and records all relevant positive and negative findings.

9. Authors note that CLEOS was programmed to ask about radiation only when patient had anterior chest pain therefore 6 patients did not have radiation data in this group. This perfectly illustrates the issue with concluding that CHT is more complete and beneficial to patient care than physician history taking.

Many experienced clinicians have the same “programming” when it comes to history taking and only include or exclude questions based on overall gestalt and patient complaint.

Response: The reviewer missed an obvious error in the programming of the CLEOS knowledge base. Patients with primary posterior chest pain and no primary anterior pain should be asked about radiation from back to front around the rib cage. The edited text mentions this error. We also have added a brief mention in the edited Discussion of the value of editing CHT software on the basis of clinical trials.

We believe too that the reviewer’s opinion that … experienced clinicians have the same “programming”… is further affirmation of the value of CHT. Thus, even experienced physicians (the authors included) make errors of omission during history taking because “programming” the human mind does not insure failure- proof performance. By contrast, CHT may make errors; but once a program error is corrected, CHT will not make the same error again.

10. Supplemantary [sic]Figure 1 and Figure 3 are very complicated and difficult to understand easily. Since this is a key component of your argument would recommend rethinking this figure.

Response: Supplementary Fig 1 has > 15,000 data fields for locations of primary pain for 410 patients in EHR and CHT records. We see no simpler way to display this amount of data than the scheme of Supplementary Fig 1. Moreover, Supplementary Fig 1 is not ... a key component of [our] argument.

The data in this figure were extracted and converted to text and Tables in the original and edited versions. The reader thus does not need to use the figure to understand the presentation of results for primary pain in the central chest. The interested reader, however, can use the figure to verify our findings. Moreover, the figure shows in a simple way, by comparison of upper and lower colored triangles across each line, the "granularity" of CHT as compared with EHR data. It also shows by inspection the extent and specifics of what is missing in EHR data as compared with CHT data.

We have removed the original Fig 3, which displays a portion of the data in Supplementary Fig 1.

11. Figures from CHT are poor resolution making them difficult to read.

Response: The reviewer is correct. We submit a better version of this Figure with the edited version.

12. There is certainly no data within this manuscript to suggest that CHT may have lead[sic] to any improvement in quality of patient care.

Response: We agree that there is no such data in our manuscript. And whereas this comment implies we claim CHT will improve outcomes, the original text makes no such claims. We only cite, in the Introduction, the literature reporting that medical history remains the salient data for diagnosis, that physician-acquired histories are incomplete and generally inaccurate and that poor history taking is associated with diagnostic error. We then state that these undisputed findings point to the need for better history-taking. We state in the Discussion that the data analyzed indicates that the 410 EHRs reviewed are inadequate to support meaningful decision support or to be a resource for future clinical research. We also state that the CHT data would support these functions. The data analyzed align with these conclusions. And as mentioned in our response to the reviewer’s item 3, we draw no conclusions about the effect of CHT on patient outcomes.

Reply to Reviewer #2. Reviewer’s words are in Italics.

Summary statement.

a. The authors have developed a tool for patients to use to enter their clinical history electronically...

Patients were asked to enter clinical data into the tool...

They conclude that there is a lot of variation between the patient and physician notes,

We believe the reviewer either did not understand the experiment we report or was unable to explain his understanding articulately. Patients did not simply enter their clinical history electronically... We collected no patient ... notes. No patient authored data was collected.

We might have contributed, however, to the reviewer's possible misunderstanding of CLEOS and expert system software generally by referencing details of the CLEOS program that we should have described more fully in Methods.

We have added a fuller description of how CLEOS operates to interview the patient.

Patients interacted with expert system software programmed to emulate the clinical reasoning of a knowledgeable clinical expert collecting a medical history from a patient with acute chest pain. The expert system software used the same pathophysiologic reasoning to pose the same types and range of questions that an expert physician would employ while interviewing the same patient. Thus data entered by the patient during CHT consists of structured data captured by systematic patient interview.

b. They enrolled a convenience sample of 410 patients.

We consecutively enrolled patients presenting with chest pain in the ED. This was not explicitly stated in the original text but is clarified in the revised text. We do not understand what the reviewer means by a convenience sample.

c. They conclude that there is a lot of variation between the patient and physician notes, then state that their tool is better than the physician evaluation of the patient.

This statement is imprecise and incorrect.

(i). There were no patient notes in the data extracted.

(ii). We do not conclude ... there is a lot of variation between the patient and physician notes. Our results show clearly that EHRs were missing for multiple, specific data elements that are significant for the differential diagnosis of chest pain and that this data was present in CHT files for corresponding patients. We also show that besides the absence of relevant data fields, EHRs had large numbers of false negative and false positive entries by comparison with the data collected by CHT interview for clinically relevant information known only to the patient.

The reviewer's apparent confusion about what the text states, to which we refer in (i) and (ii), is addressed in the edited version by the fuller description in Methods of how CLEOS is designed and operates.

(iii) We do not state that their [our] tool is better than the physician evaluation of the patient. This comment does not reflect what the text states. We report only on a comparison of EHR and CHT medical history data. Our text is mute on physician evaluations of patients. The reviewer's criticism might reflect a jump in thinking in which the reviewer connected our findings to the value of history data in evaluating patients.

Numbered items.

The structure of the manuscript - I strongly suggest that the authors review the appropriate equator guideline and rewrite their manuscript accordingly. This is a comparative observational study, so STROBE would be appropriate. If this is done, the authors will realise that multiple mandatory areas of information are absent from each section. They should end with re-writing the abstract into a structured format. Given that there is machine learning involved in the development of the tool, the authors should also consider reporting elements of the TRIPOD guidelines.

Response: There is no appropriate Equator guideline for the experimental design we used.

The reviewer concludes that we describe results from an observational study. This is correct only in the sense that all experiments involve "observations."

An "observational study" in the clinical context refers to a study in which clinical outcomes are measured in the absence of matched experimental and control cohorts. Our work is a "within person" study. In addition, we did not measure any clinical outcome. We clarify the design of the study in the revised manuscript by adding an explicit statement about the study design as the first paragraph of the revised Methods section. However, we have reorganized the text of the Abstract, Introduction, Methods and Discussion sections along the Equator lines.

Given that there is machine learning involved in the development of the tool...

Machine learning is not mentioned directly or by reference in the manuscript. Machine learning was not used to develop the expert system software used for CHT. CLEOS does not include any machine learning algorithms. These issues are clarified in the edited Methods section, which emphasizes that CLEOS interviews strictly according to pathophysiologic concepts.

2. There seems to be a major misunderstanding by the author group about the role of medical notes.

DZ, TK, JS, HB and CJS are experienced clinicians. We have all saved time by shaving details from our clinical notes. We acknowledge that electronic medical records are intended to serve as a mechanism to generate patient-specific clinical decision support and as databases for clinical research. These uses imply the expectation that physicians will record all relevant data in patients' EHRs.

We all know they don't and in fact cannot meet this standard. So we need a method like CHT that accurately records all relevant positive and negative findings. The criticism is the same as item 8 in review #1. Our reply to reviewer #2 on this issue is the same as our reply to reviewer #1.

Physicians deliberately exclude some history features from their notes, when the patient thinks they have a symptom, but when clarified, the symptom is different to the one sought by the physician...

The reviewer is correct on this point. But the idea expressed here - that physicians can divine what patient's experience - is precisely why we need CHT to collect and record medical histories. Indeed, we cited literature in the original text documenting that physicians purposely report findings that are contrary to factual information reported by their patients (c.f., Mamykina L et al J Am Med Inform Assoc 2012; 19:1025–31; Berdahl CT et al. JAMA Network Open. 2019; 2(9):e1911390; Farmer SA, et al. Ann Emerg Med 2006; 48: 78-85). These cites were in the original manuscript but were combined with the literature for post hoc analysis of clinical records. We have edited the textual reference to these papers to highlight that they deal with real-time observations of physicians corrupting data as they collect it.

Comparing patient reported symptoms to a medical note is not a reasonable comparison.

We did not compare “notes.” We compared entries for specific data elements in EHRs with entries for the same specific data elements in CHT files for data elements that can be reported only by affected patients, e.g., the site of their chest pain. Methods in the original and revised versions state explicitly how the data analyzed was extracted from EHR and CHT datasets.

3. The length of the manuscript and the discussion is [sic] overwhelming. A more concise version may be better received by readers. In contrast, the limitations section is too short, seemingly failing to understand the limitations of the study

Response: The reviewer's suggestions for revising the manuscript are replete with misstatements of fact about the original manuscript.

We have changed the heading that confused the reviewer to Associated Symptoms.

The authors might be better off asking a different research question: If the physician is presented with the patient reported information to read prior to/during/after the consultation, does this change the medical management or decision making or differential diagnosis at all?

Obviously, this would be a significant experiment. However, it not the first experiment to run in developing expert systems for CHT.

4. If readers are to understand the research, some illustrative examples should be provided of the typical outputs obtained by the tool, compared to the medical notes.

Response: The text describes how patients entered data for sites of primary pain by interacting with Fig 2.

We have added a description to Methods on the development of questions and answers, how patients select answers and how their entries are saved. In brief, the data fields analyzed were simple linguistically. For example, location of primary pain; onset during physical exercise yes/no; self-administered nitroglycerin yes/no. Methods explain how the data was acquired and how it was extracted from EHR and CHT data. The contents of Results are the typical outputs that were extracted from CHT and EHR records for specific data elements.

Suggested write up improvements:

Abstract

We have added the information to the abstract that reviewer #2 suggests we add.

Introduction

We believe the original text met the reviewer's suggestions. But we have shortened the edited Introduction.

Methods

Also needs major edits. Clinical setting, selection criteria, and recruitment need to be spelled out here, not listed in a separte [sic] document. There needs to be an explanation of how the EHR history was taken by the physician and entered into the computer as well. There needs to be a discussion of the statistical analysis. Finally, the last sentence (Four patients did not...) should be in the results section.

As noted above, we have included in the edited methods section sufficient information to replicate the experiment we report without reference to other literature.

There needs to be an explanation of how the EHR history was taken by the physician and entered into the computer as well.

We have added the explicit statement that physicians interviewed patients while delivering routine care.

Finally, the last sentence (Four patients did not...) should be in the results section.

The data for the 2 not 4 patients to whom the reviewer refers was not included in the analyses presented in Results.

Results

i. This should start with a flow diagram or statement,beginning with how many patient...

These details belong in Methods, which has been edited as described above.

ii. The length of the text is overwhelming to read.

The original text had no fluff. The edited text is shorter because we have removed details that are not known at present to be clinically relevant to the differential of chest pain.

iii. I would suggest that the narrative text that remains is more concise and less judgemental [sic] - present that something was present or absent from one set of notes or another.

The text in Results, original and edited versions, contains statements of fact in regard to the data examined, i.e., data was missing in EHRs, data in EHRs was inconsistent with the same data in the same patient's CHT interview.

Discussion

i. Much of the discussion reads as though there is a perception that the patient notes are better (more accurate) and more useful.

There are no patient notes.

That said, we review a set of data elements that are patients' perceptions of the location of primary pain, the location of radiated pain, the setting in which their pain began, when it began, whether it was episodic or constant, and so on. We compare these elements as recorded in EHRs and as recorded automatically as the elements were collected by interactions between the patient and an expert system, as described above. We found that most of the data reviewed was missing from EHRs and that much of the data that was present in EHRs was contrary to patients' perceptions of their pain that were collected by CHT. Of course, the discussion of these findings, in the context of an abundance of literature indicating that physician histories are incomplete, inaccurate and sometimes deliberately will convey the perception that the patient knows more about what they feel than their physician does.

ii. Heuristics are extremely important in clinical medicine, to effectively write this off is deeply flawed.

The medical and psychological literature show clear, negative effects of heuristics on decision making inside and outside the sphere of clinical medicine. The edited version cites the literature explicitly on this issue.

iii. Much of the discussion is giving unproven opinions.

This criticism is a non-sequitur. Opinions are not facts. So the criticism the reviewer wants to make is that the Discussion is mostly opinion. Most of the Discussion reviews the findings in the context of prior work on the quality of physician-acquired histories. We briefly speculate on the long-term significance of our results. We believe this is one of the purposes of a Discussion section, i.e., what might be the long-term significance of the work reported.

iv. It would be important to discuss multiple limitations to this work in the discussion. One very important point is that patients rarely present to the ED with differentiated, well defined symptoms - such as chest pain.

The complexity of these problems highlights the need to collect highly detailed information from the patient. Since half the author group comprises physicians with clinical experience, the authors understand first-hand the complexity of the problems presented by sick patients.

Reply to Reviewer #3. Reviewer’s words are in Italics.

Abstract

The abstract has no background or context for the study. It dives right into methods. What do we know about computerized history taking? Why is it important to compare computerized history taking to the standard EHR history? What were the primary and secondary aims of the study?

Response: We have edited the abstract to include the items the reviewer suggests and to make it less abrupt.

Introduction

Moreover, objective data cannot yet supplant medical history data.

This is a quote from the original manuscript. We have edited it out of the revised manuscript.

... What is the background, what is unknown about the problem, what knowledge holes are you hoping to fill in, what are your primary and secondary aims?

We have reread the Introduction and conclude it includes all the elements asked for here. However, we have edited the language of the Introduction to bring attention more effectively to these issues.

The introduction includes results and interpretation which are inappropriate for this part of the manuscript.

Contrary to the reviewer's opinion, this is an acceptable style in the physical science and biomedical literature. DZ, the author of the first draft of the manuscript, has written Introductions to his papers in this style for 60 years. We retain this style in the edited version but have shortened the text.

Materials and Methods

Also needs major edits. Clinical setting, selection criteria, and recruitment need to be spelled out here, not listed in a separte [sic] document. There needs to be an explanation of how the EHR history was taken by the physician and entered into the computer as well. There needs to be a discussion of the statistical analysis. Finally, the last sentence (Four patients did not...) should be in the results section.

We have added all details to Methods that previously were referenced. We also have added clarifications about the nature of the CLEOS software program. These edits include reviewer #3's reference to 4 patients in the last line of Methods. The correct and cited number is two not four patients. We keep the text on excluding data for these two of 410 patients in Methods because their data was excluded from analysis (for the detailed reasons added to the edited version).

Results

Results have been edited for clarity and to remove detailed analysis of data elements not immediately relevant to the differential of chest pain.

Our Results section presents major findings. We summarize these briefly, as the reviewer suggests, in the Abstract and at the beginning of the Discussion.

What are the Ns for each grouping?

The Tables and text clearly indicate the numbers of patients in each group and with each finding.

Are the congruences [sic] reported statistically significant?

If there were statistical noise in answers patients provide when asked about their symptoms, e.g., the site of chest pain, then medical history data obtained by any method would not be useful for any clinical purpose. In addition, the original and edited texts state how we looked for evidence, that patient’s perceptions of the site of pain might have changed across short time intervals. We mentioned in the original manuscript that the relevant evidence indicated that patient-reported pain patterns were stable.

The Reviewer's question about statistical significance called our attention to an error in our calculations of congruence. We calculated per cent congruence across the entire set of 410 patients. Congruence should have been calculated as the number of patients found by EHR data with a specific data element divided by the number of patients with the same specific data element found in CHT data. We have corrected accordingly the tables and text in the Revised manuscript.

... there is occasionally author interpretation in these results which should be in the discussion.

The Results section has text other than table and figure legends. But we do not understand what the reviewer refers to as "interpretation."

Discussion

How do these findings fit in with what is known?

This is covered in the original Introduction and Discussion. It is covered at this reviewer's suggestion as well in the edited Abstract of the edited paper.

Maybe physicians obtain what they need to in order to make the correct diagnosis and the rest is not needed.

The manuscript cites extensive literature in the Introduction and again in the Discussion that shows there is no maybe about it. The literature establishes that physicians generally do not get the data that is needed!

Our Results show specifically for selected data elements in a large sample of patients that physicians do not record complete or apparently accurate data that is immediately relevant to the differential diagnosis of acute chest pain.

Maybe the physicians do ask the questions but don't waste their time on entering it into the EHR.

As we cite in the original and more emphatically in the edited text, failure to enter negative findings does not explain the high frequency of discrepancies between EHR and CHT for clinically relevant, apparently false negative and false positive findings in the EHRs examined, e.g., no entry for site of pain, incorrect entries for which patients did or did not have central chest pain, false positive and false negative findings for pain beginning during physical activity, and so on. Failure to enter negative findings also does not explain the high frequency of missing EHR data for positive findings reported by patients during CHT.

The Reviewer objects to our statement: "It is fair to argue that history-taking by physicians has become an inefficient use of the physician's time with the patient."

We have edited out this statement.

The reviewer asks ...what about building rapport? What about the nuance that comes from non-verbal or non-written conversation?

The reviewer seems to have the jump to thinking that CHT will replace physicians. We do not suggest or imply that CHT will replace physicians. The manuscript points out instead, however, that CHT can enhance the value of the physician-patient interaction for physician and patient and especially contribute to patient trust, which is important for patient compliance.

Attachment

Submitted filename: Response to reviewers.docx

Click here for additional data file. (40.7KB, docx)

Decision Letter 1

Amit Bahl

24 Jun 2021

PONE-D-21-11324R1

Computerized History-Taking Improves Data Quality for Clinical Decision-Making. Comparison of EHR and Computer-Acquired History Data in Patients with Chest Pain.

PLOS ONE

Dear Dr. Zakim,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Thank you for reviewing and responding to the reviewer concerns. However, additional modifications are needed. Please note reviewer comments. Id advise you to pay special attention to common themes among reviewers as this usually signifies an area of opportunity. Particularly, the results section needs to be pared down to be more organized and sensible. Some data may be better displayed in tables - I strongly recommend moving some data to supplementary or supporting tables/figures. I hope you will consider the comments and make adjustments accordingly. 

Please submit your revised manuscript by Aug 08 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Amit Bahl

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: (No Response)

Reviewer #3: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: I Don't Know

Reviewer #3: No

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: (No Response)

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: (No Response)

Reviewer #3: No

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: In my first review of this paper the main issue I took was with a broad over-reaching conclusion based off of likely clinically irrelevant differences between CHT and EMR data. The authors have pared down their claims to a level that fits within the scope of their paper. However there are still several mentions such as in the abstract, “CHT out-performed history taking by physicians” yet there is still no clinically relevant indication of this. While I understand the authors opinion that CHT was created by experts and is likely a better representation of an accurate chest pain history, there is no indication that clinical outcomes are at all effected by this. The authors should carefully re-read their manuscript to eliminate any indication that CHT outperforms EMR history in any manor that would improve clinical outcomes.

Overall, this version of the manuscript is much stronger. I think that while adding some relevant findings, the results section remains too long and verbose. Much of the “results” section should be moved to discussion instead.

Reviewer #2: (No Response)

Reviewer #3: The study authors have made several edits relating to the concerns from the first submission, however the manuscript as it requires further editing or rewriting to be at an acceptable level for publication.

Abstract – lacks objective (statistical) data to support its conclusions

Introduction – authors should remove their conclusions from the introduction – specifically “Our data shows that CHT records contain more complete and more accurate representations of patients' perceptions of their symptoms than patients' corresponding EHRs.” Please refer to the STROBE checklist if there is confusion on this point.

Methods – Gender and age data should be reported in the results section. Authors need to include a description of their statistical methods.

Results – The data on location of primary pain, location of radiated pain, setting of pain onset, and frequency/duration of pain, and data for associated symptoms, use of nitroglycerin, and dimensionality is very difficult to interpret. The descriptions are very long and would be best presented in table format, with limited narrative. Also, many values are reported as integers, where percentages would be better used. Finally, only raw data is presented; testing for statistical significance in the different outcomes between the EHR and CHT data sets would greatly strengthen the results.

Discussion - The authors should situate their findings in the context of prior research comparing EHRs and CHTs (ie. this is the first study, or consistent with prior studies, or contrary to prior studies) .

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2021 Sep 27;16(9):e0257677. doi: 10.1371/journal.pone.0257677.r004

Author response to Decision Letter 1

23 Jul 2021

Replies to reviewers' objections.

Reviewer #1: In my first review of this paper the main issue I took was with a broad over-reaching conclusion based off of likely clinically irrelevant differences between CHT and EMR data. The authors have pared down their claims to a level that fits within the scope of their paper.

However there are still several mentions such as in the abstract, “CHT out-performed history taking by physicians” yet there is still no clinically relevant indication of this.

Reply: We removed the phrase and the complete sentence in which it appeared

----------------

While I understand the authors opinion that CHT was created by experts and is likely a better representation of an accurate chest pain history, there is no indication that clinical outcomes are at all effected by this. The authors should carefully re-read their manuscript to eliminate any indication that CHT outperforms EMR history in any manor that would improve clinical outcomes.

Reply: There is no mention in the manuscript that CHT will improve outcomes.

---------------

Overall, this version of the manuscript is much stronger. I think that while adding some relevant findings, the results section remains too long and verbose. Much of the “results” section should be moved to discussion instead.

Reply: Results has no text not immediately relevant to the primary data. We see no material in Results that fit better in Discussion.

______________________________________________________________________________

Reviewer #2: (No Response)

______________________________________________________________________________

Reviewer #3: The study authors have made several edits relating to the concerns from the first submission, however the manuscript as it requires further editing or rewriting to be at an acceptable level for publication.

Abstract – lacks objective (statistical) data to support its conclusions

Reply: This item is addressed below.

-----------------

Introduction – authors should remove their conclusions from the introduction – specifically “Our data shows that CHT records contain more complete and more accurate representations of patients' perceptions of their symptoms than patients' corresponding EHRs.” Please refer to the STROBE checklist if there is confusion on this point.

Reply: We removed this sentence from the Introduction.

Methods – Gender and age data should be reported in the results section. Authors need to include a description of their statistical methods.

Reply: Instructions indicated that this data was to be included in Methods.

-----------------

Results – 1. The data on location of primary pain, location of radiated pain, setting of pain onset, and frequency/duration of pain, and data for associated symptoms, use of nitroglycerin, and dimensionality is very difficult to interpret.

The descriptions are very long and would be best presented in table format, with limited narrative.

Reply: At the reviewer's suggestion, we added a table (Table 2 in the revised ms) for the data on sites of primary pain. We added a table (Table 4) for data for onset of pain during physical activity or emotional upset. (Data for other variables already is in Tables.)

-------------

Also, many values are reported as integers, where percentages would be better used.

Reply: The differences between findings in EHR and CHT data are large for the absolute findings for different variables. Absolute values for what was found enables the reader easily to replicate the statistical test data should they want to do so. We note too that there is more than 1 way to calculate per cent of some number of records because of the large numbers of missing data elements in EHRs. The reader easily can calculate any per cent value that might interest them.

---------------

Finally, only raw data is presented; testing for statistical significance in the different outcomes between the EHR and CHT data sets would greatly strengthen the results.

Reply: We have added statistical analyses for all variables. We have added a brief section in Methods describing how statistical significance was determined.

----------------

Discussion - The authors should situate their findings in the context of prior research comparing EHRs and CHTs (ie. this is the first study, or consistent with prior studies, or contrary to prior studies) .

Reply: We have added a sentence that contrasts the current work with prior work on this subject.

Attachment

Submitted filename: Response to reviewers.docx

Click here for additional data file. (10.1KB, docx)

Decision Letter 2

Amit Bahl

8 Sep 2021

Computerized History-Taking Improves Data Quality for Clinical Decision-Making. Comparison of EHR and Computer-Acquired History Data in Patients with Chest Pain.

PONE-D-21-11324R2

Dear Dr. Zakim,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Amit Bahl

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #3: The authors have satisfactorily addressed the issues I highlighted in the last review of the manuscript. I thank them for this interesting and important study.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #3: No

Acceptance letter

Amit Bahl

13 Sep 2021

PONE-D-21-11324R2

Computerized History-Taking Improves Data Quality for Clinical Decision-Making. Comparison of EHR and Computer-Acquired History Data in Patients with Chest Pain.

Dear Dr. Zakim:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Amit Bahl

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Fig. Sites of primary pain recorded in EHRs and reported by patients during CHT.

    Each line are sites of pain for the same patient. Column numbers refer to regions of the chest displayed in Fig 1. Blue-colored upper triangles for each location of pain are data from the patient’s EHR. Blue-colored lower triangles for each location of pain are data from the patient’s CHT interview. Empty upper triangles indicate that the region was not mentioned as affected by pain in EHR data. Empty lower triangles indicate that the region was not selected by the patient interacting with the image in S1 Fig. Absence of EHR data for location of chest pain indicates that the EHR did not record a specific site of chest pain. Blue-colored upper triangles in Column 0 reflect that the description of chest pain in EHR narratives was too imprecise to be associated with a specific anatomic region of the chest. Projected areas of pain on each line are data for the same patient.

    (PDF)

    Click here for additional data file. (1.1MB, pdf)
    S2 Fig

    Regions of primary pain projected onto an image of the chest for selected patients recorded in EHR data (left hand panels) and CHT data (right hand panels).

    (PDF)

    Click here for additional data file. (2.6MB, pdf)
    Attachment

    Submitted filename: Response to reviewers.docx

    Click here for additional data file. (40.7KB, docx)
    Attachment

    Submitted filename: Response to reviewers.docx

    Click here for additional data file. (10.1KB, docx)

    Data Availability Statement

    Data cannot be shared publicly because of ethical restrictions from the Swedish authorities as the data contain potentially identifying and sensitive patient information. Data could however be available for researchers who meet the criteria for access to confidential data, upon reasonable request to the authors and with permission of the Swedish Ethical Review Authority (https://etikprovningsmyndigheten.se registrator@etikprovning.se).

    Articles from PLoS ONE are provided here courtesy of PLOS

    RESOURCES