Skip to main content
NCBI home page
PLOS One logoLink to PLOS One
. 2021 Feb 25;16(2):e0247571. doi: 10.1371/journal.pone.0247571

Observational study of agreement between attending and trainee physicians on the surprise question: “Would you be surprised if this patient died in the next 12 months?”

Christopher J Yarnell 1,2,3,*, Laura M Jewell 4, Alex Astell 5, Ruxandra Pinto 6, Luke A Devine 2,7, Michael E Detsky 2,3, James Downar 8,9, Roy Ilan 10, Shail Rawal 7,11, Natalie Wong 3,7,12, John J You 13, Rob A Fowler 1,3,6,14
Editor: Jason Chia-Hsun Hsieh15
PMCID: PMC7906409  PMID: 33630939

Abstract

Background

Optimal end-of-life care requires identifying patients that are near the end of life. The extent to which attending physicians and trainee physicians agree on the prognoses of their patients is unknown. We investigated agreement between attending and trainee physician on the surprise question: “Would you be surprised if this patient died in the next 12 months?”, a question intended to assess mortality risk and unmet palliative care needs.

Methods

This was a multicentre prospective cohort study of general internal medicine patients at 7 tertiary academic hospitals in Ontario, Canada. General internal medicine attending and senior trainee physician dyads were asked the surprise question for each of the patients for whom they were responsible. Surprise question response agreement was quantified by Cohen’s kappa using Bayesian multilevel modeling to account for clustering by physician dyad. Mortality was recorded at 12 months.

Results

Surprise question responses encompassed 546 patients from 30 attending-trainee physician dyads on academic general internal medicine teams at 7 tertiary academic hospitals in Ontario, Canada. Patients had median age 75 years (IQR 60–85), 260 (48%) were female, and 138 (25%) were dependent for some or all activities of daily living. Trainee and attending physician responses agreed in 406 (75%) patients with adjusted Cohen’s kappa of 0.54 (95% credible interval 0.41 to 0.66). Vital status was confirmed for 417 (76%) patients of whom 160 (38% of 417) had died. Using a response of “No” to predict 12-month mortality had positive likelihood ratios of 1.84 (95% CrI 1.55 to 2.22, trainee physicians) and 1.51 (95% CrI 1.30 to 1.72, attending physicians), and negative likelihood ratios of 0.31 (95% CrI 0.17 to 0.48, trainee physicians) and 0.25 (95% CrI 0.10 to 0.46, attending physicians).

Conclusion

Trainee and attending physician responses to the surprise question agreed in 54% of cases after correcting for chance agreement. Physicians had similar discriminative accuracy; both groups had better accuracy predicting which patients would survive as opposed to which patients would die. Different opinions of a patient’s prognosis may contribute to confusion for patients and missed opportunities for engagement with palliative care services.

Introduction

Provision of optimal end-of-life care requires reliable identification of patients likely to die in the near future. Unfortunately, few reliable bedside tools are available to identify patients who may benefit from goals-of-care discussions due to near-term risk of death [1,2]. Predictive models based on illness severity are intended for use at the population level and may be misleading when applied to individual patients [1,3–5]. While some patients suffer from terminal diseases with predictable trajectories, many suffer from multiple comorbidities conferring an unpredictable trajectory towards death [6]. Uncertainty in patient prognosis can increase patient distress, impair communication between physicians and patients, and delay appropriate adoption of a palliative approach [7]. Further, the risk of death may not correlate with other important aspects of patient care such as uncontrolled symptoms or psychological distress [8,9]. Clinician estimates of a patient’s risk of death are fallible, although accuracy may correlate with clinician experience, and these estimates likely have a large influence on end-of-life care planning [8,10–15].

One way to gain insight into clinician estimates of prognosis is to investigate how prognostic accuracy varies with training by comparing the prognostic estimates of attending and trainee physicians. Studying differences in prognostic estimates between trainee and attending physicians provides a way to investigate the development of prognostic skills, including whether or not accuracy improves with clinical training. Prognostic discordance and covariates associated with discordance may identify an opportunity for education by highlighting the patient characteristics that make prognosis more difficult. Discordance between trainee and attending physician prognoses also has practical implications because trainee physicians play an important role for inpatients at academic health centres, where discordance may cause confusion in clinical plans, mixed messages for patients, and may partially explain observed differences between received and documented patient end-of-life care preferences [16,17]. The prevalence of discordance between attending and trainee physician prognoses is unknown.

The surprise question is a simple tool to assess patient prognosis and screen for patients who may benefit from end-of-life care [18–21]. The surprise question asks “Would you be surprised if this patient died in the next 12 months?” and a response of “No” is intended to identify a patient with potentially unmet palliative care needs [22]. In the absence of a gold standard for unmet palliative care needs, most evaluation of the surprise question has instead focused on mortality. A “no” response is correlated with increased mortality rates and discriminatory ability is similar to other prognostic indices intended for hospitalized older adults [1,2,23,24]. Hospitalized patients may be at higher risk of unmet palliative care needs and mortality because of the situation leading to admission and potential gaps in the social safety net that may precipitate hospital admission [25–27]. Improving the identification of patients at high risk of dying can improve clinical care through providing more certainty to patients, more clarity for clincians making recommendations about invasive interventions or investigations, and triggering earlier appropriate activation of palliative care services. Therefore we compared attending and trainee physician responses to the surprise question in a general internal medicine population to assess the extent and prevalence of discordance as well as the prognostic value of the surprise question with respect to mortality in this population.

Methods

Study population

This was a prospective cohort study of general internal medicine inpatients and their corresponding physicians at 7 tertiary and quaternary academic hospitals in Ontario. The hospitals varied in size (median 463 beds, range 256 to 1325). Admissions to the general internal medicine ward at each hospital occurred primarily through the emergency department as opposed to transfers from other hospitals. The majority of general internal medicine patients at each hospital were cared for by the academic teaching teams involved in this study.

Attending physicians were all certified in internal medicine through the Royal College of Physicians and Surgeons, Canada. Trainee physicians were all second (or higher) year residents in a Canadian internal medicine residency program training for the same certification. There is no mandatory requirement for specific palliative care clinical exposure in internal medicine training in Canada, although residents may participate in clinical electives to gain exposure to this area and “care of the dying” is one of the objectives of training [28].

Recruitment occurred at one vanguard site in 2014 (to hone study procedures and case reporting forms) and at the other sites between April 2017 and October 2018. All patients admitted to general internal medicine teaching teams and their corresponding physician teams were eligible for inclusion and identified on the day of survey. There were no exclusions based on patient language or cognitive status.

Surprise question responses

Clinicians were asked “Would you be surprised if this patient died in the next 12 months?” for each of the patients for whom they were responsible, totalling two responses (one trainee physician response and one attending physician response) per patient. Trainee and attending physicians were surveyed independently with no knowledge of each others’ responses. The year of medical school graduation and duration (in days) on the current service was recorded for each respondent.

Patient data collection

Baseline patient data were collected from the hospital chart including demographic data (gender, age), baseline pre-hospital data gleaned from the clinical notes (type of residence, functional status, marital status), medical information (admitting diagnosis, comorbidities, admission creatinine), length of stay on the day the surprise question was administered and documented cardiopulmonary resuscitation (CPR) status.

12-month patient mortality data collection

At every site, vital status at 12 months was recorded for each patient by reviewing the hospital electronic record or publicly available online death databases and obituaries. At all but one site, if the 12-month vital status could not be determined from the chart or death databases and obituaries, then a research assistant sent a letter to the patient’s address on file offering the opportunity to opt out of a follow-up phone call. A research assistant then attempted to contact by phone all patients with undetermined vital status who did not opt out of the follow up phone call. At one site, no attempts to contact patients after discharge were permitted by the research ethics board.

Ethics

This study was approved by the Research Ethics Boards at Sunnybrook Health Sciences Centre, St Michael’s Hospital, Sinai Health Systems and University Health Network, Kingston General Hospital, and Hamilton General Hospital. The waiver for initial patient participation was important to ensure a broad enrolment population that did not exclude vulnerable subgroups such as patients with cognitive impairment, interpretation needs, or mental illness for whom the surprise question may have different operating characteristics and for whom there may be a differential risk of death or unmet palliative care needs [29].

Statistical analysis

The primary analysis assessed agreement between attending and trainee physician 12-month surprise question responses using an adjusted Cohen’s kappa measure of interrater reliability (chance-corrected agreement) to account for clustering by physician dyad [30,31]. This was calculated from the posterior probabilities of each response using a multinomial Bayesian regression model with random effects allowing for clustering by physician (Supporting information) [32]. Modeling the clustering by physician pair was important in order to identify the extent of variability in agreement across physician dyads [33]. The prior distributions were chosen to be minimally informative. Discordance was assessed by the relative risk of attending physicians responding “No” in discordant cases using the posterior probability distributions.

The accuracy of surprise question responses with respect to 12-month mortality was derived from 2-by-2 tables including calculation of positive and negative likelihood ratios. Bayesian multilevel models were also used to assess the prognostic characteristics of the 12-month surprise question with respect to 12-month mortality for attending physicians and trainee physicians accounting for clustering by physician dyad. This also included calculation of sensitivity, specificity, and likelihood ratios.

A further exploratory model assessed for association between kappa value and selected clinically relevant patient factors: age, sex, functional status, CPR status, admission diagnosis, and comorbidity. In these models, patients with missing data were excluded. Age was modeled with splines using 4 knots.

The Bayesian modelling program Stan was used via the statistical programming language R using the package brms [34–36]. Minimally informative priors were used. Models were run for 4000–7500 iterations with 1000–5000 iterations warmup, 4 chains, and 4 cores. Chains, r-hat values and parameter distributions were inspected to assess model fit. Posterior medians with 95% credible intervals were reported. Code and Monte Carlo diagnostics are available in the Supporting information.

Patients with missing surprise question response data were excluded. The impact of missing mortality data were assessed by repeating the mortality analysis with multiple imputation using 25 datasets [37].

Results

Demographic and clinical characteristics

We gathered surprise question responses on 546 patients from 30 attending-trainee physician dyads at 7 hospitals. The median patient age was 75 years (IQR 60–85), 260 (48%) had female sex, median length-of-stay before survey was 8 days (IQR 3–19) and most common admission diagnoses included pneumonia (10%), delirium (9%), and congestive heart failure (9%) (additional details in Table 1). Three patients (not included in the 546) had incomplete surprise question response data.

Table 1. Baseline characteristics.

Characteristic Number (%)
Hospitals
    Total hospitals 7
    Patients enrolled per site (median [range]) 73 [56–104]
    Physician dyads per hospital (median [range]) 4 [3–6]
Physicians
    Total dyads 30
    Patients per dyad (median [IQR]) 18 [16–20]
Attending physician
    Years since medical graduation (median [IQR]) 13 [8–23]
    Days on current service (median [IQR]) 10 [9–14]
Trainee physician
    Years since medical graduation (median [IQR]) 2 [1–3]
    Days on current service (median [IQR]) 18 [14–21]
Patients
    Total patients 546 (100%)
Patient age (years)
    <30 27 (5%)
    30–44 32 (6%)
    45–59 83 (15%)
    60–74 126 (23%)
    75–89 232 (42%)
    ≥ 90 44 (8%)
    Missing 2 (0.4%)
Sex
    Female 260 (48%)
    Male 284 (52%)
    Missing 2 (0.4%)
Residence Type (prior to hospitalization)
    House or Apartment 422 (77%)
    Retirement Home or Long-term Care 84 (15%)
    No fixed address 19 (3%)
    Other 21 (4%)
Function (IADLs/ADLs)
    Independent for all 179 (33%)
    Dependent for some 216 (40%)
    Unknown 151 (28%)
Admitting diagnoses
    Cardiovascular 78 (14%)
    Neurologic 75 (14%)
    Infectious 156 (29%)
    Acute kidney injury/metabolic abnormality 72 (13%)
    Other 165 (30%)
Comorbidities
    Cardiovascular 401 (73%)
    Respiratory 88 (16%)
    Chronic kidney disease 68 (12%)
    Cancer 127 (23%)
CPR status
    Full code or not documented 368 (67%)
    No CPR 178 (33%)
Open in a new tab

Trainee physicians were internal medicine residents in their second or third year of postgraduate training and the median time on service before survey was 18 days (IQR 14–21). Among attending physicians, the median time between survey and graduating medical school was 13 years (IQR 8–23) and the median time on service before survey was 10 days (IQR 9–14).

Surprise question results

Among the 546 patients, attending physicians answered “No, I would not be surprised if this patient died in the next 12 months” (“No”) for 368 patients (67%), while trainees answered “No” for 316 patients (58%). Attending and trainee physicians had the same response of “No” for 272 patients (50%) and the same response of “Yes” for 134 patients (24%) with discordant responses in the remaining 140 patients (26%) (Table 2).

Table 2. 12-month surprise question responses of attending and trainee physicians.

Attending Physician
Trainee Physician “No, I would not be surprised if this patient died in the next 12 months.” “Yes, I would be surprised if this patient died in the next 12 months.”
“No, I would not be surprised if this patient died in the next 12 months.” 272 (49%) 44 (8%) 316 (57%)
“Yes, I would be surprised if this patient died in the next 12 months.” 96 (18%) 134 (25%) 230 (43%)
368 (67%) 178 (33%) 546 (100%)
Open in a new tab

Twelve-month surprise question responses by attending and trainee physicians showed moderate agreement with a Cohen’s kappa statistic (adjusted for clustering by physician dyad) of 0.54 (95% credible interval 0.41–0.66) for the average physician dyad. The Cohen’s kappa without adjustment for clustering by physician dyad was 0.46 (95% confidence interval 0.38–0.54). In discordant cases, attending physicians were more likely to answer “No” than trainee physicians (relative risk 2.36, 95% credible interval 1.22–4.98). There was moderate variation in kappa across the physician dyads (Fig 1). Results for the surprise question with respect to hospital discharge are available in the Supporting information.

Fig 1. Histogram and density plot of median kappa values by physician dyad.

Fig 1

Open in a new tab

This figure shows the distribution of kappa values according to physician dyad by histogram (bars) with a density plot overlay (light blue). Most clinician dyads had a kappa value above 0.4, but some outliers had kappa values between 0.1 and 0.3.

Associations between agreement and patient covariates

Associations between patient characteristics and physician responses were investigated through a larger adjusted Bayesian multinomial regression model. Two patients were excluded for missing covariate data. Age was modeled with restricted cubic splines using 4 knots and showed a nonlinear relationship between median kappa and age (Fig 2) with the highest agreement at approximately age 40 and the lowest agreement at approximately age 75. Agreement decreased with presence of a respiratory or cancer comorbidity and increased with presence of infection as admitting diagnosis (Table 3), although credible intervals were wide throughout the exploratory results.

Fig 2. Agreement between attending and trainee physician surprise question responses by patient age.

Fig 2

Open in a new tab

This figure shows the median Cohen’s kappa (adjusted for clustering by physician dyad) as a black line with 95% credible intervals as surrounding blue ribbon. Age was modeled with restricted cubic splines using 4 knots. Other variables were set to: Female, independent, full code, infection as admitting diagnosis, and presence of a cardiovascular comorbidity only.

Table 3. Agreement between attending and trainee physician by patient subgroup, adjusted for multiple clinical covariates.

Patient characteristic Median Kappa (95% credible interval)
Age (years)
    30 0.69 (0.34 to 0.89)
    45 0.72 (0.41 to 0.90)
    60 0.59 (0.27 to 0.82)
    75 0.47 (0.14 to 0.74)
    90 0.64 (0.31 to 0.87)
Sex
    Female 0.59 (0.26 to 0.81)
    Male 0.61 (0.35 to 0.84)
Function (IADLs/ADLs)
    Independent for all 0.59 (0.28 to 0.83)
    Dependent for some 0.55 (0.19 to 0.82)
    Unknown 0.56 (0.16 to 0.80)
CPR status
    Full code and not documented 0.57 (0.28 to 0.83)
    Not for CPR 0.46 (0.02 to 0.78)
Admitting diagnosis
    Cardiovascular 0.59 (0.29 to 0.81)
    Neurologic 0.50 (0.21 to 0.73)
    Infectious 0.71 (0.42 to 0.87)
    AKI/metabolic 0.41 (-0.07 to 0.75)
    Other 0.31 (-0.12 to 0.67)
Comorbidities
    Cardiovascular 0.46 (0.06 to 0.80)
    Respiratory 0.17 (-0.20 to 0.64)
    Chronic kidney disease 0.42 (0.00 to 0.82)
    Cancer 0.29 (-0.04 to 0.75)
Open in a new tab

Unless otherwise noted, the baseline characteristics used for calculating median kappas and 95% credible intervals were: Age 60 years, female, independent, full code, cardiovascular admitting diagnosis and cardiovascular comorbidity.

Comparing surprise question results and mortality

Vital status at 12 months was confirmed for 417 patients (76%). Of these patients, 160 (38%) had died. Across sites, confirmation of mortality status ranged from 47% to 99% and observed confirmed mortality rates ranged from 25% to 53%.

Among the patients for whom mortality data were available, the probability of death given the surprise question responses was calculated using Bayesian logistic regression accounting for clustering by physician dyad. The probability of death according to surprise question responses was 57% (95% credible interval 49% to 65%) for both attending and trainee physicians responding “No” and 8.3% (95% credible interval 3.7% to 15%) for both attending and trainee physicians responding “Yes.” The probability of death given a response of “No” was 49% (95% credible interval 42% to 56%) for attending physicians and 54% (95% credible interval 46% to 62%) for trainee physicians. Conversely, the probability of death given a response of “Yes” to the surprise question was 14% (95% credible interval 8% to 21%) for attending physicians and 17% (95% credible interval 11% to 23%) for trainee physicians. The analyses found very similar results after multiple imputation of missing data (Table E1 and Table E2 in the Supporting information).

The adjusted likelihood ratio for death given a response of “No” to the surprise question was 1.84 (95% credible interval 1.55 to 2.22) for trainee physicians and 1.51 (95% credible interval 1.30 to 1.72) for attending physicians. The corresponding negative likelihood ratios were 0.31 (95% credible interval 0.19 to 0.50) and 0.27 (95% confidence interval 0.13 to 0.45). Further details about sensitivity and specificity are available in Table E3 in the Supporting information.

Discussion

This study of 546 general medicine inpatients across 30 attending-trainee physician dyads of physicians showed that attending and trainee physicians had moderate agreement on the surprise question “Would you be surprised if this patient died in the next 12 months?” The adjusted Cohen’s kappa was 0.54, which means that the average dyad agreed on 54% of patients after removing the patients where agreement occurred by chance [30]. The classification of this value of kappa as “moderate” is based on convention [31] and a 54% agreement rate after removing chance agreements is not reassuring in the setting of identifying patients at high risk of mortality and unmet palliative care needs.

The variation across dyads was similar in magnitude to variation by clinical characteristics in exploratory analyses. Agreement by Cohen’s kappa in this study was similar to that seen in another study [15] comparing predictions of nurses and physicians, suggesting that moderate agreement on prognosis may be common to other dyads of clinicians beyond trainee-attending dyads. Taken together, these findings imply that discussion of patient prognosis between multidisciplinary clinical team members is essential to ensure appropriate engagement of palliative care services and to avoid mixed messages to patients and family members.

The surprise question may be useful to “rule out” a high risk of mortality, but it is not sufficient as a standalone screening measure for identifying patients who have high risk of mortality. A response of “Yes” on the surprise question was associated with low 12-month mortality in this and other settings, but the sensitivity and specificity remain inconsistent and unsuited to a screening tool [2,38]. The lowest 12-month mortality rate was seen in patients where both attending and trainee physician responded “Yes” to the surprise question, similar to other studies where combining predictions of multiple clinical team members yielded the best predictions [15]. In contrast to previous studies investigating physician-estimated prognoses which found correlation between accuracy and level of training [11,39–42], attending physician predictions were not more accurate than trainee physician predictions with respect to mortality in our study. This could mean that both attending and trainee physicians would benefit from educational interventions focused on prognosis. Alternatively, accurate prognostication may require clinicians to combine clinical insights with novel tools not yet in clinical use, such as automated screening tools derived from electronic medical records [43–46].

This study has strengths including a pragmatic approach, inclusive enrollment criteria, paired design, multicenter data, and statistical methods that account for clustering. The simplicity of the surprise question allowed us to include all patients within our inclusion criteria with no planned or unplanned systematic exclusions on the day of survey. Our paired design minimized patient-level confounding and comparing to the most senior trainee as comparator minimized error due to medical inexperience. The choice of Bayesian modeling permitted a more sophisticated analysis of the Cohen’s kappa coefficient which has not previously been used in analyses of the surprise question [2,38].

The main limitation of this study is the lack of data on unmet palliative care needs such as uncontrolled pain or nausea, psychological distress about the dying process, or ignorance about the available options for end-of-life care. This limitation is also present in other research on the surprise question [22]. It is unknown whether the attending or trainee physician is more likely to be correct with respect to palliative care needs when there is discordance. A related limitation is a lack of corresponding qualitative data including information about the reasoning behind the responses of each participant. It is also unknown whether the two physicians were aware when they did not agree, or if this was discussed for any of the patients in the study.

A more fundamental limitation is that the surprise question is a subjective instrument that integrates a healthcare practitioner’s expertise, knowledge, and personal biases [47]. This study includes uncertainty both from the intrinsic uncertainty in estimating a patients’ prognosis and from the uncertainty in how clinicians interpret this uncertainty in responding to the surprise question. The validity of the surprise question with respect to its intended purpose of identifying patients with palliative care needs remains unclear. Even if the surprise question reliably identifies those patients, identification alone is necessary but not sufficient for meeting those needs [48,49]. Future research may need to focus on identification of unmet palliative care needs as opposed to predicting mortality, as these needs may be a better marker of potential benefit from a palliative approach and provide tangible targets for the clinical team.

Conclusion

Using the surprise question to measure prognosis, the average general internal medicine attending and trainee physician agreed on the prognosis of only 54% of their patients after correcting for chance agreement. These data remind clinicians of the subjectivity in formulating patient prognoses, and the importance of routine discussion of patient prognosis between team members in order to ensure a coherent clinical plan including clear communication with patients and appropriate engagement of palliative care services.

Supporting information

S1 File

(DOCX)

Click here for additional data file. (95.4KB, docx)
S2 File

(DOCX)

Click here for additional data file. (16.6KB, docx)
S3 File

(CSV)

Click here for additional data file. (97.4KB, csv)

Acknowledgments

An appreciative thanks to the following people: Carol Mantle and Marilyn Swinton for assistance with data collection and project coordination at Hamilton Health Sciences. Katherine Allan for assistance with data collection and project coordination at St Michael’s Hospital. Ellen Koo for assistance with data collection and project coordination at Toronto General and Toronto Western Hospitals. Nicole Marinoff for assistance with project coordination at Sunnybrook Health Sciences Center. Julia Kruizinga for assistance with data collection at Kingston Health Science Centre.

Data Availability

All relevant deidentified data are within the manuscript and its Supporting Information files.

Funding Statement

The study was funded by a Resident Research Grant from the Physician Services Incorporated Foundation (Dr Yarnell, R15-40). Dr Yarnell is also funded by the Canadian Institutes for Health Research CGS-M, the Clinician Investigator Program, and the Eliot Phillipson Clinician Scientist Training Program at the University of Toronto. The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources including the PSI Foundation. No endorsement by any of the funding agencies is intended or should be inferred. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

Decision Letter 0

Jason Chia-Hsun Hsieh

1 Dec 2020

PONE-D-20-25533

Observational study of agreement between attending and trainee physicians on the surprise question: “Would you be surprised if this patient died in the next 12 months?”

PLOS ONE

Dear Dr. Yarnell,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

ACADEMIC EDITOR: Minor revision is required for the current manuscript. Please respond to the questions from the reviewers.

Please submit your revised manuscript by Jan 15 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Jason Chia-Hsun Hsieh, M.D. Ph.D

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

Minor revision is required for the current manuscript. Please respond to the questions from the reviewers.

3. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for sharing with me this fascinating paper to review that chimes closely with my clinical and research interests in a recent cluster trail of an intervention to better serve patients with multiple morbidities whose situations are clinically uncertain and where they were considered to be at risk of dying during their hospital say despite receiving treatment. We experienced innumerable problems in practically operationalising this issue [1].

My question as to why it's so important to be able to predict which patients are at risk of dying in the next 12 months is as present now as it was then. Perhaps, greater emphasis of its utility is required in the opening paragraphs. If we are to focus on patient need (physical and well as psychosocial issues) and need alone i.e. the ability or capacity to benefit for clinical intervention that is seen to be effective, where is the case for essentially tossing a coin? My point here is that prognostication is a contested field and I think this needs to be unpicked a bit more critically in the opening paragraphs. My view is prognostic models vary in levels of sophistication, ranging from clinical intuition to more intricate multivariate statistical models that combine multiple factors to yield an assessment [2]. We discovered that where ‘risk of dying’ is focused on there is a risk of two sampling biases being present due to the unknown (or the inconsistent) manner in which health professionals currently interpret risk – the question you are addressing in this study albeit between different levels of seniority Firstly, the unpredictable and often unreliable identification of potential cases and secondly their exclusion. Another concern I have is that although models to enhance the identification of dying patients using prognostic models are improving for patients with cancer [3] [4] [5] (a lot of this literature is indeed referred to), there is far less consensus on methods to assess patients with non-malignant conditions, which are more common in many health care settings studies [6, 7]. Forgive my rant but these are indeed important areas to focus on at some point in this paper.

I like the aim of this study, however, t I would be grateful if the authors would justify why the comparison between attending and trainee physicians? Intuitively I think I know why but it would be useful to shed light on any underlying hypotheses the authors wish to examine. For example, does the ability and confidence in prognostication get better with experience

The methods are described elegantly. I like the term vanguard site. We often refer to this as a pilot site, but I can live with that. It would be useful to have a little more contextual information about the hospitals concerning the populations they serve and how many patients they typically see each year, and at the very least how many patients the wards serve.

A question. Are you able to shed light on the education that the trainees might have previously received about assessing risk and prognostication? Is this information available from their curriculum?

The 12-month patient mortality data collection section of this paper is important. Were there any ethical issues raised by contacting the patient’s address on file offering the opportunity to opt-out of a follow-up phone call? This is not alluded to with the subsequent phone call.

Good to me that vulnerable group were also considered to be important to include in this study. Vulnerable groups are often excluded from study and palliative care study in particular. May be worth citing a reference or two to support this section. Kipnis among others have written about this is, as have I historically.

A clear statistical plan presented clearly and elegantly. Very replicable and transparent.

Results.

An ‘n’ for table 1 would be useful to include

Page 12, it appears no information was collected regarding training in prognostication

Table 2 works well. Could also be presented in the form of a figure?

Useful supplement files. Thank you.

It perhaps would have been useful to shed light why its important to understand the possible relationship between levels of agreement and co-variates. Perhaps this goes back to the original aim of the study and hypotheses that could have been set to test subsequently.

Discussion

Page 17, line 328-329. I agree with this statement that clinical team members need to communicate with each to better effect to identify patients where there is a risk of dying in the next 12mopnths. I take it this would include other members of the multi-professional team, not just physicians. It still begs the question of whether the focus should be on the needs of patients irrespective of prognosis. This is linked to the limitations of this study where the authors state there was a lack of data on the unmet palliative care needs of patients. What do they mean by this?

I agree that prognostication offers a pragmatic solution to identify a group of patients who might benefit from specialist palliative and end of life care. However, given that this approach relies heavily on subjectivity (the authors acknowledge this) in prognostication perhaps it should be just plain avoided? Instead, greater emphasis is placed on objective clinical indicators, for example, poor performance status scores, the presence and severity of cognitive impairment, weight loss, and dysphagia, and of course the presence of distressing symptoms that can be ameliorated.

1. Koffman, J., et al., Managing uncertain recovery for patients nearing the end of life in hospital: a mixed-methods feasibility cluster randomised controlled trial of the AMBER care bundle. Trials, 2019. 20(1): p. 506.

2. Yourman, L.C., et al., Prognostic indices for older adults: a systematic review. Jama, 2012. 307(2): p. 182-92.

3. Stone, P. and S. Lund, Predicting prognosis in patients with advanced cancer. Annals of Oncology, 2006. 18(6): p. 971-976.

4. Stone, P., et al., Patients' reports or clinicians' assessments: which are better for prognosticating? BMJ Supportive & Palliative Care, 2012. 2(3): p. 219-223.

5. White, N., et al., A systematic review of predictions of survival in palliative care: How accurate are clinicians and who are the experts? PLoS ONE, 2016. 11(8): p. e0161407.

6. Nutter, A.L., T. Tanawuttiwat, and M.A. Silver, Evaluation of 6 Prognostic Models Used to Calculate Mortality Rates in Elderly Heart Failure Patients With a Fatal Heart Failure Admission. Congestive Heart Failure, 2010. 16(5): p. 196-201.

7. Glimelius, B., Palliative medicine ? A research challenge Acta Oncologica, 2000. 39(8): p. 891-893.

Reviewer #2: An interesting and well written report. I do have some comments.

Abstract, page 4 line 114 – where it reads “We undertook to determine agreement…” should it read “We undertook this study to determine agreement…”

Methods, page 8 line 191 – how was this done exactly? Were attendings and trainees in the same room at the same time? Were they asked to reply verbally or in writing? If this is the case could that not allow for biases, as trainees might be more prone to agree, or fear to disagree with their attendings? This would be a limitation of the study

Results, page 13, lines 249-252: it would have been important to know and to state in the paper if any of the participating physicians had palliative care training (either basic or advance) given that, identifying patients with palliative care needs and patients at the end of their disease trajectory might be done more easily after being exposed to that specific training, as would be to make the case for answering “no” or “yes” to the surprise question. Not knowing about specific palliative care training can be a potential limitation of this study. Additionally, it could also be potentially used to make the point authors raise in the discussion, page 17, lines 331,332 “The surprise question may be useful to “rule out” a high risk of mortality, but it is not sufficient as a standalone screening measure for identifying patients who have high risk of mortality.”; lines 358, 359 “Even if the surprise question reliably identifies those patients, identification alone is necessary but not sufficient for meeting those needs” and in the conclusion, lines 365, 366 “appropriate engagement of palliative care services.”

Conclusion: although I agree with authors’ conclusion, there is a vital component missing, which I feel should be mentioned in this section, which is the importance of palliative care training for physicians working in these services.

Typos: “data was” instead of “data were” occurs throughout the paper. Please amend this.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Jonathan Koffman

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2021 Feb 25;16(2):e0247571. doi: 10.1371/journal.pone.0247571.r002

Author response to Decision Letter 0

24 Dec 2020

Reviewer 1

REVIEWER: Thank you for sharing with me this fascinating paper to review that chimes closely with my clinical and research interests in a recent cluster trail of an intervention to better serve patients with multiple morbidities whose situations are clinically uncertain and where they were considered to be at risk of dying during their hospital say despite receiving treatment. We experienced innumerable problems in practically operationalising this issue [1].

RESPONSE: Thank you for sharing your experience. We have added this work as a citation to support the negative impact of uncertainty on patients, physicians, and communication between the two. Citation was added at the end of the following new sentence: “Uncertainty in patient prognosis can increase patient distress, impair communication between physicians and patients, and delay appropriate adoption of a palliative approach.”

REVIEWER: My question as to why it's so important to be able to predict which patients are at risk of dying in the next 12 months is as present now as it was then. Perhaps, greater emphasis of its utility is required in the opening paragraphs.

RESPONSE: We have updated the Intro paragraph 3 to greater emphasise the utility of identifying which patients are at risk of dying in the next 12 months. We added: “Improving the identification of patients at high risk of dying can improve clinical care through providing more certainty to patients, more clarity for clinicians making recommendations about invasive interventions or investigations, and triggering earlier appropriate activation of palliative care services.”

REVIEWER: If we are to focus on patient need (physical and well as psychosocial issues) and need alone i.e. the ability or capacity to benefit for clinical intervention that is seen to be effective, where is the case for essentially tossing a coin? My point here is that prognostication is a contested field and I think this needs to be unpicked a bit more critically in the opening paragraphs.

RESPONSE: We agree that prognostication is difficult, nuanced, unlikely to be fully captured by a binary indicator, and that a patient’s potential benefit from involvement of palliative care or consideration of end-of-life planning is not solely determined by their prognosis in terms of time. We have added the following sentence to the opening paragraph to unpack these complex notions: “Further, the risk of death may not correlate with other important aspects of patient care such as uncontrolled symptoms or psychological distress.”

REVIEWER: My view is prognostic models vary in levels of sophistication, ranging from clinical intuition to more intricate multivariate statistical models that combine multiple factors to yield an assessment [2]. We discovered that where ‘risk of dying’ is focused on there is a risk of two sampling biases being present due to the unknown (or the inconsistent) manner in which health professionals currently interpret risk – the question you are addressing in this study albeit between different levels of seniority Firstly, the unpredictable and often unreliable identification of potential cases and secondly their exclusion.

RESPONSE: We agree that one consequence of the experimental design of this project is that we incorporate error both at the level of clinicians making an accurate prediction and at the level of clinicians interpreting how to respond to the surprise question given their prediction of risk. We have tried to emphasize this further in the discussion section.

REVIEWER: Another concern I have is that although models to enhance the identification of dying patients using prognostic models are improving for patients with cancer [3] [4] [5] (a lot of this literature is indeed referred to), there is far less consensus on methods to assess patients with non-malignant conditions, which are more common in many health care settings studies [6, 7].

RESPONSE: The inclusion criteria for this study were broad to ensure the information is applicable to patients with both cancer and non-cancer diagnoses. We have highlighted this point in the discussion with the following sentence added to the final limitations paragraph: “This study includes uncertainty both from the intrinsic uncertainty in estimating a patients’ prognosis and from the uncertainty in how clinicians interpret this uncertainty in responding to the surprise question.”

REVIEWER: I like the aim of this study, however, t I would be grateful if the authors would justify why the comparison between attending and trainee physicians? Intuitively I think I know why but it would be useful to shed light on any underlying hypotheses the authors wish to examine. For example, does the ability and confidence in prognostication get better with experience.

RESPONSE: Thank you for the feedback. We have endeavoured to better justify the rationale for comparison between trainee and attending physicians in the introduction section. The second paragraph of the introduction has been rewritten as follows:

“One way to gain insight into clinician estimates of prognosis is to investigate how prognostic accuracy varies with training by comparing the prognostic estimates of attending and trainee physicians. Studying differences in prognostic estimates between trainee and attending physicians provides a way to investigate the development of prognostic skills, including whether or not accuracy improves with clinical training. Prognostic discordance and factors associated with discordance may identify an opportunity for education, and may highlight the patients for whom prognosis is most difficult. Discordance between trainee and attending physician prognoses also has practical implications because trainee physicians play an important role for inpatients at academic health centres, where discordance may cause confusion in clinical plans, mixed messages for patients, and may partially explain observed differences between received and documented patient end-of-life care preferences. The prevalence of discordance between attending and trainee physician prognoses is unknown.”

REVIEWER: The methods are described elegantly. I like the term vanguard site. We often refer to this as a pilot site, but I can live with that. It would be useful to have a little more contextual information about the hospitals concerning the populations they serve and how many patients they typically see each year, and at the very least how many patients the wards serve.

RESPONSE: We have added contextual information about the hospitals to the methods section. The first paragraph of the Study population subsection now reads:

“This was a prospective cohort study of general internal medicine inpatients and their corresponding physicians at 7 tertiary and quaternary academic hospitals in Ontario. The hospitals varied in size (median 463 beds, range 256 to 1325). Admissions to the general internal medicine ward at each hospital occurred primarily through the emergency department as opposed to transfers from other hospitals. The majority of general internal medicine patients at each hospital were cared for by the academic teaching teams involved in this study.”

REVIEWER: A question. Are you able to shed light on the education that the trainees might have previously received about assessing risk and prognostication? Is this information available from their curriculum?

RESPONSE: We have added information about the training backgrounds of the attending and trainee physicians:

“Attending physicians were all certified in internal medicine through the Royal College of Physicians and Surgeons, Canada. Trainee physicians were all second (or higher) year residents in a Canadian internal medicine residency program training for the same certification. There is no mandatory requirement for specific palliative care clinical exposure in internal medicine training in Canada, although residents may participate in clinical electives to gain exposure to this area and “care of the dying” is one of the objectives of training.”

REVIEWER: The 12-month patient mortality data collection section of this paper is important. Were there any ethical issues raised by contacting the patient’s address on file offering the opportunity to opt-out of a follow-up phone call? This is not alluded to with the subsequent phone call.

RESPONSE: We did not encounter any issues during the phone calls. However, as mentioned in the methods, one site did not allow any contact of patients with outcomes not apparent in the hospital chart. The research ethics board for that site felt that this would only be possible if patients had provided written, informed consent to a follow up phone call during the index admission. We did not opt to go that route for both scientific reasons (this would potentially exclude many vulnerable patients with cognitive impairment or non-English first languages) and logistical reasons (lacking funding to pursue such a larger project). We have updated the last sentence in the subsection 12-month patient mortality data collection to read: “At one site, no attempts to contact patients after discharge were permitted by the research ethics board.”

REVIEWER: Good to me that vulnerable group were also considered to be important to include in this study. Vulnerable groups are often excluded from study and palliative care study in particular. May be worth citing a reference or two to support this section. Kipnis among others have written about this is, as have I historically.

RESPONSE: Thank you for the suggestion. I enjoyed reading your 2009 paper on this topic and I’m glad you pointed it out. We have added it as a citation to our subsection entitled “Ethics.”

REVIEWER: A clear statistical plan presented clearly and elegantly. Very replicable and transparent.

RESPONSE: Thank you

REVIEWER: An ‘n’ for table 1 would be useful to include

RESPONSE: Done

REVIEWER: Page 12, it appears no information was collected regarding training in prognostication

RESPONSE: No formal information was collected. All participating trainees were internal medicine residents training for Royal College of Physicians and Surgeons Canada certification.

REVIEWER: Table 2 works well. Could also be presented in the form of a figure?

RESPONSE: Thank you. For now, we have left this in the classic “2 by 2 table” format, but if the editors prefer this be transformed to a bar graph we are happy to attempt it.

REVIEWER: It perhaps would have been useful to shed light why its important to understand the possible relationship between levels of agreement and co-variates. Perhaps this goes back to the original aim of the study and hypotheses that could have been set to test subsequently.

RESPONSE: We have added text to the methods and results section to highlight the usefulness of performing a multilevel analysis of this clustered data.

REVIEWER: Page 17, line 328-329. I agree with this statement that clinical team members need to communicate with each to better effect to identify patients where there is a risk of dying in the next 12mopnths. I take it this would include other members of the multi-professional team, not just physicians.

RESPONSE: We agree and have clarified the statement to ensure that the importance of communication with the multi-professional team is evident.

REVIEWER: It still begs the question of whether the focus should be on the needs of patients irrespective of prognosis. This is linked to the limitations of this study where the authors state there was a lack of data on the unmet palliative care needs of patients. What do they mean by this?

RESPONSE: We agree that perhaps a different approach could have been to ask physicians which of their patients they felt were likely to have unmet palliative care needs, such as uncontrolled pain or nausea, uncertainty and anxiety about end-of-life care, or a desire to discuss and alleviate deeply held fears about the dying process. The process of conducting a project focused on prognosticating mortality has demonstrated to us that this approach does not necessarily capture unmet patient needs, which may be a better marker of who would benefit from increased involvement with palliative care resources. We have added these concepts to the discussion section.

REVIEWER: I agree that prognostication offers a pragmatic solution to identify a group of patients who might benefit from specialist palliative and end of life care. However, given that this approach relies heavily on subjectivity (the authors acknowledge this) in prognostication perhaps it should be just plain avoided? Instead, greater emphasis is placed on objective clinical indicators, for example, poor performance status scores, the presence and severity of cognitive impairment, weight loss, and dysphagia, and of course the presence of distressing symptoms that can be ameliorated.

RESPONSE: We agree, and we have adjusted the discussion section to try and better capture this point.

1. Koffman, J., et al., Managing uncertain recovery for patients nearing the end of life in hospital: a mixed-methods feasibility cluster randomised controlled trial of the AMBER care bundle. Trials, 2019. 20(1): p. 506.

4. Stone, P., et al., Patients' reports or clinicians' assessments: which are better for prognosticating? BMJ Supportive & Palliative Care, 2012. 2(3): p. 219-223.

5. White, N., et al., A systematic review of predictions of survival in palliative care: How accurate are clinicians and who are the experts? PLoS ONE, 2016. 11(8): p. e0161407.

6. Nutter, A.L., T. Tanawuttiwat, and M.A. Silver, Evaluation of 6 Prognostic Models Used to Calculate Mortality Rates in Elderly Heart Failure Patients With a Fatal Heart Failure Admission. Congestive Heart Failure, 2010. 16(5): p. 196-201.

7. Glimelius, B., Palliative medicine ? A research challenge Acta Oncologica, 2000. 39(8): p. 891-893.

Reviewer 2

REVIEWER: Abstract, page 4 line 114 – where it reads “We undertook to determine agreement…” should it read “We undertook this study to determine agreement…”

RESPONSE: We have adjusted the abstract.

REVIEWER: Methods, page 8 line 191 – how was this done exactly? Were attendings and trainees in the same room at the same time? Were they asked to reply verbally or in writing? If this is the case could that not allow for biases, as trainees might be more prone to agree, or fear to disagree with their attendings? This would be a limitation of the study

RESPONSE: We have described the process in more detail. The trainee and attending physicians were not aware of each others’ responses.

REVIEWER: Results, page 13, lines 249-252: it would have been important to know and to state in the paper if any of the participating physicians had palliative care training (either basic or advance) given that, identifying patients with palliative care needs and patients at the end of their disease trajectory might be done more easily after being exposed to that specific training, as would be to make the case for answering “no” or “yes” to the surprise question. Not knowing about specific palliative care training can be a potential limitation of this study. Additionally, it could also be potentially used to make the point authors raise in the discussion, page 17, lines 331,332 “The surprise question may be useful to “rule out” a high risk of mortality, but it is not sufficient as a standalone screening measure for identifying patients who have high risk of mortality.”; lines 358, 359 “Even if the surprise question reliably identifies those patients, identification alone is necessary but not sufficient for meeting those needs” and in the conclusion, lines 365, 366 “appropriate engagement of palliative care services.”

RESPONSE: We have added details about palliative care training to the methods section of the paper.

REVIEWER: Conclusion: although I agree with authors’ conclusion, there is a vital component missing, which I feel should be mentioned in this section, which is the importance of palliative care training for physicians working in these services.

RESPONSE: Thank you. We have reinforced the importance of palliative care training for ensuring high-quality end-of-life care.

REVIEWER: Typos: “data was” instead of “data were” occurs throughout the paper. Please amend this.

RESPONSE: Thank you. We have amended this.

Decision Letter 1

Jason Chia-Hsun Hsieh

10 Feb 2021

Observational study of agreement between attending and trainee physicians on the surprise question: “Would you be surprised if this patient died in the next 12 months?”

PONE-D-20-25533R1

Dear Dr. Yarnell,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Jason Chia-Hsun Hsieh, M.D. Ph.D

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

The questions were answered adequately.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for taking the time to read through my comments and co-reviewer for that matter. I believe you have now adequately addressed all the issues I raised.

Reviewer #2: All points raised were addressed by authors. The paper is improved. An extremely well written report. Useful for clinical practice and for research. I have no further comments to add.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Jonathan S Koffman

Reviewer #2: No

Acceptance letter

Jason Chia-Hsun Hsieh

11 Feb 2021

PONE-D-20-25533R1

Observational study of agreement between attending and trainee physicians on the surprise question: “Would you be surprised if this patient died in the next 12 months?”

Dear Dr. Yarnell:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Jason Chia-Hsun Hsieh

Academic Editor

PLOS ONE

Articles from PLoS ONE are provided here courtesy of PLOS

RESOURCES