Abstract
Introduction:
There is no standard for categorizing reasons for death in those who achieve return of spontaneous circulation (ROSC) after cardiac arrest but die before hospital discharge. Categorization is important for comparing outcomes across studies, assessing benefits of interventions, and developing quality-improvement initiatives. We developed and tested a method for categorizing reasons for death after cardiac arrest in both in-hospital (IHCA) and out-of-hospital (OHCA) arrests.
Methods:
Single-center, retrospective, cohort study of patients with ROSC after IHCA or OHCA between 2008 and 2017 who died before hospital discharge. Traumatic arrests and patients with "do-not-resuscitate" orders prior to their arrest were excluded. Two investigators assigned each patient to one of five predefined reasons for death. Interrater reliability was assessed using Fleiss’ kappa. For final categorization, discrepancies were resolved by a third investigator.
Results:
There were 182 IHCA and 226 OHCA included. There was substantial agreement between raters (kappa of 0.62 and 0.61 for IHCA and OHCA, respectively). Reasons for death for IHCA and OHCA were: neurological withdrawal of care (27% vs 73%), comorbid withdrawal of care (36% vs 4%), refractory hemodynamic shock (25% vs 17%), respiratory failure (1% vs 3%), and sudden cardiac death (11% vs 4%). The differences in reasons for death among the two groups were significant (p-value < 0.001).
Conclusions:
Categorizing reasons for death after cardiac arrest with ROSC is feasible using our proposed categories, with substantial inter-rater agreement. Neurologic withdrawal of care is much less common in IHCA than OHCA, which may have implications for further research.
Introduction
Targeting treatment to improve survival of a given disease requires understanding why patients die from that disease. Although most studies on cardiac arrest report survival and rates of good neurologic outcome, causes of death in patients who achieve sustained return of spontaneous circulation (ROSC) but do not survive to hospital discharge are not well characterized. (1-4)
In the out-of-hospital cardiac arrest (OHCA) population, neurologic injury has been found to be the most common cause of death in patients who achieve sustained ROSC but remain comatose after resuscitation, as well as being a major source of disability among survivors. (5-8) Consequently, post-arrest care guidelines focus heavily on minimizing neurologic injury. (9) Although it is unclear whether neurologic injury is a prominent cause of death after in-hospital cardiac arrest (IHCA), guidelines on treatment and prognostication for initial survivors of IHCA and OHCA are currently essentially the same, and based predominantly on data from OHCA studies. (10)
The Utstein criteria were designed to create uniformity in the type and quality of data collected in cardiac arrest studies. (11) This has had a major influence on improving data quality and comparability of patients across studies. However, cause of death in patients who achieve sustained ROSC but do not survive to hospital discharge is one outcome that remains largely undocumented. The Utstein committee advised that cause of death be reported, but made this a supplemental rather than a core outcome, perhaps recognizing that methods for determining this outcome have not been well established. Information on cause of death could provide valuable mechanistic information to assess benefits of targeted interventions or quality-improvement initiatives and help guide the development of new treatments.
In this study, we sought to develop and test a standard methodology for categorizing and reporting the reasons for death in patients who achieve sustained ROSC following IHCA or OHCA but do not survive to hospital discharge. We also compared the relative frequency of each reason for death between the IHCA and the OHCA population.
Methods
Population
This was a retrospective, single-center observational cohort study of cardiac arrest patients at a tertiary care center between January 2008 and December 2017. Patients were included if they achieved sustained ROSC (>20 min) following an IHCA or OHCA and died before hospital discharge. Traumatic arrests and patients for whom care was immediately withdrawn after ROSC due to delayed recognition of do-not-resuscitate (DNR) status were excluded. Patients enrolled in ongoing randomized trials were also excluded, due to the possibility of interventions to which the investigators were still blinded affecting reasons for death. The study was approved by the our medical center’s Institutional Review Board.
Categories
Five categories of reasons for death following cardiac arrest were developed by consensus among a multidisciplinary group of investigators based on their clinical and research experience. The team included experts in Emergency Medicine, Pulmonary and Critical Care, and Anesthesia/Surgical Critical Care. The categories were neurologic withdrawal of care, comorbid withdrawal of care, refractory hemodynamic shock, refractory respiratory failure and sudden cardiac death. Detailed definitions of the categories are provided in Table 1.
Table 1.
Sudden cardiac death | Recurrent cardiac arrest without return of spontaneous circulation with or without extraordinary measure (e.g. ECPR) in place*. | R† |
U‡ | ||
Progressive, refractory hemodynamic shock | Progressive, refractory hemodynamic shock despite aggressive ICU care, or withdrawal of care based on same. Hemodynamically stable patients (e.g. maintaining their mean arterial blood pressure) on aggressive ICU care (e.g. full vasopressor support) should not be included in this category. | R |
U | ||
Respiratory failure | Respiratory failure or withdrawal of care based on same. Respiratory failure may be related to hypoxemia, hypercapnia or the combination thereof. Patients who are oxygenating sufficiently on highest ventilator settings should not be included in this category. | R |
U | ||
Neurological withdrawal of care | Withdrawal of care based on expectations of a poor neurological recovery based on brain imaging, a neurologic exam, or a formal opinion of a neurologist stating that the prognosis for neurologic recovery is very poor. If an assessment off sedation is not done, there must be other evidence of severe neurologic injury (e.g. severe cerebral edema or herniation). | R |
U | ||
Comorbid withdrawal of care | Withdrawal of care or refusal of life-sustaining therapy based on the expectation of a poor quality of life. This may be related to a preexisting or newly discovered terminal illness or other serious medical condition (e.g. dementia or cancer). To categorize patients with multiple potential causes of death (e.g. refractory hemodynamic shock, respiratory failure and multi system organ failure), an attempt should be made to identify the primary cause of death or reason for withdrawal of care. | R |
U |
ECPR, Extracorporal Cardiopulmonary Resuscitation; ICU, Intensive Care Unit
Should not be considered when return of spontaneous circulation was not sustained following the index cardiac arrest.
Related to the index cardiac arrest. Reasons for death considered related to the index cardiac arrest should be those directly related to the pathophysiology leading to the arrest, the effects of decreased perfusion during the arrest, or conditions arising from the management of or the procedures carried out during the resuscitation or immediate post-resuscitation period. Time elapsed since the cardiac arrest can be considered in making this selection, but should not be the sole determinant.
Unrelated to the index cardiac arrest or the underlying cause of the index cardiac arrest but related to a later event or secondary injury occurring during hospitalization. Such events might include but are not limited to e.g. new thromboembolism, new sepsis, procedural complications or adverse drug reactions.
Data collection
Two investigators independently reviewed the hospital records of all IHCA and OHCA patients, including pre-hospital documentation if available, assigning each patient to one of the five predefined categories. Patients could not be placed in more than one category; therefore, patients meeting criteria for more than one category were categorized by their primary reason for death. Reviewers also designated each death as related or unrelated to the initial arrest. Detailed definitions of related and unrelated death appears in Table 1. After categorizing the patients in an initial training set (Training set I, Figure 1) and testing the agreement between the investigators, minor modifications were made to the phrasing of the categories before proceeding to categorize the patients in a second training set (Training set II, Figure 1). Based on finding substantial agreement between the investigators after training set II, no further modifications were made to the categories and the remaining patients were categorized (Figure 1). After calculation of the inter-rater agreement, all discrepancies were resolved by discussion between the two reviewers, with additional assessment by a third investigator when required. Final calculation of the frequency of each reason for death was done after consensus was obtained. Patients from the first training set were not included in the assessment of interrater reliability. They were, however, recategorized according to the finalized categories and included in the frequency calculation.
In addition to reasons for death, information on patient demographics, past medical history, characteristics of the initial arrest (date, location, rhythm, downtime) and post-arrest care were collected from the patients' medical record.
Statistical Analysis
The study population was summarized by descriptive statistics, using median with interquartile range for continuous variables and counts and frequencies for categorical variables. Categorical data were compared with Chi-Square Test or Fisher’s exact test, and continuous data with Wilcoxon Rank-Sum Test. The interrater reliability was assessed using Fleiss’ Kappa. A Kappa between 0.41 and 0.60 was considered moderate agreement and 0.61-0.80 was considered substantial agreement. (12, 13) All analyses were two-sided with a significance level of 0.05 and performed using R Studio.
Results
There were 504 patients between 2008 and 2017 who achieved sustained ROSC following cardiac arrest and died prior to hospital discharge. Of these, 69 were excluded due to being enrolled in ongoing randomized trials. An additional 27 were excluded due to traumatic etiology of arrest, post-arrest recognition of DNR status and consequent withdrawal of care, or lack of documentation, making determining arrest characteristics and reason for death impossible. This left 408 patients for the final analysis, of which 182 were IHCA and 226 were OHCA. (Figure 1)
IHCA patients were older, and more likely to have cancer or congestive heart failure than OHCA patients. The initial rhythm was non-shockable in 79% of IHCA and 68% of OHCA cases respectively (p=0.02), and median downtime was significantly shorter for IHCA (10 minutes [IQR: 5-17] vs 25 minutes [IQR: 15-37], p<0.001). Arrests were deemed cardiac in etiology in <50% of patients in both groups, and post-arrest cooling was more frequently done after OHCA (82% vs 39%, p<0.001). Other patient and arrest characteristics are described in Table 2. Time to death was similar in OHCA and IHCA. Survival times by reason for death in IHCA and OHCA are represented in Figure 2.
Table 2.
IHCA (n = 182) |
OHCA (n = 226) |
P-value | |
---|---|---|---|
Sex, female (%) | 76 (42) | 85 (38) | 0.43 a |
Age, median years (IQR) | 71 (60, 81) | 65 (53, 77) | 0.003 |
Race (%) | <0.001 a | ||
White | 124 (70) | 123 (55) | |
African-American | 26 (15) | 20 (9) | |
Asian | 7 (4) | 6 (3) | |
Other | 2 (1) | 3 (1) | |
Unknown | 19 (11) | 70 (32) | |
Cancer (%) | 57 (32) | 27 (12) | <0.001 a |
Congestive heart failure (%) |
62 (34) | 41 (18) | <0.001 a |
Etiology of initial arrest (%) | 0.84 a | ||
Primary cardiac | 59 (33) | 77 (35) | |
Non-primary cardiac | 96 (54) | 110 (51) | |
Other | 24 (13) | 30 (14) | |
OHCA location (%) a | |||
Private residence | 134 (60) | ||
Nursing home | 24 (11) | ||
Public space | 38 (17) | ||
Ambulance | 19 (8) | ||
Other | 10 (4) | ||
Witnessed (%) | <0.001 a | ||
Yes | 172 (95) | 145 (65) | |
No | 9 (5) | 75 (34) | |
Unknown | 0 (0) | 3 (1) | |
Initial rhythm (%) | 0.02 a | ||
Shockable | 37 (20) | 64 (28) | |
Non-shockable | 143 (79) | 154 (68) | |
Unknown | 1 (1) | 7 (3) | |
Downtimeb, median minutes (IQR) | 10 (5, 17) | 25 (15, 37) | <0.001 |
TTMc (%) | 71 (39) | 185 (82) | <0.001 |
Time to deathd, median days (IQR) | 3.5 (1.2, 9.3) | 3.6 (1.8, 5.9) | 0.85 |
IQR, Interquartile Range
Sex: 2 missing data in OHCA; Race: 8 missing data, 4 in each group; Cancer: 5 missing data, 2 in OHCA and 3 in IHCA; Congestive heart failure: 5 missing data, 3 in OHCA and 2 in IHCA; Etiology of initial arrest: 12 missing data, 9 in OHCA and 3 in IHCA; OHCA Location: 1 missing data; Witnessed: 4 missing data, 3 in OHCA and 1 in IHCA; Initial rhyhm: 2 missing data, 1 in each group
Duration from pulselessness until sustained ROSC
Targeted temperature management
Time from hospital admission to death, in days
After initial review of the 48 patients in training set I, the Kappa was 0.58 for IHCA and 0.45 for OHCA. After discussion and refinement of the criteria for each category (see Supplementary Table 1 for specific modifications), 6 IHCA and 4 OHCA patients were reviewed for training set II, resulting in a Kappa of 0.71 for both IHCA and OHCA. The final Kappa, including the 10 patients from training set II and the remaining 350 patients, was 0.62 for IHCA and 0.61 for OHCA. In cases for which reviewers did not initially choose the same category, consensus was reached on almost all patients after discussion between reviewers, requiring input from a third reviewer in only 5 cases. (Figure 1)
The reasons for death for IHCA and OHCA, respectively, were: neurological withdrawal of care (27% vs 73%), comorbid withdrawal of care (36% vs 4%), refractory hemodynamic shock (25% vs 17%), respiratory failure (1% vs 3%), and sudden cardiac death (11% vs 4%) (p-value for reasons for death in IHCA vs OHCA< 0.001). (Figure 3) The relative percentages of reasons for death were not significantly different when separated by shockable or nonshocakable initial rhythm. (Supplementary Table 2) Discrepancy between the investigators was 37.9% when categorizing IHCA and 18.1% when categorizing OHCA. Discrepancies were more common in both IHCA and OHCA when patients were categorized as dying from comorbid withdrawal of care and refractory hemodynamic shock than neurologic injury.(Fig 3) The reason for death was found to be related to the initial arrest in 98.4% of IHCA and 100% of OHCA cases.
Discussion
In the current work, we found that the reason for death in patients admitted to an ICU after an IHCA or OHCA could be categorized successfully using our five predefined categories. Neurologic withdrawal of care was the primary reason for death after OHCA, supporting the findings of past studies and validating neurologic injury as a primary target of post-arrest care. (5, 8, 14) Causes of death were more variable after IHCA however, with neurologic withdrawal of care accounting for only one third of cases, similar to what was found in the one prior study evaluating mode of death after both IHCA and OHCA that we are aware of by Laver et al. (8) This distinction raises questions about the utility of extrapolating from OHCA when forming guidelines for IHCA care. (9) A more recent study by Chen et al also included both IHCA and OHCA patients.(15) Those investigators focused primarily on aetiology of arrest, and while they did report mode of death, this was broken down by arrest aetiology and not by arrest location (OHCA vs IHCA). Their overall distributions of mode of death for the combined cardiac arrest cohort appear similar to our findings.
Although our results are similar overall to those obtained by Laver et al, our categories differ in some details from those used by the prior investigators. Three categories were utilized in the earlier study: cardiovascular (including recurrent arrest and cardiogenic shock), neurologic (similar criteria to those in the current work) and multiorgan failure (cardiovascular or neurologic failure plus hypoxemia, oliguria or evidence of infection and 3/4 SIRS criteria). Our expert panel chose to include five categories to allow for greater granularity in causes of death. The larger number of categories allowed for differentiation between causes such as recurrent sudden cardiac arrest, withdrawal of care based on persistent (but supportable) cardiogenic shock from which it was thought unlikely that a patient would have a meaningful recovery due to other comorbidities, and refractory cardiogenic shock, with inability to maintain a life-sustaining blood pressure despite maximal support. These examples could potentially be classified as “cardiovascular” under the prior criteria, but fall in separate categories by our proposed method. These differences are potentially important to consider when assessing whether a particular intervention is likely to be helpful.
There was no predominant cause of death after IHCA, with neurologic withdrawal of care, comorbid withdrawal of care and refractory shock being comparable in frequency. This may reflect the heterogeneity of the IHCA population, with many patients arresting in the context of other underlying acute illnesses. However, the setting of IHCA may also play a role. IHCA is more often witnessed, generally by medical personnel, so time from arrest to start of cardiopulmonary resuscitation (CPR) is likely much shorter than in the out-of-hospital population. In our data, total downtime (defined as the total number of minutes without a pulse from arrest until sustained ROSC) was significantly shorter in IHCA. The shorter time to ROSC may explain the lower rates of devastating neurologic injury after IHCA, while higher rates of comorbid conditions and concurrent acute illness likely contribute to the higher frequency of comorbid withdrawal of care in this population.
The relatively low percentage of IHCA patients who die from neurologic causes is important information when prioritizing future research endeavors in this patient population. Although neuroprotective treatments such as targeted temperature management remain important to help optimize the neurologic recovery of survivors, interventions targeting hemodynamic shock and the resulting multiorgan failure may be more likely to increase the number of patients who survive long enough to potentially benefit from neuroprotection. Additionally, the large percentage of people who die from comorbid withdrawal of care after IHCA raises the possibility that in some cases better goals of care discussions may be the most effective intervention in order to avoid overly aggressive care in patients unlikely to survive such an event.
In addition to determining the frequency of different reasons for death, a major goal of this work was to establish a standard methodology for reporting this outcome. Despite the recommendations of the Utstein committee, reasons for death in those who achieve sustained ROSC but do not survive to hospital discharge have been inconsistently documented in the literature, especially in IHCA studies. Lack of reporting is likely due to both the difficulty in obtaining this endpoint, which often requires extensive medical record review, and to the lack of commonly accepted categories with which to classify cause of death. We devised the categories used here with input from clinicians and researchers with expertise in cardiac arrest, with the goal of developing a method with good interrater reliability and validity, to facilitate collection of this data point in future studies.
We utilized the Fleiss’ kappa as a widely-accepted method to assess interrater agreement. What kappa level constitutes good agreement is somewhat arbitrary, but > 0.60, which we achieved for both IHCA and OHCA, is generally considered substantial agreement. (12, 13) Interestingly, although the kappas were similar, there was discrepancy of opinion in a significantly higher percentage of IHCA cases. One characteristic of kappa calculation is that when categories are more evenly distributed the kappa will often be higher with a lower percentage of agreement than in a case where one category is the predominant one. (16, 17) In this analysis, the predominant cause of death after OHCA was neurologic withdrawal of care, while there was no such dominant category in IHCA. This may account for the similar kappas despite the higher percentage of disagreement on IHCA cases. In our process, we found that discussion between the two reviewers led to rapid attainment of consensus in all but 5 cases. This suggests that, with additional training and practice utilizing these categories, reviewers could attain a significantly higher concordance.
Our study has the following important limitations. This was a single-center study, and validating the performance of this method of categorization at other centers is required before broader adaptation. Although arrest data was collected prospectively, reasons for death were ascertained retrospectively, and thus rely on the quality of the clinical documentation. For complex, often critically ill patients, there may be some subjectivity in assessing the primary cause of death. Patients enrolled in our ongoing randomized trials were not included in this analysis and most of these were OHCA subjects. Although this was a relatively small percentage of the overall post-arrest population, this could still have biased our cohort. This study was done in the United States, and there are considerable differences in medical services and medical culture between countries. Critical care interventions in the United States are offered to patients regardless of frailty or underlying chronic illness unless that patient has specified that they do not want such measures, and this is not the case in every country. This may mean that reasons for death such as comorbid withdrawal of care would be less prevalent in some countries, as patients undergoing cardiac resuscitation and ICU admission in the United States might not receive such interventions elsewhere. Conversely, in some countries withdrawal of life-sustaining treatment is almost never undertaken, which could also alter these results. Hence, testing of these categories and frequencies of reasons for death in other institutions and countries is necessary.
Conclusions
In summary, we report a methodology for categorizing reason for death after IHCA and OHCA in those who achieve sustained ROSC but do not survive to hospital discharge. The primary reason for death was neurological withdrawal of care after OHCA but this is not the case after IHCA. Categorizing reasons for death may be important for investigators when targeting an IHCA as opposed to an OHCA population.
Supplementary Material
Acknowledgements:
We would like to thank Francesca Montillo for administrative help with document preparation and submission.
Funding Support
Dr. Moskowitz is supported by the National Institutes of Health (K23GM128005-01)
Dr. Berg is supported by the National Institutes of Health (K23 HL128814-03)
Dr. Donnino is supported by the National Institutes of Health (R01HL136705, K24HL127101)
Conflict of interest statement: Dr. Moskowitz is supported by the National Institutes of Health (K23GM128005-01), Dr. Berg is supported by the National Institutes of Health (K23 HL128814-03). Dr. Donnino is supported by the National Institutes of Health (R01HL136705, K24HL127101). None of the authors have any relevant conflicts of interest to disclose.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- 1.Girotra S, Nallamothu BK, Spertus JA, Li Y, Krumholz HM, Chan PS. Trends in survival after in-hospital cardiac arrest. N Engl J Med. 2012;367(20):1912–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Girotra S, van Diepen S, Nallamothu BK, Carrel M, Vellano K, Anderson ML, et al. Regional Variation in Out-of-Hospital Cardiac Arrest Survival in the United States. Circulation. 2016;133(22):2159–68. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Schluep M, Gravesteijn BY, Stolker RJ, Endeman H, Hoeks SE. One-year survival after in-hospital cardiac arrest: A systematic review and meta-analysis. Resuscitation. 2018;132:90–100. [DOI] [PubMed] [Google Scholar]
- 4.Nolan JP, Berg RA, Callaway CW, Morrison LJ, Nadkarni V, Perkins GD, et al. The present and future of cardiac arrest care: international experts reach out to caregivers and healthcare authorities. Intensive Care Med. 2018;44(6):823–32. [DOI] [PubMed] [Google Scholar]
- 5.Dragancea I, Rundgren M, Englund E, Friberg H, Cronberg T. The influence of induced hypothermia and delayed prognostication on the mode of death after cardiac arrest. Resuscitation. 2013;84(3):337–42. [DOI] [PubMed] [Google Scholar]
- 6.Cronberg T, Lilja G, Rundgren M, Friberg H, Widner H. Long-term neurological outcome after cardiac arrest and therapeutic hypothermia. Resuscitation. 2009;80(10):1119–23. [DOI] [PubMed] [Google Scholar]
- 7.Sandroni C, D'Arrigo S, Nolan JP. Prognostication after cardiac arrest. Crit Care. 2018;22(1):150. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Laver S, Farrow C, Turner D, Nolan J. Mode of death after admission to an intensive care unit following cardiac arrest. Intensive Care Med. 2004;30(11):2126–8. [DOI] [PubMed] [Google Scholar]
- 9.Callaway CW, Donnino MW, Fink EL, Geocadin RG, Golan E, Kern KB, et al. Part 8: Post-Cardiac Arrest Care: 2015 American Heart Association Guidelines Update for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care. Circulation. 2015;132(18 Suppl 2):S465–82. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Moskowitz A, Holmberg MJ, Donnino MW, Berg KM. In-hospital cardiac arrest: are we overlooking a key distinction? Curr Opin Crit Care. 2018;24(3):151–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Perkins GD, Jacobs IG, Nadkarni VM, Berg RA, Bhanji F, Biarent D, et al. Cardiac Arrest and Cardiopulmonary Resuscitation Outcome Reports: Update of the Utstein Resuscitation Registry Templates for Out-of-Hospital Cardiac Arrest: A Statement for Healthcare Professionals From a Task Force of the International Liaison Committee on Resuscitation (American Heart Association, European Resuscitation Council, Australian and New Zealand Council on Resuscitation, Heart and Stroke Foundation of Canada, InterAmerican Heart Foundation, Resuscitation Council of Southern Africa, Resuscitation Council of Asia); and the American Heart Association Emergency Cardiovascular Care Committee and the Council on Cardiopulmonary, Critical Care, Perioperative and Resuscitation. Resuscitation. 2015;96:328–40. [DOI] [PubMed] [Google Scholar]
- 12.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74. [PubMed] [Google Scholar]
- 13.Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005;85(3):257–68. [PubMed] [Google Scholar]
- 14.Lemiale V, Dumas F, Mongardon N, Giovanetti O, Charpentier J, Chiche JD, et al. Intensive care unit mortality after cardiac arrest: the relative contribution of shock and brain injury in a large cohort. Intensive Care Med. 2013;39(11):1972–80. [DOI] [PubMed] [Google Scholar]
- 15.Chen N, Callaway CW, Guyette FX, Rittenberger JC, Doshi AA, Dezfulian C, et al. Arrest etiology among patients resuscitated from cardiac arrest. Resuscitation. 2018;130:33–40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Kottner J, Streiner DL. The difference between reliability and agreement. J Clin Epidemiol. 2011;64(6):701–2; author reply 2. [DOI] [PubMed] [Google Scholar]
- 17.de Vet HC, Terwee CB, Knol DL, Bouter LM. When to use agreement versus reliability measures. J Clin Epidemiol. 2006;59(10):1033–9. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.