Agreement between TOAST and CCS ischemic stroke classification

Abstract

Objective:

The objective of this study was to assess the level of agreement between stroke subtype classifications made using the Trial of Org 10172 Acute Stroke Treatment (TOAST) and Causative Classification of Stroke (CCS) systems.

Methods:

Study subjects included 13,596 adult men and women accrued from 20 US and European genetic research centers participating in the National Institute of Neurological Disorders and Stroke (NINDS) Stroke Genetics Network (SiGN). All cases had independently classified TOAST and CCS stroke subtypes. Kappa statistics were calculated for the 5 major ischemic stroke subtypes common to both systems.

Results:

The overall agreement between TOAST and CCS was moderate (agreement rate, 70%; κ = 0.59, 95% confidence interval [CI] 0.58–0.60). Agreement varied widely across study sites, ranging from 28% to 90%. Agreement on specific subtypes was highest for large-artery atherosclerosis (κ = 0.71, 95% CI 0.69–0.73) and lowest for small-artery occlusion (κ = 0.56, 95% CI 0.54–0.58).

Conclusion:

Agreement between TOAST and CCS diagnoses was moderate. Caution is warranted when comparing or combining results based on the 2 systems. Replication of study results, for example, genome-wide association studies, should utilize phenotypes determined by the same classification system, ideally applied in the same manner.

Trial of Org 10172 in Acute Stroke Treatment (TOAST)¹ and the Causative Classification of Stroke (CCS)^2–4 are 2 well-established systems for classifying ischemic stroke. They use broadly similar categories of stroke diagnoses, e.g., large-vessel, small-vessel, and cardioembolic stroke, but may not necessarily be interchangeable. TOAST and CCS require different data and use different classification criteria and decision-making rules. It is therefore critical to understand the agreement rate between these 2 systems in diverse clinical and research settings. Delineation of the level of agreement between TOAST and CCS would be important to assess the validity of combining ischemic stroke subtyping using these 2 systems.

This report investigates the agreement between TOAST and CCS within the Stroke Genetics Network (SiGN), a collaborative study involving a network of international genetic research centers. This analysis is a retrospective pooled analysis of several independent research efforts, each of which enrolled patients under different research protocols.⁵ TOAST and CCS were compared by assessing identical subtype assignment and accounting for agreement by chance by using a κ statistic. Because there is no gold standard in etiologic stroke classification, we make no qualitative judgments regarding which system is “better” at subtype assignment, rather report agreement to help inform whether the 2 systems make similar assignments.

METHODS

The Stroke Genetics Network (SiGN) is a multinational collaboration with the goal of finding genetic determinants of stroke.⁵ The SiGN Study standardized the phenotyping of the cases across all genetic research centers. The CCS system was chosen to facilitate study administration because of the Web-based, semiautomated, and evidence- and rule-based nature of the system (https://ccs.mgh.harvard.edu).² In addition to classifying stroke cases by subtype, the CCS system also has the practical benefit for large consortia of standardizing and centralizing all individual data points that underlie subtype classification.² A centralized committee of 4 expert neurologists met weekly to monitor subtype data quality and site performance. This panel aimed to ensure consistency of CCS assignments across all SiGN centers but did not contribute to subtype classifications directly.

CCS subtyping of stroke cases for this report was performed based on reviews of data available in study-specific case report forms and medical records by 41 adjudicators from 10 European and 10 US sites. Adjudicators included neurology residents (n = 10), neurologists (n = 17), stroke fellows (n = 12), one nurse, and one student. Adjudicators completed an interactive online training module and a certification module available at the CCS Web site (https://ccs.mgh.harvard.edu). Data adjudication began in June 2011 and was still ongoing at the time of data analysis for this report. This study included 13,596 cases adjudicated as of July 7, 2013. A centralized committee of 4 expert neurologists met weekly to monitor data quality and site performance. Feedback was provided during subtyping to ensure quality of data.

TOAST subtypes were determined locally by site investigators following individual study protocols without benefit of central oversight. Of note, TOAST subtypes were determined using the same data sources that were available for the CCS classifications. TOAST and CCS classifications were done by different physicians and at different time points in the majority of study sites but using the same study or site-specific case report forms. CCS adjudicators were required to confirm that they were fully blind to TOAST results before they began to enter patient data into CCS.

Two deviations from the above-mentioned protocol warrant acknowledgment. One center (STGEORGE) had completed case phenotyping using the CCS system before the initiation of the SiGN Study. Therefore, this center did not conduct CCS subtyping under the oversight of the expert panel. One other study (BASICMAR) utilized a computer algorithm rather than a certified adjudicator to extract data from a study data source to populate the required fields in CCS for 389 cases of their total of 1,088 cases.

For the purposes of this report, a complete investigation was defined as having head imaging (CT, MRI, or both), vascular imaging of both the intracranial and extracranial vasculature, and a cardiac evaluation consisting of echocardiography (either transthoracic or transesophageal) unless cardiac source of embolism was identified by medical history, physical examination, or ECG. Cranial imaging (either CT or MRI) was required for inclusion in the SiGN case set.

Statistical analysis was performed using SAS 9.2 (SAS Institute, Cary, NC). Percent of absolute agreement in subtype assignments is reported. Agreement was also estimated by κ statistics, and 95% confidence intervals (CIs) are provided for interpretation. Nonweighted κ values were calculated for 5 major stroke subtypes common to both systems: large-artery atherosclerosis, cardioaortic embolism, small-artery occlusion, other causes, and undetermined causes (i.e., cryptogenic causes, unclassifiable cases because of multiple competing etiologies, and incomplete evaluation).

RESULTS

In total, 16,267 cases were enrolled in SiGN via the Web-based CCS system as of July 7, 2013. Of those, 13,596 stroke cases had subtypes classified using the TOAST system by the individual studies. Sample characteristics varied considerably by study design among the participating sites (table 1). For example, GEOS (Genetics of Early Onset Stroke)⁶ targeted recruitment among young stroke patients. WHI, the Women's Health Initiative, recruited women only and reported low levels of current smoking. The diversity of study designs and populations allows for evaluation of the agreement between TOAST and CCS across a variety of clinical and research settings.

Table 1.

Descriptive characteristics of the cases in the SiGN Study with both CCS and TOAST classifications, by genetic research center

The overall agreement between CCS and TOAST was moderate (table 2) (κ = 0.59, 95% CI 0.58–0.60), although the agreement varied across study sites (χ² [df = 19] = 782; p < 0.0001) (table 2). Agreement on specific subtypes was highest for large-artery atherosclerosis (κ = 0.71) and lowest for small-artery occlusions (κ = 0.56). Table 3 provides the cross-tabulation for subtype agreement. The 2 systems identified approximately equal number of cases as undetermined (CCS 4,673 cases and TOAST 4,664 cases), but did not show much agreement on which those undetermined cases were (table 3). The agreement of TOAST and CCS for undetermined cases was only κ = 0.44 (95% CI 0.43–0.46). Agreement between TOAST and CCS regarding undetermined cases was primarily based on cases CCS determined to be “incomplete evaluations” (κ = 0.30). Cases classified by CCS as either “cryptogenic embolism” or “unclassified” had no agreement with the TOAST category of “undetermined” (κ < 0.05).

Table 2.

Agreement statistics, κ (95% confidence interval), between CCS and TOAST for the SiGN Study

Table 3.

Cross-classification of 13,596 cases by TOAST and CCS

Stroke subtype agreement varied substantially across genetic research centers (see table 2). Part of this variability across study sites could be attributable to the variable process used to implement TOAST subtyping across sites. Additional variability could be attributable to the presence or absence of certain diagnostic evaluations available to each center (table 4). Agreement was slightly higher in the presence of vessel imaging, but slightly lower when a cardiac evaluation was performed. Agreement between TOAST and CCS was slightly lower when a complete evaluation was conducted (defined as the presence of brain imaging, cardiac evaluation, vascular imaging of the intra- and extracranial circulations). Regardless, the slight variation in agreement in the presence or absence of certain diagnostic evaluations (table 4) is not sufficient to account for the large variation seen across genetic research centers (table 2). The overall agreement reported here (κ = 0.59) belies the fact that in any particular center agreement ranged from excellent (STGEORGE κ = 0.85) to poor (BRAINS κ = 0.12).

Table 4.

Agreement statistics, κ (95% confidence interval), between CCS and TOAST for the SiGN Study by the presence of diagnostic evaluations

A sensitivity analysis was performed removing the 2 centers that deviated from the network protocol. Removing the STGEORGE and relevant BASICMAR cases resulted in lower overall agreement (κ = 0.57).

DISCUSSION

To accelerate advances in stroke treatment, prevention, and discovery of genetic and other novel risk factors, the heterogeneity of ischemic stroke must be addressed. Identifying the genetic determinants of many complex diseases has proven challenging⁷; stroke is no exception. Success is more likely to occur in large studies and active consortia of individual studies.⁸ Standardization and harmonization of phenotypes will reduce misclassification error when combining analysis efforts in consortia. In the study of stroke, this often means the standardization of subtyping among cases.

Previously reported levels of agreement between the TOAST and CCS classification systems were high.^9,10 In a prospective cohort study of North Dublin, a single physician performed data abstraction and classification in both TOAST and CCS in 381 patients with first-ever ischemic stroke. An overall agreement was not reported, but agreement between the 2 systems on specific subtypes ranged from excellent (κ = 0.95 for cardioembolism) to moderate (κ = 0.69 for other and undetermined causes). Another study of 690 ischemic stroke patients from a single center (also included in this report, STGEORGE) reported excellent overall agreement (κ = 0.85). We report a lower overall agreement between the 2 systems, with some centers witnessing much less agreement between TOAST and CCS. This could be attributable to differences among the studies in their ability to take into account the whole spectrum of variance in determining etiologic stroke subtypes. The present study, in our opinion, offers less bias because of its larger sample size, higher number of raters, multicenter design, and methodology that required a blinded assessment of CCS and TOAST subtypes. The current report provides agreement statistics stratified by study center and demonstrates variability in agreement rates with many genetic research centers having poor agreement (14 of 20 have κ < 0.60). One could interpret our results to indicate that in many research settings, the agreement between the 2 systems is quite low and data from standardized chart abstractions studies may not reflect the complexities of many practical implementations.

The lack of good agreement between TOAST and CCS is not surprising because these 2 systems use different classification criteria, definitions for subtypes, and diagnostic investigation requirements (table 5). Furthermore, their internal reliability also differs; existing studies by independent investigators demonstrate a moderate interrater reliability for the TOAST classification system with κ values ranging between 0.42 and 0.54.^11–15 In contrast, interrater reliability of the CCS ranges between 0.8 and 0.9.^2–4 These studies tended to be small and with varying methodologies. Well-powered reliability studies of both systems are still needed. While TOAST and CCS differ from each other in several ways, they both are subject to variability because of differences in adjudicators' ability to interpret diagnostic test findings as well as variability in the completeness and quality of available diagnostic investigations. Regarding the latter point, we found that agreement for a subtype was generally higher when diagnostic investigations were complete for that particular subtype. For instance, in patients with complete cardiac evaluation, the agreement for cardiac embolism was almost twice as high compared to those with incomplete cardiac investigation (κ = 0.66 vs 0.35; table 4). Likewise, agreement for small-artery occlusion was higher in the presence of brain MRI (κ = 0.61 vs 0.43). In contrast to cardiac embolism and small-artery occlusion, there was no difference in agreement for large-artery atherosclerosis between cases with and without complete vascular evaluation. This may be attributable to diagnosis of large-artery atherosclerosis being contingent on extracranial carotid artery stenosis—the most common site for large-artery atherosclerosis—which does not require a complete assessment of both extracranial and intracranial vessels.

Table 5.

Characteristics of the TOAST and CCS classification systems

These findings suggest that availability of objective diagnostic information reduces the subjective component in decision-making for stroke subtypes regardless of the classification system used. Nevertheless, too much diagnostic information provides the opportunity for differential investigator interpretation in the absence of rule-based criteria and this may in turn reduce agreement between CCS and TOAST. In line with this, we found that overall agreement rate was lower, albeit slightly, when all investigations (brain imaging, brain vascular imaging, and cardiac evaluation) were complete than when they were incomplete (κ = 0.58 vs 0.60). We also found that the agreement rate for the undetermined group was lower than rates in other etiologic categories. The undetermined group is a heterogeneous category consisting of cryptogenic stroke (undetermined-unknown), multiple competing etiologies (undetermined-unclassified), and missing diagnostic tests (incomplete evaluation). Lower agreement rate in the undetermined category reflects differences between TOAST and CCS in dealing with multiple potential causes and missing diagnostic tests. CCS takes into account the completeness of diagnostic investigations and strength of evidence favoring one mechanism over others in the presence of multiple mechanisms in identifying stroke subtypes. In contrast, TOAST provides limited guidance on these issues leading to room for opinion in many practical implementations and hence variance in subtype assignments.

The present study required a uniform Web-based training and certification of investigators to be able to perform CCS. The same standardization was not applied to the TOAST classification. The TOAST classification was done locally, using local interpretations of the TOAST classification system, and before the formation of the SiGN collaboration. This differential application of TOAST is likely responsible for the variability in agreement seen between the centers. Thus, the overall agreement captures both differences in the subtyping systems and differences in their applications. The optimal test to compare the 2 classifications would have included prospective data-quality assessment and centrally trained certified investigators for performing also the TOAST classification at the same time as the CCS. However, we address this limitation by assessing agreement separately within each genetic research center, and the agreement between the 2 systems was modest at best for a majority of them. In only 2 of the 20 studies can agreement between TOAST and CCS be classified as excellent (κ > 0.80). Of note, both of those studies used protocols for CCS assignment that deviated from the recommended consortium design. In addition, a computerized tool for TOAST classification has also been made available¹² and an additional question of interest may be how well the computerized TOAST agrees with CCS.

The agreement between CCS and TOAST reported here is lower than previously reported. The low agreement between the 2 systems described here simply means that the 2 systems classify stroke cases in different categories, although perhaps unfortunately, the names of the categories are similar. The practical implication of this finding is that combining or comparing classifications across systems should proceed with caution, and where possible, rephenotyping should be encouraged before combining data. For example, replication of results from genetic association studies should be made using phenotypes from the same classification system. A large benefit of the CCS system is the standardization of input and output data across cases from different sites. This feature allows for flexible analysis and further stratification of stroke phenotypes and hence promises utility in genetic studies such as SiGN.

GLOSSARY

CI

confidence interval

CCS

Causative Classification of Stroke

NINDS

National Institute of Neurological Disorders and Stroke

SiGN

Stroke Genetics Network

TOAST

Trial of Org 10172 Acute Stroke Treatment

AUTHOR AFFILIATIONS

From the University of Maryland School of Medicine (P.F.M.), Division of Endocrinology, Diabetes and Nutrition; University of Maryland (S.J.K., J.W.C., M.H.); Massachusetts General Hospital (H.A.), Departments of Neurology and Radiology, Harvard Medical School; Mayo Clinic (R.D.B.), Rochester, MN; Mayo Clinic Florida (J.F.M.), Jacksonville; University of Miami (T.R., M.K., R.L.S.); Albert Einstein College of Medicine (S.W.-S., R.C.K., D.L.L.); University of Cincinnati (D.W.); Skåne University Hospital (G.A., H.D.); Massachusetts General Hospital and Broad Institute (A.B., J. Rosand, N.S.R.); University of Alabama Birmingham (D.A.B.); NINDS-NIH (R.C., K.G.); University Medical Center Utrecht (P.I.W.d.B.), the Netherlands; Institute for Stroke and Dementia Research (M.D., S.T.), Klinikum der Universität München, Ludwig-Maximilians-Universität, Munich; Munich Cluster for Systems Neurology (SyNergy) (M.D.), Germany; Neuroscience Institute (R.P.G., L.R.P.), Saint Francis Medical Center; Sahlgrenska Academy at University of Gothenburg (C.J., K.J., A.P., P.R.); IMIM-Hospital Universitari del Mar (J.J.-C., J. Roquer); Medical University Graz (P.K.); Laboratory of Neurobiology (R.L., V.T.), Vesalius Research Center, VIB, Leuven; Experimental Neurology (Department of Neurosciences) and Leuven Research Institute for Neuroscience and Disease (LIND) (R.L., V.T.), University of Leuven (KU Leuven); Department of Neurology (R.L., V.T.), University Hospitals Leuven, Belgium; Oxford University (L.L.); Lund University and Skåne University Hospital (A.L.); St. George's University of London (H.S.M., R.V.); Jagiellonian University Medical College (J.P.); Nuffield Department of Clinical Neurosciences (P.M.R.), University of Oxford; John Radcliffe Hospital (P.M.R.), Oxford, UK; Imperial College London (P.S.); Jagiellonian University Medical College (A.S.); University of Edinburgh (C.S.); and University of Virginia Health System (B.B.W.), Departments of Neurology and Health Evaluation Sciences.

AUTHOR CONTRIBUTIONS

Patrick F. McArdle: drafting/revising the manuscript, study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data, statistical analysis, study supervision. Steven J. Kittner: drafting/revising the manuscript, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, study supervision, obtaining funding. Hakan Ay: drafting/revising the manuscript, study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data, statistical analysis, study supervision. Robert D. Brown, Jr.: drafting/revising the manuscript, study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval. James F. Meschia: drafting/revising the manuscript, study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data, study supervision. Tatjana Rundek: drafting/revising the manuscript, study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, critical revision of the manuscript. Sylvia Wassertheil-Smoller: drafting/revising the manuscript, study concept or design, accepts responsibility for conduct of research and will give final approval, acquisition of data, obtaining funding. Daniel Woo: drafting/revising the manuscript, study concept or design, accepts responsibility for conduct of research and will give final approval, acquisition of data, study supervision, obtaining funding. Gunnar Andsberg: drafting/revising the manuscript, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval. Alessandro Biffi: analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval. David A. Brenner: analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, medical chart review of more than 600 stroke patients, computer entry into the CCS system. John W. Cole: drafting/revising the manuscript, study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval. Roderick Corriveau: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, study supervision. Paul I.W. de Bakker: study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data. Hossein Delavaran: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data. Martin Dichgans: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data. Raji P. Grewal: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data, study supervision. Katrina Gwinn: drafting/revising the manuscript, study concept or design, accepts responsibility for conduct of research and will give final approval, study supervision. Mohammed Huq: analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data. Christina Jern: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data. Jordi Jimenez-Conde: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data. Katarina Jood: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data. Robert C. Kaplan: study concept or design, accepts responsibility for conduct of research and will give final approval, acquisition of data. Petra Katschnig: analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval. Michael Katsnelson: analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data. Daniel L. Labovitz: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data. Robin Lemmens: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data. Linxin Li: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data. Arne Lindgren: drafting/revising the manuscript, study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data, obtaining funding. Hugh S. Markus: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data. Leema R. Peddareddygari: analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data, CCS adjudicator. Annie Pedersén: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data. Joanna Pera: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data. Petra Redfors: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data. Jaume Roquer: analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data. Jonathan Rosand: drafting/revising the manuscript, study concept or design, accepts responsibility for conduct of research and will give final approval, acquisition of data, obtaining funding. Natalia S. Rost: drafting/revising the manuscript, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data. Peter M. Rothwell: study concept or design, accepts responsibility for conduct of research and will give final approval, contribution of vital reagents/tools/patients, acquisition of data. Ralph L. Sacco: drafting/revising the manuscript, accepts responsibility for conduct of research and will give final approval, acquisition of data, study supervision. Pankaj Sharma: drafting/revising the manuscript, study concept or design, accepts responsibility for conduct of research and will give final approval, acquisition of data. Agnieszka Slowik: analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data. Cathie Sudlow: drafting/revising the manuscript, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data. Vincent Thijs: drafting/revising the manuscript, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data. Steffen Tiedt: analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data. Raffaella Valenti: analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data. Bradford B. Worrall: drafting/revising the manuscript, study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data, study supervision, obtaining funding. The writing group included Dr. McArdle, Dr. Kittner, Dr. Ay, Dr. Brown, Dr. Meschia, Dr. Rundek, Dr. Wassertheil-Smoller, Dr. Woo, and Dr. Worrall.

STUDY FUNDING

The SiGN Network was funded by a cooperative agreement grant from the National Institute of Neurological Disorders and Stroke (NINDS): U01 NS069208. Funding for network members is presented in appendix e-1 on the Neurology® Web site at Neurology.org.

DISCLOSURE

The authors report no disclosures relevant to the manuscript. Go to Neurology.org for full disclosures.