Development and Reliability of a User-friendly Multicenter Phenotyping Application for Hemorrhagic and Ischemic Stroke

Abstract

Background

AIM on ClearCanvas Enriched Stroke- phenotyping Software (ACCESS) is a novel standalone computer software application that allows creation of simple standardized annotations for reporting brain images of all stroke types. We developed the ACCESS application and determined its inter-rater and intra-rater reliability in the Stroke Investigative Research and Educational Network (SIREN) study to assess its suitability for multicenter studies.

Methods

One hundred randomly selected stroke imaging reports from five SIREN sites were reevaluated by four trained independent raters to determine inter-rater reliability of ACCESS (version 12.0) software for stroke phenotyping. To determine intra-rater reliability, six raters reviewed the same cases previously reported by them after a month interval. Ischemic stroke was classified using the Oxfordshire Community Stroke Project (OCSP), TOAST and ASCO protocols, while hemorrhagic stroke was classified using the SMASH-U protocol in ACCESS. Agreement among raters was measured with Cohen’s kappa statistics.

Results

For primary stroke type, inter-rater agreement was 0.98 (95%CI: 0.94–1.00) while intra-rater agreement was 1.00 (95%CI: 1.00). For OCSP subtypes, inter-rater agreement was 0.97 (95%CI: 0.92–1.00) for the Partial Anterior Circulation Infarcts (PACI), 0.92 (95%CI: 0.76–1.00) for the Total Anterior Circulation Infarcts(TACI) and excellent for both Lacunar Infarct (LACI) and Posterior Circulation Infarcts (POCI). Intra-rater agreement was 0.97 (0.90–1.00) while inter-rater agreement was 0.93 (95%CI: 0.84–1.00) for TOAST subtypes. Inter-rater agreement ranged between 0.78 (cardioembolic) to 0.91 (large artery atherosclerotic) for ASCO subtypes and was 0.80 (95%CI: 0.56–1.00) for SMASH-U subtypes.

Conclusion

The ACCESS application facilitates concordant and reproducible classification of stroke subtypes by multiple investigators, making it suitable for clinical and multicenter research.

INTRODUCTION

Stroke is a leading cause of death and disability globally, especially in low and middle income countries (LMIC) which bear the brunt of its worldwide burden. (1–5) Neuroimaging forms the basis for its correct definition, accurate diagnosis, classification,; appropriate treatment and prognostication. (4;5)

Combined large accurate datasets from diverse settings are required to fully understand the genomic and environmental underpinnings of stroke and its discrete subtypes as well as the relationship between stroke neuroimaging phenotype and its clinical manifestations, prognosis and outcome (6;7). Such multicenter research involving multiple investigators and large datasets require secure archiving, as well as accurate and reproducible standard classification formats. Thus, there is a need for the development of a comprehensive application for archiving and phenotyping all types of stroke in a standard and secure format that can be shared among researchers and clinicians for training, clinical use, and research purposes; and connected with other clinical, laboratory and genomic databases.

Therefore, we created a user-friendly software application for this purpose. The Annotation and Image Markup (AIM) on Clear Canvas Enriched Stroke phenotyping Software (ACCESS) application captures both ischemic and hemorrhagic stroke unlike previous methods (8–13) which focus solely on ischemic stroke. This is of global relevance because hemorrhagic stroke remains quite common in Africa and other LMIC, where it represents about 30–40% of all stroke cases. (2;5;8;9) ACCESS combines resources from neuroimaging, information technology, programming language and the internet to provide a platform for uniformly reporting on stroke type, subtype, vascular territory, anatomical distribution of lesion, lesion age (hyperacute/acute/subacute/chronic), lesion size, concomitant vascular brain injury (presence and severity of cerebral atrophy and white mater changes) and intracranial atherosclerosis.

We herein describe the development and reliability of the ACCESS software application for accurate concordant stroke phenotyping in multicenter studies.

METHODS

Development of the ACCESS tool

Our goal was to develop simple annotations that are reproducible and consistent for the same type of imaging study (Figures 1, 2, Supplementary Figure 1, Table 1.). The Aim Template Builder (ATB)(Supplementary Figure 2))(10–14) allowed us to generate a set of well-structured questions and answer choices to facilitate collection of information for the neuroimaging aspect of the SIREN study (Supplementary materials). (11–14) These questions and answers were captured as coded elements and codes were obtained for the questions and answers from standard lexicon (RadLex, MESH and CaDSR.) (11–14)Where codes could not be found for a particular question and answer choice, a user-defined term or private code was created. These sets of questions and possible answers were contained in SIREN AIM Extensible Markup Language (XML) template which was then imported into an AIM-enabled application, the ACCESS software.

Figure 1. Flowchart for ACCESS development.

Schematic process involved in obtaining code for lexicon terms for stroke phenotyping.

Figure 2. Screenshot of the ACCESS software application.

Screenshot of the ACCESS software application showing axial non-contrast (left) and reformatted sagittal (right) CT images of a stroke patient with acute hemorrhage in right putaminal region with associated mass effect evidenced by compression of the ipsilateral lateral ventricle and effacement of the cerebral sulci.

Assessment of neuro-images, CT/MRI is usually initialized by clicking (A) to open the aim template (B) on the right of the screen. Clicking on (C) selects the modality and reporting begins by selecting appropriate drop down buttons from (D). After all buttons that characterize the lesion are selected, an annotation is automatically created by clicking on (E) which may then be used for analysis on an Excel spreadsheet or any other statistical package.

Table 1.

Features of ACCESS designed to address major challenges of existing stroke phenotyping platforms

Existing Challenge		ACCESS advantage
i.	Existing software focus on ischemic stroke type only	ACCESS applicable to both major stroke types and assesses other subclinical vascular brain injury
ii.	Ill-defined etiological stroke subtypes	ACCESS software has well-referenced, well-defined, and rule-based assignment of etiological subtypes according to the OCSP, TOAST, ASCO and SMASH-U systems.
iii.	Variability due to differences in interpretation of rules that standardize subtype assignments.	ACCESS automated system eliminates this source of variability by permitting only a uniform language for data entry.
iv.	Existing systems may not accommodate more than one stroke subtype.	ACCESS system accommodates the co-existence of multiple etiologies for individual stroke patients as well as the coexistence of multiple vascular brain injuries of various types and various durations in the same patient
v.	Variability due to disparity in data extraction and the application of the extracted data by the examiners into the software.	ACCESS reduces this through use of a standard manual that requires extraction of test reports confirmed by relevant experts rather than relying on non-physician interpretation of test results.
vi.	Disparity in extracted data application by examiners. Subjective interpretation of neuroimaging, clinical and laboratory data	ACCESS minimizes this by introducing programing language which prevents the user from entering inconsistent data including a dropdown menu that contains a set of predefined answer choices for each question.
vii.	Existing systems may require the interpretation of imaging and clinical information to be typed in.	Default settings are for no lesions in all regions of the brain, such that what is required is for the rater to just select appropriate options for only the affected regions of the brain.

The image information captured in the SIREN ACCESS application includes both anatomical location (e.g. temporal, frontal, occipital, parietal, basal ganglia, brainstem; cortical, subcortical, cortical/subcortical and characteristics of lesions such as laterality, size, volume, age and inference such as stroke subtype (Supplementary Table 1). (7;15–21)

The SIREN ACCESS software application, was developed through twelve series of iterations with modification of the default settings, in order to finally achieve a robust user-friendly version that is simple and easy to complete. The final version promotes speed and efficiency by allowing the reporting physician to select only appropriate standardized answers for each question selected from drop-down menus based on the assessment of the neurologist/radiologist. The data obtained are then exported as an AIM XML document (annotation) to a server or other sites. The data from the AIM XML document can then be mapped into Excel spreadsheet and exported into any statistical program for analysis.

Stroke phenotyping using the ACCESS application

The ACCESS software application allows standardized reporting of stroke type (ischemic, hemorrhagic, ischemic with hemorrhagic transformation, concomitant discrete ischemic and hemorrhagic strokes); size (widest dimension in cm), volume in cm³ (using the ellipsoid equation ABC/2)(22), lesion age (hyper-acute, acute, subacute, chronic) as well as anatomical location; and main arterial territory. We classified spontaneous intra-cerebral hemorrhage based on SMASH-U (Structural lesion, Medication, Amyloid angiopathy, Systemic disease, Hypertensive angiopathy and Undetermined) method.(15) We classified ischemic stroke combining Oxfordshire Community Stroke Project (OCSP)(16;17) (clinical syndromes), Trial of Org 10172 in Acute Stroke Treatment. (TOAST) (18;19) (single dominant causative classification) and A-S-C-O (Atherosclerosis, Small-vessel disease, Cardiac source, Other cause) ASCO (20;21) (recognizing coexisting phenotypes and etiologies) systems. The classification systems were combined so as to improve concordance and reproducibility because the user would have to subtype ischemic stroke through three iterations thereby minimizing errors.

Therefore, we incorporated rules based on the SIREN investigation algorithm (7) (Supplementary Figure 1) to synthetize neuroimaging, clinical and laboratory data to subtype, (7) stroke based on these classification systems. To enhance reliability while activating the Hawthorne effect, we utilize a three-tier adjudication process within SIREN. The first step involves the neurologist/neuroradiologist team in each site for an initial/first review, then another neurologist/neuroradiologist team from a different site and finally a central adjudication panel to review selected cases. (7)

Hyperacute and acute parenchymal changes such as edema, herniation and sulcal effacement are also reported within the system. ACCESS captures abnormalities of the major arterial territories; cerebral atrophy and periventricular white mater disease (absent, mild, moderate and severe) by visual rating. The estimation of generalized central atrophy is based on a template that allowed independent assessment of the width/dilatation of the temporal horn of the lateral ventricle and the width of the Sylvain fissure on an axial image for determining the degree of atrophy: normal = absent or no dilatation, mild = minimally dilated, moderate = moderately dilated, and severe = markedly dilated temporal horn/Sylvian fissure.

For reporting white matter changes, we used the Fazekas visual rating scale (24) for assessment of both periventricular white matter and deep white matter lesions to determine severity based on the Fazekas scale from 0–3.

Reliability of the ACCESS Application

During the SIREN site activation workshops, raters (neurologists and neuroradiologists) were trained by the primary ACCESS software developers (MOO, GIO, KIS) using the SIREN stroke algorithm and phenotyping manual. Practical hands-on sessions were conducted on all sections of stroke imaging interpretation and phenotyping using the ACCESS software (Patent Registration Number: NG/PT/NC/2016/2007).

Raters were tested and a consensus phenotype was achieved for each illustrative case vignette.

To determine intra-rater reliability, 100 randomly selected imaging cases (70 CT and 30 MRI scans) of stroke patients (both ischemic and hemorrhagic) from the SIREN archive from five study sites were included. Forty-nine studies were rated by 2 independent raters from Ibadan, the remaining 51 studies were rated by 4 raters from the Kumasi, Ife, Accra and Kano sites. None were part of the team that developed the software. The same raters then re-assessed exactly the same images after one month while being blinded to their initial report.

For inter-rater reliability, 4 raters working independently from different SIREN sites, blinded to previous reports, reviewed and submitted phenotyping reports for the same selected 100 stroke cases using the ACCESS application. All raters were either trained stroke neurologists or neuroradiologists from each of the five SIREN sites. Relevant clinical and laboratory data for phenotyping were provided from case summaries prepared by the study coordinators from laboratory findings in the case report forms and expert-validated reports by the study cardiologists (ECG, echocardiography), and sonologists (carotid Doppler ultrasound) who did not participate in the rating process. Raters utilized the information along with brain images on the ACCESS platform to classify stroke into its subtypes.

Statistical Analysis

The design of the present study allows for comparison between raters (or two consecutive rating by the same rater – intra rater) of stroke phenotype in a subset of study participants. As a result, the Cohens’ Kappa statistic was used to assess the intra and inter-rater reliability between raters of the stroke phenotype. The Kappa coefficient (K) measures the proportion of agreement between raters (or consecutive ratings by the same rater), that occurs beyond what should have attributable to chance. (25;26)

For phenotypes cortical atrophy, central atrophy and white mater hperintensity (WMH) (Fazekas-visual rating) where the numeric ranking of results in categorical data has an ordinal structure (or where ranking means a worsening or severity of the situation), it was necessary to differentially penalize disagreements based on the magnitude of the disagreement. (25;26) Hence, weighted Kappa statistics was used to evaluate the intra and inter-rater reliability. (25;26)

Kappa values <0 was regarded as poor; 0–0.20 as slight; 0.21–0.40 as fair; 0.41–0.60 as moderate; 0.61–0.80 as substantial and 0.81–1.00 as almost perfect. (25;27) The k statistics with their respective standard error (as well as 95%CI) were calculated using a combination of IBM SPSS Statistics version 20 and Microsoft Excel 2010.

RESULTS

The mean age of the 100 stroke patients was 57.10±14.74 years, with 68% being male. All patients had a cranial CT and/or MRI scan, 97 (97%) had echocardiography, 95 (95%) had electrocardiography, and 98 (98%) had carotid Doppler ultrasonography.

The k-value was 0.98 (95%CI: 0.94–1.00) for inter-rater agreement and perfect for intra-rater agreement for primary stroke type (Table 2).

Table 2.

Intra-rater and inter-rater agreement of Neuroimaging analyses with ACCESS

	Number of patients	Test-retest Reliability		Inter-rater Reliability
		Reliability kappa (95%CI)	Discordant cases	Reliability Kappa (95%CI)	Discordant cases
Stroke type	100	1.00(1.00)	0	0.98(0.976–0.98)	1
Ischemic Hemorrhagic Ischemic with hemorrhagic transformation Both Undetermined	57	0.98(0.976–0.98)	1	0.98(0.976–0.98)	1
	42	1.00(1.00)	0	1.00(1.00)	0
	1	–	–	–	–
	0	–	–	–	–
	0	–	–	–	–
Ischemic stroke subtype OCSP	57	1.00(1.00)	0	0.98(0.976–0.98)	1
PACI TACI LACI POCI	24	1.00(1.00)	0	0.97(0.97–0.98)	1
	6	1.00(1.00)	0	0.92(0.90–0.93)	1
	17	1.00(1.00)	0	1.00(1.00)	0
	10	1.00(1.00)	0	1.00(1.00)	0
TOAST	57	0.98(0.976–0.98)	1	0.98(0.976–0.98)	1
Large artery	29	0.90(0.89–0.91)	4	0.95(0.94–0.96)	2
Small vessel occlusion	18	0.88(0.87–0.89)	4	0.88(0.87–0.89	4
Cardioembolic	5	1.00(1.00)	0	0.88(0.86–0.91)	1
Other determined etiology Vasculitis	1	–	–	–	–
Undetermined etiology	4	–	–	–	–
**ASCO
Atherosclerotic	32	1.00(1.00)	0	0.91(0.83–0.10	1
Small vessel	39	0.98(0.93–1.00)	2	0.88(0.78–0.98	3
Cardioembolic	13	1.00(1.00–1.00)	0	0.78(0.63–0.93)	3
Others	11	0.96(0.88–1.00)	1	0.88(0.75–1.00)	2
Hemorrhagic stroke subtype SMASH_U	42	1.00(1.00)	0	1.00(1.00)	0
Structural	1	1.00(1.00)	0	1.00(1.00)	0
Medication related	1	1.00(1.00)	0	0.00(0.00)	1
Amyloid angiopathy	1	1.00(1.00)	0	1.00(1.00)	0
Systemic disease	0	–	–	–	–
Hypertension	39	1.00(1.00)	0	0.98(0.97–0.98)	1
Undetermined	0	–	–	–	–
Atrophy (generalized)	42	1.00(1.00)	0	0.98(0.95–1.00)	1
Mild	29	1.00(1.00)	0	0.97(0.92–1.00)
Moderate	11	1.00(1.00)	0	1.00(1.00)	0
Severe	2	1.00(1.00)	0	1.00(1.00)	0
*WMH(Fazekas-visual rating)	54	1.00(1.00)	0	1.00(1.00)	0
Mild	34	1.00(1.00)	0	1.00(1.00)	0
Moderate	18	1.00(1.00)	0	1.00(1.00)	0
Severe	2	1.00(1.00)	0	1.00(1.00)	0

^**some patients were classified into more than one subclass for ASCO

OCSP- Oxfordshire Community Stroke Project

LACI- Lacunar Infarct

TACI- Total Anterior Circulation Infarcts

PACI- Partial Anterior Circulation Infarcts

POCI- Posterior Circulation Infarcts

TOAST- Trial of Org 10172 in Acute Stroke Treatment

^*WMH- White Matter Hyperintensities/Hypodensities

The intra-rater agreement for the OCSP subtype of ischemic stroke was perfect (1.00), while the inter-rater k-value was 0.97(95%CI: 0.92–1.00) for the Partial Anterior Circulation Infarct (PACI), 0.92 (95%CI: 0.76–1.00) for the Total Anterior Circulation Infarct (TACI) and perfect for both Lacunar Infarct (LACI) and Posterior Circulation Infarct (POCI). (Table 2).

Similarly, for all subtypes of TOAST, intra-rater agreement was 0.97(95%CI:0.90–1.00) while inter-rater agreement was 0.93(95%CI:0.84–1.00). For ASCO subtypes, inter-rater agreement ranged between 0.78 (cardio-embolic) to 0.91 (large artery atherosclerotic), while inter-rater agreement for haemorrhagic stroke was 0.80 (95%CI: 0.56–1.00) using SMASH-U classification.

DISCUSSION

The ACCESS software application is a novel user-friendly and reliable system for accurate phenotyping of stroke for clinical use, research and training purposes. It provides options which can be selected from a drop-down menu following a simple right mouse click on each variable. Furthermore, the default settings are such that only the sections relevant to observed lesions need be completed because other sections are automatically pre-populated to indicate that other brain regions and parameters are normal. The intra-rater and inter-rater reliability (k =0.78 to 1.00) of the ACCESS software in 100 patients appears to be excellent compared to similar packages in stroke imaging and phenotyping evaluations. (28)

This is probably because the adjudication process was made rigorous as rule-based inferences for OCSP, TOAST, ASCO, were only possible after systematic evaluation of all brain regions and synthesis of all available expert-validated summary of neurovascular, cardiovascular, laboratory and clinical investigations. The iterative nature of a combination of OCSP, TOAST, and ASCO systems and the Hawthorne effect of three-tier adjudication may also have contributed to the excellent performance of the ACCESS tool.

Additional Strengths: Inherent features designed to reduce inter-rater variability

The challenges overcome by the ACCESS software are summarized in Table 1. Subjective interpretation of neuroimaging, clinical and laboratory data are an important source of variability in etiologic stroke classification. Unlike pre-existing systems,(29) the ACCESS software reduced this limitation by introducing a well-referenced, well-defined, and rule-based assignment of etiological subtypes according to the ASCO, TOAST and SMASH-U systems. (7;15–21)

To minimize variability in interpretation of rules that standardize subtype assignments; apart from rigorous training of adjudicators, the ACCESS automated system permitted only a uniform language for data entry. Additionally, the ACCESS system recognizes the co-existence of multiple etiologies for stroke patients as well as the co-existence of multiple vascular brain injuries of various types and various ages. (20;21) This information was incorporated and presented in an easy to understand and standardized format.

Finally, the variability in data extraction by examiners was reduced through the use of a single standard manual that required extraction of structured test reports confirmed by relevant experts (e.g. echocardiography reports reviewed by cardiologists), rather than relying on non-expert interpretation of test results. The expert-validated report of relevant laboratory, clinical, cardiovascular and neurovascular imaging report for each patient is used by all adjudicators for the index patient. The disparity in extracted data application by examiners was minimized by introducing system checks that prevented the user from entering inconsistent data. These include a drop-down menu that contains a set of predefined answer choices for each question. Furthermore, the concluded annotation is mapped into our REDCap electronic medical record database which allows cross-query and browsing across different sets of data elements from different databases.

Limitations and Future directions

ACCESS is an imaging based application and excludes all non-imaging cases. The system does not admit continuous variables or support text entry. Data is entered only as ordinal variable, ranked according to stratified range of values (e.g. stratified range of volumes). The application may be improved to accommodate secure cloud creation, storage and retrieval of images linked with annotations for clinical research. Another enhancement feature would be the inclusion of volumetric analysis capabilities for lesions and brain segments and spaces.

Majority of stroke patients in LMIC are still unable to access neuroimaging and other ancillary tests required for adequate phenotyping of stoke with the ACCESS software. This may restrict the use of the software to research settings and limit its clinical utility in certain areas. This limitation may become increasingly less important with the growing increase and broader availability of neuroimaging facilities in many more LMICs.

Utility and Implications

The standalone ACCESS application provides immediate feedback, independent of server availability or network connection, making it, not only useful in high income countries but also a viable resource for LMIC with limited or poor internet or network infrastructure. The ACCESS application has inbuilt features for characterizing hemorrhagic stroke. It also allows reporting of other indices of vascular brain injury such as atrophic brain changes and incorporates the rating of white mater hyperintensity commonly associated with stroke, unlike pre-existing systems. Finalizing a report automatically makes the information (which incorporates multiple components of the stroke work-up) available across the platform and on the application server. The archived data may be shared as an XML file. The XML file may be converted to Microsoft Excel spreadsheet and exported to other databases for analysis and other research purposes.

Conclusion

ACCESS fulfills an obvious need for an algorithmic classification system for consistent classification of all stroke types and subtypes. It ensures accurate phenotyping of stroke by facilitating standardized entry, rigorous and systematic interpretation of relevant clinical, laboratory, neurovascular and brain imaging data; and eventual iterative synthesis of all evidence for rule-based adjudication of stroke types and subtypes with excellent intra-rater and inter-rater reliability. The ACCESS data can be merged with other databases for extensive exploration of numerous research questions to promote the understanding of the genetic and environmental risk factors for stroke, its subtypes, severity and outcome. It is being used in the SIREN multicenter stroke study and is recommended for use in other multicenter stroke studies in LMIC and elsewhere.

Supplementary Material

1

Supplementary 1

Other Members of SIREN Team

Supplementary Box:

• AIM Platform;
• ACCESS Structure;
• Supplementary Figure 1: SIREN investigations algorithm
• Supplementary Figure 2: Screenshot of AIM Template Builder (2.0 interface)
• Supplementary Table 1: ACCESS Annotation concept

2

Supplementary 2

Summary of Investigations of participants