Adjudication of Transient Ischemic Attack and Stroke in the Multi-Ethnic Study of Atherosclerosis (MESA)

Abstract

Background

To describe adjudication of transient ischemic attack (TIA) and stroke in an observational study.

Methods

We detail the process used to adjudicate TIA and stroke in the Multi-Ethnic Study of Atherosclerosis (MESA), a large longitudinal cohort study. Two of three vascular neurologists adjudicated each event using specific protocols. We examined initial agreement, effect of imaging on diagnosis of TIA versus ischemic stroke, and effect of strict and less strict criteria on the number of ischemic stroke subtypes classified as undetermined.

Results

Of 573 adjudicated events over 13.5 years of follow-up, 95 (16.5%) were TIA, and 269 (47.0%) were stroke: 211 (78.4%) ischemic, 43 (16.0%) hemorrhagic, and 15 (5.6%) other. Disagreements occurred on 16% of initial adjudication of events. Using results from imaging, the number with TIA decreased by 8.6% and with ischemic stroke increased by 4.1%. Using less strict criteria to classify ischemic stroke subtypes reduced the number classified as undetermined, from 137 to 59, and numbers classified as cardioembolic and small vessel doubled.

Conclusions

We hope that this work will motivate and facilitate investigators to use MESA data to investigate issues concerning TIA and stroke and will inform investigators seeking to adjudicating TIA and stroke in other studies.

BACKGROUND

In observational studies, consistent and accurate adjudication of stroke is critical, allowing investigators to estimate incidence rates and identify novel etiologic risk factors. The challenge is greater in observational studies than in clinical trials because the investigator has little control over the diagnostic evaluation that is performed during routine medical care to determine whether a stroke occurred and, if so, the type and subtype. In the Multi-Ethnic Study of Atherosclerosis (MESA), a large multi-center longitudinal cohort study, we designed and here in this methods paper describe an efficient online system involving two of three vascular neurologists adjudicating each event and deciding if a transient ischemic attack (TIA) or stroke had occurred. Besides evaluating the reliability of the stroke determination, we also wanted to assess how imaging results affected the diagnosis of TIA and ischemic stroke, given imaging-based definitions of these clinical outcomes [1, 2]. Finally, we wanted to gauge the effect of utilizing both strict and less strict criteria for classification of ischemic stroke subtype on the number classified as infarcts of undetermined cause. This information may help not only those investigating TIA and stroke in MESA but also those seeking an efficient method to adjudicate TIA and stroke in other observational studies.

METHODS

The MESA is an ongoing prospective observational cohort study that was initiated in 2000 July with recruitment of 6,814 participants, 52.8% women, between 45 and 84 years of age from six U.S. communities: Baltimore City and Baltimore County, Maryland; Chicago, Illinois; Forsyth County, North Carolina; Los Angeles County, California; Northern Manhattan and the Bronx, New York; and St Paul, Minnesota [3]. The MESA is truly multi-ethnic with 38% of the cohort identifying themselves as being non-Hispanic white; 28%, non-Hispanic black; 22%, Hispanic; and 12%, Chinese. Participants were excluded if they had a physician diagnosis of heart attack, stroke, transient ischemic attack, heart failure, angina, atrial fibrillation, or history of any cardiovascular procedure. Participants with weight greater than 300 pounds, pregnancy or any medical conditions at the time of initial recruitment that would prevent long-term participation were also excluded. The primary goal of the study, when designed, was to investigate prevalence and progression of subclinical cardiovascular disease (CVD). The institutional review boards at all participating centers approved the study, and all participants gave written informed consent.

At intervals of 9 to 12 months, MESA participants are contacted by phone and have been seen in person at four follow-up examinations since the initial examination. Multiple methods are used to identify potential events, and once they are identified, MESA staff members review the following documents: medical records from inpatient hospitalizations, death certificates, and autopsy reports. For out-of-hospital events, participants are interviewed, and outpatient summaries and diagnostic test results are reviewed. For out-of-hospital deaths, physicians and family members are interviewed. After being anonymized, all of these records are made available for adjudicators.

Adjudication committees evaluate all potential study events including CVD or interventions aimed at CVD, as well as TIA, stroke, peripheral vascular disease, and mortality [3]. Cardiovascular disease includes acute myocardial infarction, other forms of coronary heart disease, and heart failure. Three vascular neurologists reviewed all events that occurred up to 2013 December 31 and were thought possibly related to TIA or stroke, including fatal and nonfatal events that did or did not result in hospitalization.

Adjudicated events are classified as TIA, stroke, or neither based on symptoms, signs, and imaging results. If the event is adjudicated as neither, no additional information is collected. If an event is coded as a stroke, it is categorized into one of four stroke types: hemorrhagic, ischemic, other, or undetermined. Detailed definitions are presented online in the MESA Events Manual of Operations (https://www.mesa-nhlbi.org/manuals.aspx). The “undetermined” type is assigned when the evaluation is limited, and not enough information is available to decide if one of hemorrhagic, ischemic or other type is appropriate. The “other” type is assigned for conditions that do not clearly meet criteria for either hemorrhagic or ischemic type, for example a venous sinus occlusion. For hemorrhagic stroke type, the subtypes are classified as subarachnoid hemorrhage, intraparenchymal hemorrhage, and other hemorrhage with the source of the hemorrhage specified.

For ischemic stroke type, the subtypes are classified as: large vessel, extracranial (LV EC); large vessel, intracranial (LV IC); cardioembolic (CE); small vessel (SV); other specific mechanisms; multiple mechanisms; or undetermined. For ischemic stroke type, we developed an extension of the Trial of Org 10172 in Acute Stroke Treatment (TOAST) scheme [4, 5] to try to reduce the number classified as undetermined. The first approach requires a complete evaluation and a strict adherence to a pre-specified set of definitions based on the TOAST criteria. An evaluation is considered complete if it included: imaging of the brain, intracranial vessels and extracranial vessels; echocardiogram; and electrocardiogram. If the first approach yields a classification of an undetermined subtype, a second approach is tried that does not require a complete evaluation and allows the adjudicator to choose the best subtype using his or her clinical judgment. The second approach thus allows the use of less strict criteria. Finally, if the first approach yields a classification of multiple subtypes or conflicting mechanisms, the third approach is selected allowing the event to be classified as having two different subtypes. The location and vascular territory of the ischemic stroke are also coded separately using up to five selections from standardized lists.

When a possible vascular event of the brain is identified, packets of relevant records are sent simultaneously to two of the three vascular neurology adjudicators. Each adjudicator then independently fills out an online form with a series of questions about the event, first indicating whether it is a TIA, stroke, or neither, and then answering more detailed questions about types, subtypes and location of strokes. Appendix 1 shows the form that adjudicators use as a guide to complete the MESA online form. Immediately after the second reviewer has entered his or her online responses, any disagreements are instantly brought to that reviewer’s attention. That reviewer then has three options: (1) change the review to agree with the first reviewer, (2) return to the review to update it and then forward it to the first reviewer, or (3) leave the review as is and forward it to the first reviewer. That first reviewer can then change his or her response to agree with the second reviewer or trigger a joint discussion between the two reviewers to achieve consensus. Responses are finalized once the two reviewers agree.

A separate MESA online form is completed if an event is coded as a death due to stroke, as detailed in Appendix 2. Adjudicators indicate the duration between onset of stroke symptoms and death and specify one or more mechanisms of stroke death. Again, responses are finalized once the two reviewers agree.

Based on the two reviewers’ initial response, we evaluated inter-rater reliability for the event type and, among events classified as strokes, for the stroke type. The kappa statistic was used to quantify inter-rater reliability, using the bootstrap method to estimate the standard errors in order to account for patients with adjudication of multiple events. For everyone adjudicated as having TIA or ischemic stroke, we examined the effect of the imaging results on the proportions of TIA and stroke. Since not everyone with a diagnosis of TIA had brain imaging performed, we repeated these analyses for the subgroup where everyone had imaging performed. For those adjudicated as having ischemic stroke, we evaluated the effect of using less strict criteria compared to strict criteria on the frequency of ischemic stroke subtypes, especially the number classified as undetermined. Statistical analyses were conducted in Stata (v11.2, StataCorp LP, College Station, TX).

RESULTS

As of June 2015, the final adjudicated diagnosis of all events that occurred up until 2013 December 31 yielded 95 (16.5%) TIAs, 269 (47.0%) strokes, and the remaining 209 (36.5%) determined not to be vascular events of the brain (Table 1). Vascular risk factors listed in the table were defined as previously and were determined at the baseline examination [6]. Of these 573 events, information was available on the responses of both reviewers in 536: 91 TIAs, 249 strokes, and 196 events that were neither. The two reviewers disagreed 16.0% of the time on the adjudicated diagnosis on initial review; for 12.5%, the initial disagreement was resolved when one of the reviewers changed the diagnosis to agree with the other reviewer; and for the remaining 3.5%, a persisting disagreement required a discussion to reach a consensus. The kappa for initial agreement in determining TIA, stroke, or not vascular event of the brain was 0.75 (standard error (SE) 0.03) and in determining stroke or not stroke was 0.82 (SE 0.03) [7]. For those adjudicated as having a stroke (n=249), the kappa was even better at 0.92 (SE 0.03) for initial agreement in determining the stroke type: hemorrhagic, ischemic, or other including undetermined. For those adjudicated as stroke, the two reviewers disagreed 12.0% of the time on the adjudicated stroke type on initial review; for 2.4%, the initial disagreement was resolved when one of the reviewers changed the stroke type to agree with the other reviewer; and for the remaining 9.6%, a persisting disagreement required a discussion to reach a consensus.

Table 1.

Characteristics of participants with 573 adjudicated events.

	Adjudicated event			Stroke Type
Characteristics*	Neither 209 (36.5%)	TIA 95 (16.5%)	Stroke 269 (47.0%)	Ischemic 211 (78.4%)	Hemorrhagic 43 (16.0%)	Other 15 (5.6%)
Age, years (SD)	69.4 (8.6)	66.8 (10.0)	68.0 (9.4)	67.8 (9.4)	66.3 (8.6)	75.1 (7.9)
45-54 years	19 (9.1)	14 (14.7)	29 (10.8)	24 (11.4)	4 (9.3)	1 (6.7)
55-64 years	27 (12.9)	25 (26.3)	60 (22.3)	47 (22.3)	13 (30.2)	0 (0.0)
65-74 years	96 (45.9)	27 (28.4)	102 (37.9)	81 (38.4)	18 (41.9)	3 (20.0)
75-84 years	67 (32.1)	29 (30.5)	78 (29.0)	59 (28.0)	8 (18.6)	11 (73.3)
Women	102 (48.8)	50 (52.6)	137 (50.9)	105 (49.8)	24 (55.8)	8 (53.3)
Race/Ethnicity
Non-Hispanic white	107 (51.2)	40 (42.1)	105 (39.0)	84 (39.8)	16 (37.2)	5 (33.3)
Non-Hispanic black	43 (20.6)	27 (28.4)	75 (27.9)	59 (28.0)	11 (25.6)	5 (33.3)
Hispanic	46 (22.0)	26 (27.4)	71 (26.4)	56 (26.5)	10 (23.3)	5 (33.3)
Chinese	13 (6.2)	2 (2.1)	18 (6.7)	12 (5.7)	6 (14.0)	0 (0.0)
Education > high school	78 (37.5)	41 (43.2)	99 (36.9)	79 (37.6)	15 (34.9)	5 (33.3)
Ever smoking	109 (52.4)	50 (52.6)	132 (49.3)	101 (48.1)	24 (55.8)	7 (46.7)
Hypertension	142 (67.9)	59 (62.1)	190 (70.6)	154 (73.0)	25 (58.1)	11 (73.3)
Diabetes	41 (19.6)	19 (20.0)	71 (26.6)	57 (27.1)	9 (20.9)	5 (35.7)
Hyperlipidemia	52 (25.1)	8 (8.5)	60 (22.6)	52 (24.9)	6 (14.3)	2 (13.3)
Brain imaging	107 (53.0)	87 (91.6)	250 (92.9)	209 (99.1)	40 (93.0)	1 (6.7)

^*For age, mean (standard deviation) are displayed, and for discrete variables, frequencies (column percentages) are displayed. Those in the higher education level are people who completed any certificate or degree beyond a high school diploma. The table includes 152 recurrent events occurring in 103 participants who had already had an adjudicated event.

Table 1 provides information on participants according to the results of the final adjudication. The table includes 152 recurrent events occurring in 103 participants who had already had an adjudicated event, representing 26.5% of all 573 events. Thus, 421 participants had one or more events adjudicated. Adjudication of the initial event was most likely to result in stroke (68%), while adjudication of recurrent events was most likely to result in neither TIA nor stroke (59%), perhaps reflecting in some events a recrudescence of stroke symptoms and signs [8]. Based on all 6,814 MESA participants and the crude numbers in Table 1, the percentages with stroke, increased with advancing age: 1.5% (29/1,947) for 45-54-year-old participants; 3.2% (60/1,884) for 55-64; 5.1% (102/2,018) for 65-74; and 8.1% (78/965) for 75-84. The percentages by race and ethnicity were similar for all strokes, being slightly higher for Hispanics and slightly lower for Chinese: 4.0% (105/2,622) for non-Hispanic whites; 4.0% (75/1,892) for non-Hispanic blacks; 4.7% (71/1,496) in Hispanics; and 2.2% (18/804) in Chinese. The percentages by sex were similar: 3.8% (137/3,601) for women and 4.1% (132/3,213) for men. As per Table 1, only 15 strokes (5.6%) were adjudicated as the other stroke type, which included 14 of undetermined type. The other category was most common in the oldest age group, often reflecting an incomplete work up. Ischemic stroke (n=211, 78.4%) was far more common than hemorrhagic stroke (n=43, 16.0%), which was comprised of 36 intra-parenchymal hemorrhages, 6 subarachnoid hemorrhages, and 1 exclusively intraventricular hemorrhage. The adjudicated stroke was judged to be the cause of death in 42 (28%) of 152 overall deaths in the 421 participants with adjudicated events. The factor most frequently identified as a primary cause of stroke-related death was critical brain injury (n=19, 45%). Most adjudicated strokes were in participants who were hospitalized (89%), but some were not (11%). Brain imaging was performed in 444 (78%) of the 573 events. The adjudicators did not record the proportion having CT, MRI or both and only reviewed imaging reports not the actual images.

If brain imaging results were not considered in the diagnosis of TIA but only duration of symptoms being less than 24 hours, adjudications would have yielded 101 TIAs and 193 strokes. If imaging results were considered, participants with symptoms less than 24 hours but with imaging findings of a relevant infarct were classified not as TIA but as stroke. This approach yielded 93 TIAs and 201 strokes, decreasing the number of TIAs by 7.9% (95% binomial exact confidence interval (CI) 3.5 to 15.0%) and increasing the number of strokes by 4.1% (95% CI 1.8 to 8.0%). Excluding the 9 events in which brain imaging was not performed and considering only those events in which it was performed yielded a TIA decrease of 8.6 % (95% CI 3.8 to 16.2%) and stroke increase of 4.2% (95% CI 1.8 to 8.0%).

For those events adjudicated as ischemic stroke, Figure 1 shows in the first row the distribution of 211 ischemic stroke subtypes when applying strict criteria. The second row and Figure 2 show the results of using the less strict criteria to reclassify the undetermined subtype. The number classified as undetermined was reduced by 57% from 137 to 59, and the numbers classified as cardioembolic and small vessel doubled. For the 7 events with multiple subtypes categorized by the less strict criteria, the 14 additional subtypes, since each event yielded 2 subtypes, were added to the numbers in the second row These additions yielded the less strict+ group listed in the third row, dropping the numbers of multiple subtype from 7 to 0. The third row maximized the number of participants with a particular ischemic stroke subtype by reducing those classified as undetermined or multiple subtypes. Additional details are provided in the footnote to Figure 1.

Figure 1. Ischemic stroke subtype classifications based on three criteria.

Ischemic stroke subtypes are: large vessel, extracranial (LV EC); large vessel, intracranial (LV IC); cardioembolic (CE); small vessel (SV); other specific mechanisms (Other); multiple mechanisms (Multiple); or undetermined. The diagonal arrows show how those in the undetermined subtype in the first row and multiple subtype in the second row are redistributed. In the first row (N=211), strict criteria were applied, and the undetermined subtype included those in the multiple subtype. In the second row (N=211), less strict criteria were applied to those classified as undetermined based on strict criteria. For the multiple subtype with less strict criteria, all 7 had SV combined with LV EC in 2, LV IC in 2, CE in 2 and Other in 1. These 14 additional subtypes were added to the numbers in the second row to yield the Less Strict+ criteria listed in the third row (N=218), allowing each event to have 2 subtypes.

Figure 2. Frequency of ischemic stroke subtypes based on strict or less strict criteria.

Ischemic stroke subtypes are: large vessel, extracranial (LV EC); large vessel, intracranial (LV IC); cardioembolic (CE); small vessel (SV); other specific mechanisms (Other); multiple mechanisms (Multiple); or undetermined.

Discussion

We have described the methods used to adjudicate events suggesting TIA or stroke in MESA, a large longitudinal cohort study, ongoing since 2000. In an effort not to miss any outcomes of interest, all events that could potentially represent a TIA or stroke were adjudicated, and over a third were found to be neither. The results of reliability testing showed imperfect agreement on the initial decisions by two reviewers on whether or not an overt vascular brain injury had occurred. These results suggest that such adjudication in observational studies should not be performed by a single person but by at least two. Over time, we plan to assess the reliability of the entire process by re-adjudicating previously adjudicated events to see if the final diagnosis from the two adjudications agrees.

When designing the process, we collected information that would allow us to quantify the effect of imaging-based diagnoses of ischemic stroke. Whether examining all events or just those in which brain imaging was performed, imaging findings of a relevant infarct resulted in an increase of events diagnosed as stroke by about 4% and a decrease of the events diagnosed as TIA by about 8-9%. These numbers classified as stroke are likely to increase over time as more patients with a TIA diagnosis based on duration of symptoms undergo imaging, specifically imaging with the more sensitive MRI than CT. For example, in a population-based study during 2011 from the L’Aquila district of central Italy, among 210 events, the decrease due to imaging in the proportion diagnosed as TIA based on duration of symptoms alone was 13.8% [9]. Hence, the distribution between the diagnoses of TIA and stroke will depend in part upon how frequently early MRI is performed in patients with a brief duration of symptoms and upon the assumption that early restricted diffusion on MRI is irreversible. Most events diagnosed as stroke had imaging performed, but in some, it was without a relevant lesion – most commonly with CT but sometimes with MRI.

When we applied strict criteria to the assignment of ischemic stroke subtypes based on the TOAST scheme and required a complete stroke evaluation, close to two-thirds of ischemic strokes were classified as undetermined. Using less strict criteria and not requiring a complete evaluation, just over a quarter of ischemic strokes were classified as undetermined. For example, with the strict criteria, 12.8% were classified as the cardioembolic subtype of ischemic stroke, whereas with the less strict combined with the strict criteria, the number almost doubled with 25.2% classified as the cardioembolic subtype. In this community-based study, the evaluation of stroke was often incomplete, presumably when providers felt confident about the cause of the stroke, for instance with the small vessel and cardioembolic subtypes. More focused evaluations will likely continue to be common in observational studies as opposed to other research settings when the stroke evaluation may be prescribed.

Although the adjudication process is standardized, only a relatively small number of events have been adjudicated to date. With time the numbers will increase, but problems will persist with the amount of information available to adjudicate a particular event, and as a consequence, less strict criteria with less precise diagnoses may result. Having both strict and less strict criteria available will allow investigators of ischemic stroke subtypes to perform sensitivity analyses.

Conclusion

The MESA provides investigators many opportunities to examine issues concerning vascular disease of the heart and brain. We hope that this work will motivate investigators to study the risk factors and manifestations of overt vascular brain injury in MESA. For those faced with the challenges of adjudicating TIA and stroke in other observational studies, we hope that this work will make the task easier.