Abstract
Objective:
Generalized tonic-clonic seizures (GTCS) are a major risk factor for sudden unexpected death in epilepsy (SUDEP). We investigated whether ictal/postictal cardiac features were dependent on seizure type within individual patients.
Methods:
ECG data from patients with medically refractory temporal lobe epilepsy (TLE) undergoing presurgical investigation who had both complex partial seizures and secondarily GTCS during video-EEG telemetry were retrospectively reviewed. Peri-ictal heart rate (HR), corrected QT interval (QTc), HR variability, and cardiac rhythm were assessed.
Results:
Twenty-five patients were included in this study. Secondarily GTCS led to higher ictal HR, persistent postictal tachycardia, and decreased postictal HR variability. Moreover, abnormal shortening of QTc occurred in 17 patients mainly during the early postictal phase and significantly more often in secondarily GTCS. Abnormal QTc prolongation occurred in 3 patients with no significant association with GTCS. Benign cardiac arrhythmias occurred in 14 patients and were independent of seizure type.
Conclusions:
Our data suggest a substantial disturbance of autonomic function following secondarily generalized tonic-clonic seizures (GTCS) in patients with medically refractory temporal lobe epilepsy. The observed alterations could potentially facilitate sudden cardiac death and might contribute to the association of sudden unexpected death in epilepsy with GTCS.
GLOSSARY
- CI
= confidence interval;
- CPS
= complex partial seizure;
- GTCS
= generalized tonic-clonic seizures;
- HR
= heart rate;
- HRV
= HR variability;
- QTc
= corrected QT interval;
- SUDEP
= sudden unexpected death in epilepsy;
- TLE
= temporal lobe epilepsy.
Sudden unexpected death in epilepsy (SUDEP) is the most frequent epilepsy-related cause of death and is particularly prevalent in patients with chronic epilepsy.1 In younger people with epilepsy (age 20–40 years), the risk for sudden death is increased about 24-fold compared to the general population.2 Peri-ictal cardiorespiratory alterations are likely to be involved in the pathophysiology of SUDEP and include tachyarrhythmias and bradyarrhythmias as well as central or obstructive hypoventilation and neurogenic pulmonary edema.1 Since generalized tonic-clonic seizures (GTCS) are a major risk factor for SUDEP,3,4 then determining the specific cardiac characteristics of GTCS may give us a considerable insight into the mechanisms underlying SUDEP. Abnormalities in cardiac repolarization, persistent elevations of heart rate (HR) after exercise, and a decreased HR variability (HRV) are established predictors for cardiac mortality and sudden cardiac death in other medical conditions or in healthy populations.5–7 Previous studies have suggested a greater impact of GTCS on cardiac excitability than nongeneralized seizures with higher ictal heart rates in chronic epilepsy and SUDEP patients4,8–11 and possibly more cardiac arrhythmias following GTCS.8,10 These studies, however, are confounded by the failure to compare the cardiac features in different seizure types within individual patients and they therefore did not take into account individual differences in autonomic nervous system responses; these can be altered to varying degrees in chronic epilepsy patients.12 Here, we asked whether the occurrence of ECG predictors for cardiac mortality is dependent on seizure type in individual patients.
METHODS
We reviewed EEG and ECG data from patients with medically refractory TLE who underwent standard presurgical assessment at the National Hospital for Neurology and Neurosurgery between 1993 and 2007 and who had at least one habitual complex partial seizure (CPS) and one secondarily GTCS arising from the same hemisphere. In this time period, a total of 1,033 patients with medically refractory focal epilepsy underwent presurgical video EEG telemetry. To avoid potential bias, one CPS and one secondarily GTCS were analyzed per patient. If there were more than one CPS or secondarily GTCS, we selected according to the following criteria: chronological order of occurrence (first seizure of each seizure type was analyzed in order of chronological occurrence during video-EEG telemetry); second, EEG pattern at onset of the seizure was similar (same region, same pattern); third, ECG data were not compromised throughout the recording due to muscle artifacts. The latter was especially an issue during GTCS, in which ictal QT and RR intervals were necessarily determined immediately before the generalized tonic-clonic phase and immediately after seizure cessation. Following a GTCS patients were usually administered oxygen via a face mask.
Recordings were performed using conventional scalp EEG recordings (10–20 system) or intracranial recordings (2 patients) at a sampling rate of 200 Hz. A modified lead-I ECG (adhesive electrodes placed below the clavicles of either side) was recorded simultaneously with EEG. HR was determined at different time points (1 minute before, during the seizure at variable time points when HR was highest, immediately after seizure cessation, 1 minute, 3, 5, 10, 15, and 30 minutes after seizure cessation) by averaging 3 to 5 consecutive RR intervals (figure 1A). HRV was assessed by analysis of consecutive RR intervals during 30-s epochs in the 1 minute before seizure onset and 5 minutes after seizure cessation and expressed as SD (SDNN, in ms) of all RR intervals.13 QT intervals were manually measured from the start of the QRS complex to the end of the T wave (defined by the intersection with the isoelectric line) at the same time points as RR intervals before, during, and immediately after seizure, 1, 3, and 5 minutes after seizure cessation. QT intervals and preceding RR intervals were determined from 3 to 5 successive ECG complexes and the resulting corrected QT intervals (QTc) were averaged. Four different formulas (Bazett, Fridericia, Hodges, and Framingham) were used to calculate QTc.14 To reduce a bias error of putatively pathologic QTc intervals (due to higher HR during and after seizures), we used modified normal limits for QTc intervals as determined by Luo et al.15 and, conservatively, considered only those QTc intervals to be pathologic on which at least 2 correction formulas agreed.
Intraindividual comparison was performed using 2-sided paired Student t test. Statistical dependence was tested using 2-sided Fisher exact test or 2-sided McNemar test where appropriate for categorical data, and linear regression for continuous data. Dependence of short QTc on HR was tested using a mixed linear regression model which allowed adjustment for individual seizures and patients. p Values <0.05 were regarded as significant. The primary null hypotheses of this study (ictal and postictal HR, HRV, and QTc interval are the same in CPS and secondarily GTCS) and respective p values were adjusted for multiple comparisons using the Holm-Bonferroni stepwise correction. Data are given as mean ± SEM.
Standard protocol approvals, registrations, and patient consents.
This study is part of a retrospective audit of the utility of ECG in patients with epilepsy, approved as such by the local ethics review board. Therefore, as a retrospective anonymized chart review, it did not require informed patient consent.
RESULTS
Twenty-five patients were included in this study (clinical information is summarized in the table and table e-1 on the Neurology® Web site at www.neurology.org). Routine 12-lead ECG analysis (available in 21 patients) was normal.
HR was higher during secondarily GTCS (figure 1A) and persisted at levels above 100 bpm throughout the postictal observation period. After 30 minutes, HR was still about 35% higher after secondarily GCTS, whereas it substantially decreased within the first 5 minutes after CPS. Furthermore, HRV was decreased after secondarily GTCS (figure 1B). Although secondarily GTCS lasted longer (figure 1C), there was no significant association between ictal (or immediately postictal) HR and seizure duration (CPS: p = 0.7; secondarily GTCS: p = 0.06) or duration of the tonic-clonic phase (p = 0.26; duration of tonic-clonic phase 78 ± 11 s, n = 25). Benign arrhythmias were observed in 9 CPS and 10 secondarily GTCS (in 14 patients) and included ictal or postictal prominent sinus arrhythmia and premature atrial/ventricular beats. Their occurrence was not dependent on seizure type (p = 1.0, OR 1.25; 95% confidence interval [CI] 0.34–4.66) or side of seizure onset (right vs left side: p = 0.42; OR 2.25; 95% CI 0.41–12.44). Lengthening of QTc above normal upper limits was found in 3 seizures (2 secondarily GTCS, 1 CPS) and 3 patients. Shortening of QTc below normal limits (figure 2A) was observed in 17 out of 25 patients mainly during the early postictal phase (figure 2C) and occurred significantly more often in secondarily GTCS (p = 0.047; figure 2B). Importantly, occurrence of short QTc was not simply dependent on HR (p = 0.156, 95% CI 0.01–0.06). Also, gender (female vs male: p = 1.0; OR 0.69; 95% CI 0.10–4.72) or side of seizure onset (right vs left; p = 0.22; OR 0.13; 95% CI 0.01–2.85) had no clearcut influence on occurrence of short QTc in our study population.
DISCUSSION
Our data indicate a persistent imbalance in autonomic function with prevailing sympathetic influence following secondarily GTCS. Postictal HRV was reduced and HR recovery (return of HR to baseline) was significantly slower following secondarily GTCS compared to CPS. Impaired HR recovery after exercise and reduced HRV both predict cardiac mortality and sudden cardiac death.6,7 Predominant catecholaminergic activity can be arrhythmogenic and,16 together with seizure-related metabolic, hormonal (other than catecholamines), and respiratory alterations,17–19 could enhance the risk of fatal arrhythmia. Also, repeated secondarily GTCS (and the concurrent catecholaminergic activity) may lead to structural modifications of cardiac tissue which then in turn could facilitate the occurrence of cardiac arrhythmias.20
Alterations of cardiac repolarization are established predictors of sudden cardiac death,5,21 and we and others have shown abnormal QTc lengthening with seizures.4,22 Abnormal QT prolongation was found in 6% of the seizures and 12% of the patients, thus relatively similar to the observed prevalence in our previous study.4 Prolonged QTc interval is an established risk factor for life-threatening “Torsade de Pointes” tachycardia.5,21 Thus, peri-ictal QT prolongation could facilitate fatal ventricular tachyarrhythmia.
Unexpectedly, we also observed frequent shortening of QTc below normal limits particularly with secondarily GTCS.15 Importantly, 2 observations argue against short QTc being simply a reflection of higher HR in secondarily GTCS: short QTc predominantly occurred in the early postictal phase, and not during ictal HR peaks, and short QTc did not correlate with absolute HR.
We and others have undoubtedly underestimated the prevalence of QT shortening.4 This is most likely related to the fact that QT shortening seems to occur predominantly immediately or 1 minute after seizure cessation, a time period not usually assessed in other studies.
The clinical relevance of enhanced occurrence of short QTc with secondarily GTCS is unclear at present. Short QTc syndrome with a genetic etiology has only recently been described.23 Mutations in cardiac voltage-gated potassium and calcium channels shorten QTc and thereby reduce ventricular refractory period as a potential cause for life-threatening re-entry ventricular tachyarrhythmia. Clinical data on the predictive value of short QT in otherwise healthy populations are, however, controversial.24–26 Likewise, little is known about the significance of acquired, e.g., drug-induced shortening of QTc and therapeutic consequences are uncertain.23,27,28 Primidone and rufinamide can both shorten QTc,29,30 but these drugs were not used in our patients. Although the use of carbamazepine could have influenced our results, this drug is associated with bradyarrhythmias and has only minor effects on QT intervals.31,32 Further, QTc was normal during interictal periods and also we compared the effects of different seizure types within individual patients.
What could be mechanisms of a predominant occurrence of short QTc in the early phase after secondarily GTCS and how could this potentially contribute to the pathophysiology of SUDEP? Shortening of QT is substantially mediated by circulating catecholamines and can be enhanced by hyperkalemia and acidosis.23,27,28,33 All these conditions occur particularly during or shortly after GTCS.18,19,34 Furthermore, chronic epilepsy patients display altered autonomic function with a presumably increased sympathetic tone and enhanced QT dispersion (spatial heterogeneity of cardiac repolarization), which are both independent predictors of cardiac mortality.4,7,35,36 The latter finding is of particular interest, as enhanced QT dispersion facilitates ventricular tachyarrhythmia with short QT interval.37
Thus, one plausible explanation for SUDEP would be peri-ictal occurrence of pathologic cardiac repolarization resulting in sudden onset of ventricular tachyarrhythmia. Our findings also suggest that assessment of QT dispersion and peri-ictal QTc could be useful in epilepsy patients with frequent GTCS in order to appreciate the potential risk of fatal tachyarrhythmia.4 Indeed, ventricular tachyarrhythmia was recently reported to occur in a near-SUDEP event with a secondarily GTCS in a patient without underlying cardiac disease.38 This also raises the question whether the use of anti-arrhythmic treatment or implantation of cardiac defibrillator devices might be potential preventive measures in epilepsy patients at high risk for SUDEP. The observation of peri-ictal cardiac repolarization as potential underlying mechanisms for SUDEP may also open up a new avenue for both genetic and therapeutic research.39 Fatal tachyarrhythmia, however, are only one possible etiology for SUDEP. Other potential etiologies include, e.g., peri-ictal central apnea and bradyarrhythmias.1,40
We have retrospectively selected patients with medically refractory TLE of either side who had at least one CPS and one secondarily GTCS during presurgical video-EEG telemetry. Therefore, our study has exploratory character, and caution is warranted in generalizing our results to other epilepsy and seizure types. Potential confounders include differing day or night time of seizure onset, different levels of antiepileptic drugs, and chronological order of seizure occurrence. As these factors are very difficult to control for, we checked whether they might have systematically influenced our data (table e-1). CPS and GTCS occurred both at day or night time, thereby excluding a systematic bias. Drug reduction was instituted in 18 of the patients and reinstituted after occurrence of secondarily GTCS. It is unlikely that our results were influenced by this, because the impact of seizure type on ECG was no different in those patients with drug reduction and those without (data not shown). Patients were usually administered oxygen via a face mask following a GTCS, but not following a CPS. This treatment is unlikely to induce, e.g., sustained tachycardia, as it improves the overall oxygen supply. Despite these limitations, major confounders such as different properties of cardiovascular and autonomic nervous system in different patients were controlled for by the intraindividual comparison, thereby strengthening our conclusions.
AUTHOR CONTRIBUTIONS
Statistical analysis was conducted by Dr. Surges and Dr. Walker.
DISCLOSURE
Dr. Surges received research support from Deutsche Forschungsgemeinschaft. C.A. Scott received travel expenses for a lecture not funded by industry. Dr. Walker serves on scientific advisory boards for GlaxoSmithKline, Eisai Inc., UCB, and Sierra Neuro; has received travel expenses and honoraria for lectures or educational activities not funded by industry; serves as an Associate Editor of Epilepsia and Therapeutic Advances in Neurological Disorders and on the editorial boards of The Open Neurology Journal, Expert Opinion on Emerging Drugs, and Metabolic Brain Disease; receives royalties from publishing Understanding Epilepsy (BMA Family Doctor Books, 1995); has received speaker honoraria from Eisai Inc. and UCB; serves as a consultant to Sierra Neuro; estimates that 20% of his clinical effort is on Video-EEG telemetry; and has received/receives research support from the Medical Research Council [G0400136 (PI), G0802158 (Co-I), G0701050 (Co-I), and G069336 (Co-I)], the European Community, Wellcome Trust, and Epilepsy Research UK.
Supplementary Material
Address correspondence and reprint requests to Dr. Rainer Surges, Department of Epileptology, University of Bonn, Sigmund-Freud-Strasse 25, 53105 Bonn, Germany rainer.surges@googlemail.com
Supplemental data at www.neurology.org
Study funding: This work was undertaken at UCLH/UCL, which receives a proportion of funding from the Department of Health's NIHR Biomedical Research Centres funding scheme. R.S. was supported by a stipend from the Deutsche Forschungsgemeinschaft. The funding sources had no role in the study design, data collection, data analysis, data interpretation, or writing of the manuscript.
Disclosure: Author disclosures are provided at the end of the article.
Received June 19, 2009. Accepted in final form November 5, 2009.
REFERENCES
- 1.Tomson T, Nashef L, Ryvlin P. Sudden unexpected death in epilepsy: current knowledge and future directions. Lancet Neurol 2008;7:1021–1031. [DOI] [PubMed] [Google Scholar]
- 2.Ficker DM, So EL, Shen WK, et al. Population-based study of the incidence of sudden unexplained death in epilepsy. Neurology 1998;51:1270–1274. [DOI] [PubMed] [Google Scholar]
- 3.Langan Y, Nashef L, Sander JW. Case-control study of SUDEP. Neurology 2005;64:1131–1133. [DOI] [PubMed] [Google Scholar]
- 4.Surges R, Adjei P, Kallis C, et al. Pathological cardiac repolarization in pharmacoresistant epilepsy and its potential role in sudden unexpected death in epilepsy: a case-control study. Epilepsia. Epub ahead of print 2009 Oct 8. [DOI] [PubMed]
- 5.Schouten EG, Dekker JM, Meppelink P, et al. QT interval prolongation predicts cardiovascular mortality in an apparently healthy population. Circulation 1991;84:1516–1523. [DOI] [PubMed] [Google Scholar]
- 6.Watanabe J, Thamilarasan M, Blackstone EH, et al. Heart rate recovery immediately after treadmill exercise and left ventricular systolic dysfunction as predictors of mortality: the case of stress echocardiography. Circulation 2001;104:1911–1916. [PubMed] [Google Scholar]
- 7.Engel G, Beckerman JG, Froelicher VF, et al. Electrocardiographic arrhythmia risk testing. Curr Probl Cardiol 2004;29:365–432. [DOI] [PubMed] [Google Scholar]
- 8.Nei M, Ho RT, Sperling MR. EKG abnormalities during partial seizures in refractory epilepsy. Epilepsia 2000;41:542–548. [DOI] [PubMed] [Google Scholar]
- 9.Nei M, Ho RT, Abou-Khalil BW, et al. EEG and ECG in sudden unexplained death in epilepsy. Epilepsia 2004;45:338–345. [DOI] [PubMed] [Google Scholar]
- 10.Opherk C, Coromilas J, Hirsch LJ. Heart rate and EKG changes in 102 seizures: analysis of influencing factors. Epilepsy Res 2002;52:117–127. [DOI] [PubMed] [Google Scholar]
- 11.Zijlmans M, Flanagan D, Gotman J. Heart rate changes and ECG abnormalities during epileptic seizures: prevalence and definition of an objective clinical sign. Epilepsia 2002;43:847–854. [DOI] [PubMed] [Google Scholar]
- 12.Ronkainen E, Ansakorpi H, Huikuri HV, et al. Suppressed circadian heart rate dynamics in temporal lobe epilepsy. J Neurol Neurosurg Psychiatry 2005;76:1382–1386. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Schroeder EB, Whitsel EA, Evans GW, et al. Repeatability of heart rate variability measures. J Electrocardiol 2004;37:163–172. [DOI] [PubMed] [Google Scholar]
- 14.Aytemir K, Maarouf N, Gallagher MM, et al. Comparison of formulae for heart rate correction of QT interval in exercise electrocardiograms. Pacing Clin Electrophysiol 1999;22:1397–1401. [DOI] [PubMed] [Google Scholar]
- 15.Luo S, Michler K, Johnston P, Macfarlane PW. A comparison of commonly used QT correction formulae: the effect of heart rate on the QTc of normal ECGs. J Electrocardiol 2004;37 suppl:81–90. [DOI] [PubMed] [Google Scholar]
- 16.Tisdale JE, Patel R, Webb CR, et al. Electrophysiologic and proarrhythmic effects of intravenous inotropic agents. Prog Cardiovasc Dis 1995;38:167–180. [DOI] [PubMed] [Google Scholar]
- 17.Bateman LM, Li CS, Seyal M. Ictal hypoxemia in localization-related epilepsy: analysis of incidence, severity and risk factors. Brain 2008;131:3239–3245. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Lipka K, Bülow HH. Lactic acidosis following convulsions. Acta Anaesthesiol Scand 2003;47:616–618. [DOI] [PubMed] [Google Scholar]
- 19.Simon RP, Aminoff MJ, Benowitz NL. Changes in plasma catecholamines after tonic-clonic seizures. Neurology 1984;34:255–257. [DOI] [PubMed] [Google Scholar]
- 20.Natelson BH, Suarez RV, Terrence CF, Turizo R. Patients with epilepsy who die suddenly have cardiac disease. Arch Neurol 1998;55:857–860. [DOI] [PubMed] [Google Scholar]
- 21.Morita H, Wu J, Zipes DP. The QT syndromes: long and short. Lancet 2008;372:750–763. [DOI] [PubMed] [Google Scholar]
- 22.Kändler L, Fiedler A, Scheer K, et al. Early post-convulsive prolongation of QT time in children. Acta Paediatr 2005;94:1243–1247. [DOI] [PubMed] [Google Scholar]
- 23.Schimpf R, Borggrefe M, Wolpert C. Clinical and molecular genetics of the short QT syndrome. Curr Opin Cardiol 2008;23:192–198. [DOI] [PubMed] [Google Scholar]
- 24.Algra A, Tijssen JG, Roelandt JR, et al. QT interval variables from 24 hour electrocardiography and the two year risk of sudden death. Br Heart J 1993;70:43–48. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Anttonen O, Junttila MJ, Rissanen H, et al. Prevalence and prognostic significance of short QT interval in a middle-aged Finnish population. Circulation 2007;116:714–720. [DOI] [PubMed] [Google Scholar]
- 26.Gallagher MM, Magliano G, Yap YG, et al. Distribution and prognostic significance of QT intervals in the lowest half centile in 12,012 apparently healthy persons. Am J Cardiol 2006;98:933–935. [DOI] [PubMed] [Google Scholar]
- 27.Lu HR, Vlaminckx E, Hermans AN, et al. Predicting drug-induced changes in QT interval and arrhythmias: QT-shortening drugs point to gaps in the ICHS7B guidelines. Br J Pharmacol 2008;154:1427–1438. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Holbrook M, Malik M, Shah RR, Valentin JP. Drug induced shortening of the QT/QTc interval: an emerging safety issue warranting further modelling and evaluation in drug research and development? J Pharmacol Toxicol Methods 2009;59:21–28. [DOI] [PubMed] [Google Scholar]
- 29.DeSilvey DL, Moss AJ. Primidone in the treatment of the long QT syndrome: QT shortening and ventricular arrhythmia suppression. Ann Intern Med 1980;93:53–54. [DOI] [PubMed] [Google Scholar]
- 30.Cheng-Hakimian A, Anderson GD, Miller JW. Rufinamide: pharmacology, clinical trials, and role in clinical practice. Int J Clin Pract 2006;60:1497–1501. [DOI] [PubMed] [Google Scholar]
- 31.Kasarskis EJ, Kuo CS, Berger R, Nelson KR. Carbamazepine-induced cardiac dysfunction: characterization of two distinct clinical syndromes. Arch Intern Med 1992;152:186–191. [DOI] [PubMed] [Google Scholar]
- 32.Saetre E, Abdelnoor M, Amlie JP, et al. Cardiac function and antiepileptic drug treatment in the elderly: a comparison between lamotrigine and sustained-release carbamazepine. Epilepsia Epub 2009 Mar 21. [DOI] [PubMed]
- 33.Arrowood JA, Kline J, Simpson PM, et al. Modulation of the QT interval: effects of graded exercise and reflex cardiovascular stimulation. J Appl Physiol 1993;75:2217–2223. [DOI] [PubMed] [Google Scholar]
- 34.Mameli O, Caria MA, Pintus A, et al. Sudden death in epilepsy: an experimental animal model. Seizure 2006;15:275–287. [DOI] [PubMed] [Google Scholar]
- 35.Hilz MJ, Devinsky O, Doyle W, et al. Decrease of sympathetic cardiovascular modulation after temporal lobe epilepsy surgery. Brain 2002;125:985–995. [DOI] [PubMed] [Google Scholar]
- 36.Akalin F, Tirtir A, Yilmaz Y. Increased QT dispersion in epileptic children. Acta Paediatr 2003;92:916–920. [DOI] [PubMed] [Google Scholar]
- 37.Extramiana F, Antzelevitch C. Amplified transmural dis-persion of repolarization as the basis for arrhythmogenesis in a canine ventricular-wedge model of short-QT syndrome. Circulation 2004;110:3661–3666. [DOI] [PubMed] [Google Scholar]
- 38.Espinosa PS, Lee JW, Tedrow UB, Bromfield EB, Dworetzky BA. Sudden unexpected near death in epilepsy: malignant arrhythmia from a partial seizure. Neurology 2009;72:1702–1703. [DOI] [PubMed] [Google Scholar]
- 39.Nashef L, Hindocha N, Makoff A. Risk factors in sudden death in epilepsy (SUDEP): the quest for mechanisms. Epilepsia 2007;48:859–871. [DOI] [PubMed] [Google Scholar]
- 40.Nashef L, Walker F, Allen P, Sander JW, Shorvon SD, Fish DR. Apnoea and bradycardia during epileptic seizures: relation to sudden death in epilepsy. J Neurol Neurosurg Psychiatry 1996;60:297–300. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.