Data integration: Combined Imaging and Electrophysiology data in the cloud

Abstract

There has been an increasing effort to correlate electrophysiology data with imaging in patients with refractory epilepsy over recent years. IEEG.org provides a free-access, rapidly growing archive of imaging data combined with electrophysiology data and patient metadata. It currently contains over 1200 human and animal datasets, with multiple data modalities associated with each dataset (neuroimaging, EEG, EKG, de-identified clinical and experimental data, etc.). The platform is developed around the concept that scientific data sharing requires a flexible platform that allows sharing of data from multiple file-formats. IEEG.org provides high and low-level access to the data in addition to providing an environment in which domain experts can find, visualize, and analyze data in an intuitive manner. Here, we present a summary of the current infrastructure of the platform, available datasets and goals for the near future.

1. INTRODUCTION

Intracranial Electroencephalography (iEEG) and Magnetic Resonance Imaging (MRI) are equally important neuroimaging modalities for studying neural activity and structure [1], [2]. Accurate dynamics of neural activities and interactions can be studied across large number of electrodes and with high temporal resolution using iEEG [3]–[6]. Precise localization of this neuronal activity along with structural descriptions of neuronal pathways can be determined using MRI with different contrasts [7]. In addition, complementary features of neuronal activity can be studied using fMRI [8]–[10]. The explosion of technology in these modalities has spurred interest in their use in studying neuroscience and advanced translational research in neurology, particularly epilepsy.

Joint analysis of these multimodal datasets allows neuroscientists to study functional and structural relationships and has great potential to lead to vital discoveries in epilepsy otherwise not captured when studying modalities independently [11]. Additional data-modalities further impact the overall richness of a dataset, such as patient clinical history, genetics, electrocardiogram (EKG), etc. Integrating, analyzing, and sharing these complex datasets pose unique challenges to the data science community. Individual institutions have access to small data sets. Significant advancements in this field of study will be made using large data sets shared across multiple institutions. Unique challenges of sharing human biomedical data arise such as data format interoperability, de-identification of protected health information (PHI), and adherence to mandated government regulations.

The ability to share clinical metadata along with high-resolution data, such as iEEG and neuroimaging across multiple sites, often geographically sparsely distributed, requires novel infrastructure with a focus on data integration. A simple approach such as allowing collaborators to download the data to their local site for analysis is often not feasible, especially with the terabytes (TB) of data that comprise these datasets.

Over the past four years, our team of neuroscience and computer science experts has established a cloud-based resource for data sharing and collaboration, http://IEEG.org [12]. This platform provides data sharing and analysis capabilities to the neuroscience community, particularly in the epilepsy domain. Multiple neuroscience research centers are making their data available through the platform for collaborations where data access is controlled, and access to data is controlled by the data contributors.

Currently, the IEEG portal contains high quality iEEG and multimodal imaging from over 1200 subjects including 576 animal models (dog, mouse, rat, sheep, primate) and 733 patients with epilepsy. As this time of writing this article, there are 487 public datasets, 667 registered users, and 162 publicly accessible clinical datasets. These data were collected across multiple institutions throughout the world. Users originate from all 5 continents and represent institutions like UCLA in Los Angeles, CA and University Hospital Motol in Prague, Czech Republic. Table 1 summarizes the data available on the IEEG-Portal.

Table 1.

The number of patients by Imaging Modality per Institution (from the IEEG Portal dataset)

Number of Patients by Imaging Modality per Institution	Mayo Clinic	Hospital of the University of Pennsylvania
Structural Modalities
3T MRI - T1W	23	23
T2W	2	19
FLAIR	10	19
DWI/DTI		8
ECoG Pre-Implant Imaging (MRI/CT)	23	14
ECoG Post-Implant Imaging (MRI/CT)	23	21
6 Months Post-Implant Imaging (MRI/CT)		13
7T MRI		20
HARDI Imaging		20
Metabolic/Functional Modalities
SPECT (Ictal and Interictal)	20
18(F)-FDG PET	1
BOLD		20
ASL		20
MRS		20

Abbreviations for Imaging: Positron Emission Tomography (18(F)-FDG PET), T1-Weighted MRI (T1W), T2-Weighted MRI (T2W), Fluid Attentuated Inversion Recovery sequence (FLAIR MRI), Diffusion Weighted Imaging (DWI)/Diffusion Tensor Imaging (DTI), X-Ray Computed Tomography (CT), Single-Photon Emission Computed Tomography (SPECT), Blood oxygenation level dependent (BOLD), Arterial Spin Labeling (ASL), High angular resolution diffusion-weighted imaging (HARDI), 1(H) Magnetic Resonance Spectroscopy (MRS). All image sequences available as 3D NIfTI files.

Each human dataset can contain up to 100 electrodes and are recorded for 1–4 weeks continuously using sample rates as high as 32 kHz. Standard clinical epilepsy protocol images are also provided, including T1-weighted (T1W) isotropic axial, T2-weighted (T2W) coronal, FLAIR coronal and diffusion weighted image sequences. In addition, de-identified clinical metadata, such as patient medical and family history, medication history, Epilepsy Monitoring Unit reports and scalp EEG findings are available for a subset of the patients on the portal. There are currently 46 patients with intractable epilepsy who have at least intracranial EEG, pre-operative T1W/T2W/DWI MRI, ECoG post-implant MRI or CT as well as a full clinical report. Half of these patients are from the Hospital of the University of Pennsylvania and the other half are from the Mayo Clinic. Table 2 shows four example patients along with a sampling of the data available on the portal for these patients. All patients who have clinical reports available have lesional findings on histopathology and neuroimaging reported in the respective sections in the reports. These include lesions such as malformations of cortical development (focal cortical dysplasia or schizencephaly), vascular malformations (AVMs or cavernomas), and low-grade glial tumors. A significant minority of the patients currently on the portal has these lesional findings present on either histopathology or imaging.

Table 2.

Sample Patient Profiles from the IEEG Portal.

Patient ID	I002_P002	Mayo Clinic Study 005	I002_P005	I002_P006
Institution	Hospital of the University of Pennsylvania	Mayo Clinic	Hospital of the University of Pennsylvania	Hospital of the University of Pennsylvania
Epilepsy Type	Localization-related; Left frontal onset, left cortical dysplasia	Bi-Temporal Onset	Localization- related; Left temporal onset (secondary to hemorrhagic HSV encephalitis)	Localization-related; Symptomatic meningitis
Engel Outcome	1	1	4	1
Seizure Type	Simple partial and generalized tonic- clonic of left temporal onset	Focal Complex partial-endpoint of seizure- not evolving to secondary generalized	Complex partial seizures of left temporal onset and status epilepticus	Complex partial seizures of right temporal onset with secondarily generalized tonic-clonic seizure
# of Seizures	1		0	5
Age during First Seizure	4	21	30	11
Age at Admission	20	26	35	32
Sample Past Anti-Epileptic Medications	carbamazepine, topiramate, levetiracetam, clobazam, valproic acid	-	phenytoin, valproic acid, levetiracetam	carbamazepine, zonisamide
Imaging available	T1 MPRAGE, T2 FLAIR, Post-Implant T1 MPRAGE and FLAIR, Post-Implant CT, 6 months post- op T1, T2, FLAIR, DWI	MRI T1 pre- implant, SPECT ictal/interictal, MRI T1 post implant, CT post implant	Pre-Implant T1 MPRAGE and T2, Post-Implant T1 MPRAGE and FLAIR, Post-Implant Head CT	T1 MPRAGE, T2 FLAIR, T2 Susceptibility-Weighted, DTI, Pre-Implant T1 MPRAGE and T2 FLAIR, Post-Implant T1 MPRAGE and FLAIR, Post-Implant CT, 6 months post-op T1, T2, FLAIR

Four patients and their profiles on the IEEG Portal. Dates of hospital admission for epilepsy monitoring have been standardized to start on Jan-01-2000. All other dates have been adjusted accordingly. Note that some data is not available and marked by -. All patients who have lesional epilepsy will have lesion findings noted under MRI report summaries in the respective patients’ clinical reports. All lesional patients will have appropriate pathology and imaging findings reported, whether lesions are malformations of cortical development, vascular, or low-grade glial tumors.

Analyzing large scale EEG and neuroimaging data requires substantial computational resources. Leveraging cloud resources provides a scalable solution to benchmark experiments, share gold standard datasets, and advance towards more integrative collaborative research in the neuroscience community.

IEEG.org, like other databases such as the Human Connectome Project [13], the European EEG database (http://epilepsy-database.eu) [14], [15] and LONI IDA [16] are critical to the standardization of neuroimaging data analyses, avoiding bias, and allowing for significant research advances [15]. Benchmarking experiments (i.e. testing algorithms on novel data) requires a central body to curate “gold standard” training data and withhold testing data. IEEG.org has been developed to allow users to share data and use its resources to validate and benchmark new algorithms.

2. Architecture of IEEG.org

The IEEG-Portal is developed using the Google Web Toolkit in Java, and JavaScript. It is hosted on Amazon’s EC2 service and the data is stored on the Amazon S3 service with reduced redundancy. Figure 1 shows an abstract schematic of the various components of the IEEG.org infrastructure. All meta-information associated with the datasets as well as user-information and portal-state information is hosted using on Amazon RDS. Separate pipeline servers handle data import, wrangling and cloud-based data analysis. The IEEG.org platform has been developed with a continuous focus on scalability and security. Users have full control over who can see their data and track their usage. A variety of tools are available to upload, access, edit, and annotate data by the user in both a web-environment and through multiple API’s. MongoDB (http://www.mongodb.org) can be used as a persistent cache to increase efficiency and reduce traffic/cost from the cloud. Accessing data through MATLAB is easy and fast; the toolbox fetches 470,000 values within 10 milliseconds utilizing the local cache. Fetching the same amount directly from the portal takes about 25 milliseconds (depending on bandwidth). Efficiency of data transfer increases when larger blocks of data are requested through the MATLAB toolbox. Images, which are currently saved as zip files, are quickly downloaded over FTP.

Figure 1.

IEEG Portal Setup

Schematic of the cloud infrastructure supporting IEEG.org platform. While a local persistent cache is available for clients to work with parts of data locally, majority of the data computation occurs on the cloud, including storage and pipelining.

The platform provides mechanisms to automatically import and wrangle datasets from various data-formats. The import pipeline process converts the data to a standardized format, associate meta-information with the datasets, and makes them available through the platform. Files that are not recognized as imaging or time series data are made available as direct downloads. All EEG and imaging are then de-identified again by stripping all header information in data files. In addition, clinical metadata and reports are re-written by removing any identifiable information, including physician names and recorded seizure times.

Investigators interested in using the portal can request a user account by visiting http://www.IEEG.org. All users of the portal have to agree to a Data Usage agreement. Data providers can set permissions for each dataset specifying which users can find, view, and edit their datasets, or they can make the datasets globally visible to the portal user community with a click of a button. This makes the platform ideally suited for inter-institutional collaborations where data is selectively shared with a research team.

Recognizing the various user-groups of the platform (data scientist, clinicians, educators) and levels of expertise in these groups, we provide an architecture with data and metadata services that may be invoked from a Web browser or from a remote MATLAB or Java “toolbox”. In addition, to facilitate the wide range of scenarios regarding data sharing, the platform provides flexible authorization and access control capabilities. The platform will support OAuth authorization in the near future, allowing users to login using their LinkedIn, Google, or Facebook account (by March 2015).

2.1 Web Console and Viewer

The web-console provides the ability to view time series and imaging data (Figure 2). Users can download additional files that are associated with these datasets, such as clinical metadata, zipped imaging data, other metadata. Users can also track usage of their datasets by other users of the platform and engage in group-discussions within “Project Groups”. The web-interface also allows users to upload data and edit information for each dataset they own.

Figure 2.

Annotated figure of a seizure (from EEG.org)

A screenshot of the IEEG.org portal’s web viewer. The annotated window (top) shows a recorded seizure from Patient ‘Mayo Clinic Study 005’ (Table 2). On the bottom is an example of imaging dataset: T1-weighted MPRAGE image for this patient. In this example, the patient is viewed using the radiological orientation (LAS).

The EEG Viewer runs in all major web browser and provides comprehensive functionality with a small local computational footprint. It provides the ability to search for a particular dataset, or for particular events within datasets to allow the user to quickly select only segments of data that are of interest to them. This feature is essential in the large data sets available on IEEG.org. For instance, this search feature can be used to filter the dataset to a particular imaging sequence or animal dataset.

The documentation for interfacing with the portal is available both on http://www.IEEG.org as well as supplementary material S1. The documentation describes the functionality of the IEEG-Portal and its exposed API for Java and Matlab. It contains multiple example tutorials and a thorough explanation of the terminology used on the IEEG-Portal.

2.2 APIs for 3^rd party software and Other Tools

The IEEG.org platform provides a set of APIs to interact, and manage data on the platform. This includes methods to upload datasets and annotations, download datasets and stream data to a local host. Although most API methods are targeting time series at this point, we intent to provide similar programmable access to all imaging data. A MATLAB toolbox is available that allows users to easily interface with the platform from their MATLAB session. Returned content can be locally cached using MongoDB to minimize redundant traffic from the cloud platform.

3. Imaging Datasets

Most currently available images are the standard epilepsy imaging sequences used at clinical institutions. These images are saved in NIFTI images and are not altered in any other way from the original DICOM images. Figure 2 shows an example MPRAGE MRI image sequence for patient ‘I002_P002_D001’, viewed in ITK-SNAP [17]. All identifiable header information is stripped from the images. The names of the images identify the image contrast and imaging modality. All images are set in the original coordinate space of the scanner. The origin of each image is similarly dependent on the origin set during scan time. All images can be reoriented into the appropriate coordinate space and displayed in the radiological orientation (LAS) using ITK-SNAP. Table 3 summarizes some of the different modalities of imaging for some of the patients on the portal.

Table 3.

Imaging Modalities and the associated contrast/sequence.

Modality	Contrast/Sequence	Notes
3T MRI	Localizer
3T MRI	T1 (MPRAGE)	Standard 1mm isotropic
3T MRI	T2	T2-weighted
3T MRI	FLAIR
3T MRI	ASL	Pseudo-Continuous Arterial Spin Labeling
3T MRI	DWI/DTI	Diffusion Weighted Imaging
3T MRI	Susceptibility-Weighted
3T MRI	Contrast Enhanced (Gadolinium)
7T MRI	T1 (MPRAGE)	7T images taken at research scanners
CT	Pre-Implant Stealth	CT Scan taken right before electrode grid implantations
CT	Post-Implant Stealth
PET	(18)F-FDG

Notes indicate details on the imaging sequence used.

There are a number of imaging modalities and contrasts available for the datasets, depending on the clinical epilepsy imaging protocol used at the institution. For instance, the Mayo Clinic (Table 1) regularly obtains interictal and ictal SPECT for SISCOM imaging. Most institutions obtain Head CTs both pre-implant and post-implant but not all institutions acquire MRI images post-electrode implantation.

In addition to these iEEG tools, the web console also contains a BrainMapper application, which is an easy to use “drag and drop” application that automatically co-registers CT and MRI electrode implantation images in order to provide an accurate and reliable 3D reconstruction of the electrodes on the surface of a patient’s brain [18]. Figure 3a shows a snapshot of the BrainMapper application. Within the application, the co-registered images can be viewed in 3-dimensional space. Figure 3b shows the final image in the application, which uses rigid and non-rigid registration along with prior-based probabilistic segmentation in order to minimize errors caused by parenchymal shift during the implantation procedure as well as “unbury” surface electrodes that appear buried in cortical tissue.

Figure 3.

BrainMapper Application

IEEG.org provides a desktop tool to coregister MRI and CT to visualize subdural electrode locations on a 3D parcellated brain. This tool has been quantitatively validated and is available for free download [18]. (A) Screenshot of BrainMapper application, which shows an intuitive user interface along with a progress bar (B) Sample results that can be visualized in BrainMapper. Cortical parcellations allow users to visualize location of each electrode in an implant.

The application uses the atlas of Non-Rigid Image Registration Evaluation Project (NIREP http://www.nirep.org) from the University of Iowa, which is based on 16 normal adult T1-weighted brain scans and contains 32 cortical gray matter labels. Skull and other non-brain tissues in the subject MR images are stripped by applying the BET [19]. The patient MR image is non-rigidly registered to the NIREP atlas using cross-correlation as the similarity metric. The high level of accuracy of our co-registration method was quantitatively confirmed [18]. The images are all in the patient’s original T1 image coordinate frame.

In the future, there will be multiple expanded tools for images for all patients on the portal. Specifically, we are in the process of implementing an online DICOM and 3D surface render viewer using WebGL API. This tool will allow users to view imaging datasets on the portal directly in their web browser rather than simply downloading the dataset. The pipeline for the BrainMapper will be included in this viewer, allowing for users to view the co-registered electrodes on the surface of brains in all patients who have Pre-Implant and Post-Implant MRI and CT imaging. In addition to these electrodes, important computational iEEG and imaging biomarkers, such as seizure onset zones, marked structural dysplasias and BOLD activated regions will be displayed on cortical surface renderings as density heat maps. This tool can display other user-generated computed surface maps, allowing for improved integration of EEG and multimodal imaging.

4. Patient Consent and De-Identification

All patients from the Hospital of the University of Pennsylvania and the Mayo Clinic are consented post-surgery in allowing their clinical metadata, ECoG recordings and neuroimaging to be uploaded onto the portal. Since the University of Pennsylvania and the Mayo Clinic are founding members of the portal, our institutional IRBs allow us to upload patient data from consenting patients. Different institutions that have contributed and would like to contribute in the future can consent their patients through permission of their institutional IRB.

Other institutions that would like to contribute should make sure they first get consent from the patient and has an approved IRB to included these patients. In addition, these institutions should de-identify all data before uploading on the portal. IEEG.org provides a tool that can de-identify DICOM images before they are uploaded to the platform. In the supplementary materials, we have included a copy of the user agreement (Figure S2) as well as a copy of the agreement that an owner of a dataset agrees to when contributing data through the portal (Figure S3) to the supplementary information of this article. Finally, a basic diagram of the flow of data from submission to loading onto the portal is shown in Figure S4.

5. Long term goals

Table 4 shows a timetable of the expected changes we hope to implement in the coming year. The platform has been developed from the beginning with sustainability in mind, and currently these resources are provided to the scientific community free of charge. Additional funding strategies may be implemented in the near future to ensure its continuation. For example, having a free tier of data-access, plus additional ‘pay-as-you-go’ services where costs are passed onto the user depending on the amount of data that is downloaded/analyzed.

Table 4.

Timetable for upcoming changes and improvements to the IEEG.com portal

Expected Date	Updates and Improvements	Notes
Fall 2015	Interactive Web-based DICOM, and NIFTI viewer, linking between EEG and Imaging data.	Allows viewing of BrainMapper results (electrode grid location), results from image analysis tools (registration/segmentation)
Late 2015	Image analysis on the cloud	Running various image analysis pipelines over imported imaging data.
		Search textual and numeric content of documents, metadata
		LinkedIn, Facebook, and Mendeley
Within 2 years	Sophisticated Search Capability, Interlink with external databases, Social Networking, scaling of service.	Databases include LONI IDA, GenBank, Pubmed, Google Scholar

The IEEG.org platform initially focused primarily on large iEEG datasets, but now handles many other data formats including imaging. Over the next several years, imaging will become an increasingly larger part of the platform and, specifically, integration between iEEG and imaging datasets. A particular effort will be to interlink with other, external databases that already exist such as GenBank (genetics) and LONI (imaging). Providing a ‘single sign-on’ could be an initial step towards providing a truly integrated user-experience.

As the platform becomes more diversified, through inclusion of new data-modalities and support for additional data-formats, it becomes more important to provide useful methods to search for particular information within these multi-modal datasets. For example, one can be interested in all imaging data from people with frontal lobe epilepsy that have a minimum of 3 number of seizures in the associated iEEG. This requires matching of keywords in text (Patient Information), expert annotations (‘Seizures’), and return specific imaging data (Figure 4). We are developing sophisticated search capabilities, which will facilitate search over the textual and numeric content of documents, metadata, and data associated with a data set – and queries linking these resources [20].

Figure 4.

Querying information on the Portal

Data graph relating patient notes, electrophysiology and imaging. Key-word search capabilities allow users to query across data modalities such as documentation, annotations in iEEG and Electro-placement information in 3D models of MRI Imaging.

Management of data on the IEEG.org platform will rely predominantly on the users of the platform and specifically the owners of the data hosted on the platform. We envision that there will be a subset of datasets on the platform that will be used more often by the community and, similar to Wikipedia articles, be meticulously maintained over time. Datasets that are less used might be archived to cheaper storage (i.e. Amazon Glacier Storage) but remain accessible.

As mentioned prior, we plan to provide integration with several professional platforms, such as LinkedIn, Facebook, and Mendeley to emphasize the social aspects of data sharing. This will allow users to look up, and contact owners of particular datasets and find publications that are associated with particular datasets. The entire platform is developed as an open-source project. The modular framework of the platform fully supports the development of additional viewers, analysis, or conversion tools.

A short-term goal is to provide an interactive Web-based DICOM, and NIFTI viewer which users can use to view and annotate imaging data. We expect to have this capability before the end of 2015. Novel search capabilities, will also be available to search for imaging datasets that meet specific search criteria (e.g. “has 7T MRI” or “contain Radiologist-identified lesions”).

We believe that over time, some users will want to customize the platform to their own needs. Therefore, the entire platform is provided as an open-source project that users can deploy on their own machines. Source-code is available at: https://code.google.com/p/braintrust/. The portal continues to grow thanks to contributions from many epilepsy researchers across the world. Though initially geared towards epilepsy research, the database is capable of handling any large dataset. For instance, neuroscience research datasets including combined EEG and fMRI can be easily uploaded on the portal. High fidelity long-term recordings from animals are already uploaded for use in seizure prediction and detection analysis. The platform is already capable of handling scalp EEG data, which will allow storage of gold standard scalp EEG datasets. These gold standard EEG datasets can validate inverse problem methods that users can run using the available APIs. Such methods already exist for iEEG [21], [22], which can be further developed and validated on the patients currently on the portal.

5. Conclusion

We introduce the platform IEEG.org (www.IEEG.org) and highlight its added value for sharing and analyzing large complex biosignal datasets, including imaging. The development of the platform is focused on providing a truly scalable solution for connecting large, complex scientific data with domain experts, and to provide intuitive access for various user communities (i.e. educational, data-scientists, clinicians). IEEG.org leverages the unique novel features that today’s ‘cloud-environments’ support to provide a scalable, cost-efficient solution to sharing and analyzing datasets.

Taking advantage of the power and utility of data assembled in a collaborative way, this method of data sharing allows investigators to test specific algorithms (such as seizure predication algorithms in the field of epilepsy) on a wide range of patients and disease patterns while maintaining balance in the dataset. In addition, novel biomarkers can be discovered efficiently by quickly accessing large collections of data, such as the EEG and neuroimaging (MRI/CT/PET/SPECT) currently available on the portal. A large investment by the NIH allowed for a design of a fast, secure and easy-to-use portal. We expect that the platform will continue to grow in both size and functionality over the upcoming years, and extend to fields outside of epilepsy research.

Highlights.

• We introduce an online epilepsy data platform capable of handling large datasets.

• There are more than 1200 animal and human datasets (both public and private).

• IEEG.org provides toolboxes for analysis of EEG and neuroimaging data.

• IEEG.org is a versatile platform for neuroscience data sharing.

Abbreviations

iEEG

Intracranial Electroencephalogram

EEG

Scalp Electroencephalogram

EKG

Electrocardiogram

API

Application Program Interface

T1W

T1-Weighted MRI

T2W

T2-Weighted MRI

FLAIR

Fluid-Attenuated Inversion Recovery MRI

BOLD

Blood Oxygen Level Dependent imaging

ASL

Arterial Spin Labeling

HARDI

High angular resolution diffusion imaging

MRS

Magnetic Resonance Spectroscopy

PET

Positron Emission Tomography

SPECT

Single-photon Emission Computed Tomography

MP-RAGE

Magnetization-Prepared Rapid Gradient-Echo MRI

DICOM

Digital Imaging and Communications in Medicine

NIfTI

Neuroimaging Informatics Technology Initiative Data Format