Heads in the Cloud: A Primer on Neuroimaging Applications of High Performance Computing

Abstract

With larger data sets and more sophisticated analyses, it is becoming increasingly common for neuroimaging researchers to push (or exceed) the limitations of standalone computer workstations. Nonetheless, although high-performance computing platforms such as clusters, grids and clouds are already in routine use by a small handful of neuroimaging researchers to increase their storage and/or computational power, the adoption of such resources by the broader neuroimaging community remains relatively uncommon. Therefore, the goal of the current manuscript is to: 1) inform prospective users about the similarities and differences between computing clusters, grids and clouds; 2) highlight their main advantages; 3) discuss when it may (and may not) be advisable to use them; 4) review some of their potential problems and barriers to access; and finally 5) give a few practical suggestions for how interested new users can start analyzing their neuroimaging data using cloud resources. Although the aim of cloud computing is to hide most of the complexity of the infrastructure management from end-users, we recognize that this can still be an intimidating area for cognitive neuroscientists, psychologists, neurologists, radiologists, and other neuroimaging researchers lacking a strong computational background. Therefore, with this in mind, we have aimed to provide a basic introduction to cloud computing in general (including some of the basic terminology, computer architectures, infrastructure and service models, etc.), a practical overview of the benefits and drawbacks, and a specific focus on how cloud resources can be used for various neuroimaging applications.

Introduction

Continued technological advances in magnetic resonance imaging (MRI) hardware that promote increased spatial and temporal resolution imaging, as well as the increasing sophistication of post-processing pipelines for advanced neuroimaging modalities—eg, functional MRI, diffusion imaging, myelin water imaging, etc.—continue to increase the computational resources required for the subsequent data analysis. Particularly in large-scale, multi-center studies such as the Human Connectome Project (http://humanconnectome.org)1–3, the sheer amount of imaging data (which is expected to be on the order of 100TB by project’s end)—and therefore the computational resources needed to perform the corresponding analyses—are extremely large. Considering the ever-increasing need for storage and computing resources, a relatively new concept called cloud computing was introduced.4 Cloud computing hides most of the details about the underlying computer resources and their allocation, and offers end-users an illusion of unlimited resources through simplified interfaces (Fig. 1). In the past decade, several companies (eg, Microsoft, Google, Amazon, etc.) have started offering various cloud computing solutions, and an increasing number of government research institutes and research universities now have access to cluster and/or grid computing platforms.

Figure 1.

Basic diagram of cloud- or grid-based computing. Each of the service models discussed in this primer—ie, IaaS (Infrastructure as a Service), PaaS (Platform as a Service), SaaS (Software as a Service), and BiAaaS (Brain Imaging Analysis as a Service)—are depicted, with examples, from the lowest level of abstraction (left) to the highest level of abstraction (right) within the cloud. It should be noted that as a result of this abstraction, PaaS includes the underlying IaaS, and both SaaS and BiAaaS (which is really just a specialized instance of SaaS) include the underlying PaaS and IaaS necessary to run the particular application or analysis.

Although cloud computing has obvious benefits for storing and analyzing extremely large data sets—eg, freely available data sets from the OpenfMRI database (https://openfmri.org/), NeuroVault database (http://neurovault.org/), Human Connectome Project (http://humanconnectome.org), 1000 Functional Connectomes Project (http://fcon_1000.projects.nitrc.org/), etc.—there are also other neuroimaging analysis techniques for which cloud-based computing may be more efficient, more expedient, and perhaps necessary, even for smaller datasets or single subjects. Therefore, some of these more routine applications of cloud computing will also be addressed in this review, along with some examples and a practical discussion about how most neuroimaging researchers— even those with relatively limited technical proficiency—can access and leverage high-performance computing resources to improve their day-to-day workflow.

CPUs, GPUs, Clusters and Clouds

Since this article is aimed primarily at neuroimaging end-users (eg, cognitive neuroscientists, psychologists, psychiatrists, neurologists, neuroradiologists, etc.) who may not have much in the way of formal computer science training, it is perhaps worth briefly reviewing some of the basic terminologies and principles regarding computer hardware.

In colloquial terms, the central processing unit (CPU) is the brain that serves as the computational hub of most computers. It is the primary circuit responsible for carrying out the basic arithmetic, logic, control and input/output (I/O) operations that are required to run most software (ie, computer programs). Although commonly thought of as such, CPUs are not technically single entities, but rather integrated circuits that comprise separate sub-components, including: 1) an arithmetic logic unit (ALU) that performs all of the arithmetic and logical operations, 2) several very fast but very small storage units (eg, a few kilobytes) called processor registers and level 1 (L1) cache that store outputs from the ALU, and 3) a control unit (CU) that communicates with the memory and coordinates the operations of the ALU (Fig. 2); however, practically all modern CPUs are integrated circuits in which these components are combined on the same silicon chip.5 Taking this concept even further, multi-core processors (eg, dual-core, quad-core, etc.)— where each CPU actually has multiple CUs and ALUs within the same circuit—are now becoming increasingly popular due to their enhanced parallel processing abilities and lower power consumption (and heat generation) per clock cycle compared to conventional (ie, single-core) CPUs.

Figure 2.

Simplified diagrams to highlight the main features of (and differences between) single-core CPUs, multi-core CPUs, and GPUs. Although each of the ALUs in single- and multi-core CPUs are more powerful than those found in contemporary GPUs, the large number of ALUs (ie, hundreds or thousands) make GPUs well-suited for processing large numbers of operations that can be run in parallel.

While CPUs consist of either one or a relatively small number of very powerful processing units, and are optimized for performing large, serial (ie, sequential) operations, graphics processing units (GPUs) have massively parallel architectures that are composed of hundreds or thousands of less powerful cores. For these reasons, GPUs are much less expensive to produce and much more energy-efficient than CPUs on a per core basis, and therefore offer an efficient solution for running multiple—albeit smaller—tasks in parallel (ie, simultaneously). Although combining GPUs with CPUs was initially developed to handle ultra-high-definition graphics (hence their name), the application of GPU-accelerated computing is rapidly growing in popularity for other data intensive applications that can be run in parallel, including image processing, genomic analyses, and many other scientific/engineering applications.6

From an infrastructure standpoint, hybrid CPU-GPU (ie, combined CPU + GPU) workstations can either be standalone towers or rack-mounted systems, and can range in price from approximately $1000 USD (for a relatively simple tower set-up) to tens of thousands of dollars for top-of-the-line systems with several multi-core CPUs, multiple GPUs, and relatively large amounts of random access memory (RAM).

In contrast to CPU and hybrid CPU-GPU standalone systems, there are also distributed high performance computing architectures that interconnect and therefore leverage the collective computational power of multiple machines (or nodes). For example, a cluster is a group of nodes (ie, CPUs or GPUs) that are connected by a high bandwidth and low latency local area network (LAN). While clusters can vary drastically in terms of their size and computational power (a cluster could technically be as small as two or three computers), the other common types of distributed systems, called grids and clouds, are typically larger in scale and are often geographically distributed. Overall, the concepts of grids and clouds are very similar, except for the fact that a grid is typically a collection of interconnected clusters that are each owned by different institutes or organizations— and are geographically distributed in multiple locations—to leverage the collective computational power of these shared resources. One example of a well-known grid is SETI@ home (http://setiathome.ssl.berkeley.edu/), which uses donated clock cycles from registered users’ internet-connected computers to analyze radio telescope data in the search for extraterrestrial intelligence (SETI). On the other hand, a cloud is usually a collection of computers or clusters that are all owned by a single entity (eg, Amazon Web Service, IBM Cloud Computing, Google Cloud Platform, etc.). The primary advantage of clusters, grids and/or clouds is that they can offer levels of performance and processing speed that are not possible on standalone CPU or GPU workstations, particularly for computations or analyses that can be run in parallel. Because of these overarching similarities, the terms cloud, cluster, and grid will be used interchangeably for the remainder of this paper.

Types and Availability of Cloud Computing Resources

Generally speaking, cloud resources may be offered in various service models, and of these service models, the three most common include:

1. Infrastructure as a Service (IaaS) is the most basic service model (ie, the lowest level of cloud computing; Fig. 1), where the cloud service provider simply supplies computer hardware (eg, CPU and/or GPU clock cycles for computational processing, hard disk space for data storage, or other infrastructure components) to its users, and takes responsibility for handling the routine system maintenance, backups, etc. IaaS customers typically pay on a fee for service, pay-as-you-go model (eg, by the hour, week, month, or based on the number of clock cycles, the amount of hard disk space occupied, or the amount of data uploaded/downloaded from the cloud); however, it should be noted that members of many government research labs, private research institutes and academic institutions (ie, universities and colleges) may have access to free IaaS resources if their organization is affiliated with a supercomputing grid. Either way, however, the IaaS model eliminates the up-front expenses of purchasing in-house hardware and the ongoing expenses associated with system repairs, upgrades and administration. Moreover, since IaaS platforms offer highly scalable resources, they are exceptionally well suited for users with infrequent or intermittent high-performance computing needs. Examples of leading IaaS providers include Amazon Web Services, Google Compute Engine, Google Cloud Storage, DropBox, Microsoft Azure and IBM SmartCloud, etc. Also, it should be noted that most government and academic computer clusters and research grids offer IaaS for their members.

2. Platform as a Service (PaaS) is an intermediate cloud service model, in which a cloud service provider delivers both the hardware and basic software tools (eg, an operating system and basic middleware such as Java, C++, etc.) necessary to create, develop and test new applications in a run-time environment. We mention PaaS here for completeness; however, because PaaS platforms are almost exclusively used by software developers, and are rarely (if ever) used by neuroimaging researchers, they will not be discussed further.

3. Software as a Service (SaaS) is a software distribution model in which applications are hosted by service providers and made available to customers over a network, which is why SaaS products are sometimes also referred to as hosted applications. In this model, a vendor can either host commercially available software for customers and deliver it over the web, or give customers network-based access to a single copy of an application created specifically for SaaS distribution. In contrast to the traditional model of software distribution (ie, when software is purchased for and installed on standalone computers), the SaaS model removes the need for organizations to manage software installations and also offers several other benefits, including automatic updates, better compatibility for collaborative projects (ie, since users will all be using the same software version), global accessibility, and reduced cost (ie, compared to buying permanent standalone versions). Again, the potentially reduced cost may be attractive, especially for software that is infrequently or only intermittently used over a fixed time period. Examples of SaaS models include Adobe Creative Cloud, Microsoft Office 365, Google Apps, Google Docs, Google Translate, Concur, Cisco WebEx, etc. It should be noted that basic versions of SaaS software are sometimes free (or free for a limited “trial period”), and that many government institutions and research universities have subscriptions or site licenses for at least some commercial SaaS applications. For this reason, it is advisable to check for basic, trial, and/or institutional licenses before purchasing a particular SaaS product. Brain Imaging Analysis as a Service (BiAaaS) is an emerging sub-class of SaaS that provides end-to-end solutions—including software, image processing pipelines, brain atlases and/or data sets, and also integrates the underlying platforms and infrastructure—for cloud-based neuroimaging analyses. Two recent examples of BiAaaS include MRICloud and NITRC-CE, which will be discussed in greater detail below.

Furthermore, each of these cloud service models can be deployed in publicly accessible clouds that are owned and operated by Cloud Service Providers to offer cloud resources (usually for a fee) or private clusters/grids/clouds that are operated by research institutions and are usually free for institutional members. Each of these infrastructure models will be briefly discussed below.

Publicly accessible clouds

Google and Amazon are two examples of publicly available cloud service providers whose resources can be utilized by anyone, including neuroimaging researchers, for a number of different applications. The Google Cloud, for example, provides everything from storage capabilities (eg, for data storage, static web hosting, etc.) via Google Cloud Storage (https://cloud.google.com/storage/) to scalable ultra high-performance computing infrastructure for data analysis via the Google Compute Engine (https://cloud.google.com/compute/), which bills by the minute with no upfront costs, so users do not pay for unused computing time. Similarly, the Amazon Web Services offers a full range of storage solutions via the Amazon Simple Storage Service (Amazon S3) (https://aws.amazon.com/s3/) to ultra-high performance computing via the Amazon Elastic Compute Cloud (Amazon EC2) (https://aws.amazon.com/ec2/). In both cases, large data sets can be stored and large-scale workloads can be run on the cloud provider’s infrastructure. A summary of cloud-based services provided by Google and Amazon is given in Table 1, and a partial list of available cloud solutions and tools is provided in Table 2.

Table 1.

Service models of Google and Amazon cloud computing solutions.

SERVICE MODEL	GOOGLE	AMAZON
IaaS (Infrastructure as a Service)	Google Compute Engine (LP), Google Cloud SQL, Google Cloud Storage, etc.	DynamoDB, ElastiCache, RDS, etc.
SaaS (Software as a Service)	Google Docs, Google Prediction API, Google PageSpeed, Google Translate, Google BigQuery etc.	Cloud Search, SES, SNS etc.
PaaS (Platform as a Service)	Google App Engine	Appstream

Table 2.

Partial list of cloud computing solutions and tools, their primary applications, and cost models (adapted in part from Tsaftaris).59

SOLUTION NAME	DESCRIPTION	PRIMARY APPLICATION(S)	COST MODEL
cycle computing	Offering HPC clusters at an hourly rate based on Amazon Web Service (AWS) solutions	• Big jobs capable of using thousands of CPUs • No tool to build your own cloud-based infrastructure	• Hourly billing
StarCluster	An open source toolkit to automate building, configuring, and managing clusters of virtual machines on Amazon cloud	• Starting scientific computation in a cloud • Users should install specialty software packages	• N/A
OpenCPU	The OpenCPU server provides HTTP API to run R programs remotely	• To run R programs on the cloud by installing the OpenCPU server	• N/A
CPUsage	Provides an environment to install and configure applications. It then packages and deploys the local environment to the cloud	• Starting scientific computation in a cloud	• Per minute billing
Wakari	A Python-based data analytics tool hosted in the cloud. Each Wakari node is configured with extensive Python packages and it is browser accessible	• To do Python-based data analysis	• Basic free and monthly
CBRAIN	A data processing platform specifically designed for neuroimaging research	• To do neuroimaging research	• Free for Compute Canada users
Compute canada	A cloud solution enabling users to install their own operating system and software	• Start scientific computing in a cloud	• Free for compute Canada users
PiCloud	Acts as middleware between users and Amazon Web Service (AWS), with the goal of streamlining and simplifying scientific computing in the cloud for end-users	• Run Python-based scientific computing in a cloud	• Sub-second billing for CPU time • Additional charges for data storage and download

Private clouds

In addition to public clouds, there are also private or semi-private grids that are accessible to users from particular institutions and/or geographical regions. For example, Compute Canada (https://www.computecanada.ca/) offers freely available cloud computing and storage solutions to researchers affiliated with Canadian Universities. The infrastructure and support for these kind of systems are often provided at no charge to individual researchers, and therefore offer tremendous value. For this reason, researchers who are interested in getting involved in cloud computing for data storage and/or analysis purposes should: 1) find out whether their institutional affiliation entitles them to access any private research grids, 2) determine what, if any, costs are associated with using these resources (often they are free), 3) familiarize themselves with the capabilities of the available clusters and/or grids, and 4) determine whether there is any local IT support that may be able to assist with initial setup, etc.

If institutional access is available, these free (or ultra low-cost) private clouds will be sufficient for most neuroimaging applications, and therefore an attractive option because of the low cost and abundant technical support. Using Compute Canada as an example, the available IaaS platform is completely free to institutional members, is scalable and capable of handling extremely large computing and data storage requirements, and is flexible enough to accommodate various research themes (eg, drug development, meteorological modeling, genomic analyses, advanced image processing, etc.). Additionally, the Compute Canada infrastructure platform is also combined with local and regional support services in order to help new users get started and to ensure that all users are making efficient use of the shared resources. Compute Canada’s regional partners include ACENET in Atlantic Canada, Calcul Quebec, Compute Ontario, and WestGrid in Western Canada, each of which works together with their associated institutions in order to ensure that everyone has access to both the platform and local support.

Although Compute Canada and other large-scale research grids offer enormous computational power at low or no cost, one potential disadvantage is that jobs can get backlogged in a queue depending on the overall demand for computational resources. In order to minimize this issue, Compute Canada invites prospective users to write a proposal every year and identify the number of CPU hours and the amount of storage that they would like to access. Depending on the proposal and past history of resource utilization by the researcher, national computing resources are then pre-allocated. However, especially if these resources have not been pre-allocated, one alternative is to run certain jobs within smaller grids (eg, WestGrid) or on local clusters, where there may be a shorter queue and/or jobs can be given a higher priority within the sequence. This will of course depend on the size and complexity of the proposed analyses, but we have had great success in the past running our large batch processes on the Grex cluster at the University of Manitoba, rather than using higher levels of the cloud (ie, WestGrid or Compute Canada). The cluster—which, in our case, has 316 nodes and approximately 4,000 total cores—may have a shorter queue for “walk-on” CPU time, and still offers enough computational power for all but the largest image-processing and analysis jobs (eg, as long as the number of subjects and/or time-points in the data set does not exceed the number of nodes in the cluster, they could all still theoretically be processed in parallel).

MRI-Specific Cloud Resources

In addition to the generic types of cloud service and infrastructure models discussed above, there are also a growing number of MRI-specific cloud computing resources that are available for neuroimaging researchers. Although it is beyond the scope of this review to provide an exhaustive list of these resources (not least of which because the list is continually evolving and would therefore be almost immediately out of date), we would like to point to a few specific examples.

Given the unique and computationally demanding nature of brain imaging research, and all of the specialized software necessary to analyze advanced brain imaging data, a new category of cloud service model—referred to as Brain Imaging Application as a Service (BiAaaS)—has recently been proposed,7 which combines the IaaS and SaaS models, and is specifically tailored for brain imaging analyses. In addition to the DiffeoMap toolbox within MRIStudio (discussed below within the context of performing high-dimensional, non-linear normalizations), another MRI-specific cloud computing resource that falls under the BiAaaS model is the recently released MRICloud (https://mricloud.org/), which is a web-based neuroimaging analysis platform that was developed to provide fully-automated, online diffusion tensor imaging (DTI) analysis pipelines that integrate quality control measures8,9 and advanced preprocessing methods such as automated brain parcellation using Multiple-Atlas Likelihood Fusion.10,11 The advantages of this or any other BiAaaS model are three-fold: 1) the developers can deploy upgrades to their pipelines and/or algorithms in real-time, 2) all of the computationally-demanding analyses automatically use supercomputing resources to drastically reduce the processing time compared to having individual users run the software on standalone workstations, and 3) BiAaaS platforms seamlessly integrate the software and hardware, and are relatively easy to use (compared to more general cloud computing alternatives). However, the downside of BiAaaS platforms is that they are usually tailored to a relatively small number of specialized neuroimaging purposes, and are therefore inherently limited to those specific applications.

However, in order to bypass this limitation, the Neuroimaging Informatics Tools and Resources Clearinghouse (NITRC) has recently released the NITRC Computational Environment (NITRC-CE; https://www.nitrc.org/projects/ nitrc_es/), which is an on-demand, cloud computing platform that has been pre-configured with a wide range of common neuroimaging software packages—eg, AFNI (http://afni.nimh.nih.gov/afni/), ANTS (http://picsl.upenn.edu/software/ ants/), FSL (http://fsl.fmrib.ox.ac.uk/fsl/), FreeSurfer (http://freesurfer.net/), etc.—and allows users to add their own commercial or open-source software packages as well—eg, Matlab (http://www.mathworks.com/products/matlab/), SPM (http://www.fil.ion.ucl.ac.uk/spm/), etc. In this way, NITRC-CE offers extreme flexibility and the convenience of allowing the users to use whatever software packages they are already familiar with. Another nice feature of NITRC-CE is that it can be set up and run on either private (ie, institutional) or publicly-accessible (ie, commercial) clouds. If the users’ university or research institution has a high-performance computing cluster or is part of a larger research grid that has enough computing power to run the analyses, then they may build their own computational environment on the local cluster or grid. (see http://www.nitrc.org/plugins/mwiki/index.php/nitrc:User_Guide_-_Building_Your_Own_NITRC-CE for instructions). Alternatively, if the user requires more computing power than is available to them, it is also possible to launch and run NITRC-CE through commercial cloud providers. (see http://www.nitrc.org/plugins/mwiki/index.php/nitrc:User_Guide_-_Launching_NITRC-CE_with_AWS_Marketplace for instructions).

When Cloud Computing Might (or Might Not) Make Sense

It should be noted that since standalone desktop workstations provide sufficient computational power for many routine tasks, these will continue to be a mainstay in neuroimaging analyses for the foreseeable future. Nonetheless, a subset of computationally intensive analyses may significantly benefit from (or require) cluster, grid, or cloud-level resources.

However, because of the setup time typically required to: 1) transfer the raw data to the cloud, 2) upload, modify and troubleshoot the analysis software for offline analyses, and 3) download the final results, there are inherent trade-offs between the convenience of running analyses on desktop workstations (ie, where one’s data are likely located, one’s directories are already set up, and one is familiar with the operating system and software) and the potential gains in computer processing speed. Therefore, when confronted with having to choose between mutually exclusive alternatives, given limited resources—in this case, the amount of time, effort and money required to analyze a particular neuroimaging data set—one must weigh the opportunity costs of each possible approach, as well as the trade-offs (eg, between the amount of extra human time and effort required to set up a cloud-based analysis vs the expected gains in computer processing time). Of course, factors such as the amount of time required to set up an offline analysis will critically depend on several factors (not least of which is the level of one’s experience in running such analyses); however, in cases involving simple analyses with minimal computational demand (eg, jobs that are expected to take~45 minutes or less on a standalone workstation), most hypothetical gains in computing time will be negated for even the most proficient cloud-computing users.

However, with the advent of multi-band (aka, simultaneous multi-slice) acquisition techniques such as simultaneous echo-refocused EPI12 or controlled aliasing techniques such us CAIPIRINHA13,14, it is now possible to acquire whole-brain fMRI data with temporal resolutions below 1 s/volume—ie, more than 600 images for every 10 minutes of scanning—and high angular resolution diffusion imaging (HARDI)15 or diffusion spectrum imaging (DSI)16 data sets containing several hundred diffusion-weighted images. As a result, it is now possible, even for single subjects, to acquire extremely large amounts of data that may necessitate correspondingly large increases in image-processing and analysis time; and even for smaller data sets, there are certain computationally-demanding analyses—eg, non-linear normalizations,17 automated brain segmentation,18 lesion segmentations,19 etc.—which might benefit from more computing power than is available on conventional laptop or desktop workstations.

At the most abstract level, there are at least two general categories of neuroimaging analyses for which cloud computing is likely well suited (ie, worth the extra set-up time and effort). These include: 1) any large, computationally-intense job that exceeds the computational limits of conventional workstations (eg, due to enormous physical memory requirements that cause software crashes and/or out-of-memory errors), and/or 2) relatively large numbers of analyses that could be run individually (ie, in serial) on conventional workstations, but would collectively take a very long time (eg, days, weeks, or months).

An example of a large, computationally-demanding job that we routinely perform with cloud computing recourses is non-linear normalization of either T₁- or diffusion-weighted images via the DiffeoMap toolbox in MRIStudio (https://www.mristudio.org/, Johns Hopkins University School of Medicine, Baltimore, MD, USA). In this procedure, individual subject data are first warped to a standard template20–22 using a 12-parameter affine (linear) transformation, followed by high-dimensional, non-linear warping with the large deformation diffeomorphic metric mapping (LDDMM) algorithm.23 Since the second step of this procedure (ie, the portion involving the non-linear LDDMM processing) cannot be run on conventional workstations, the creators of DiffeoMap and MRIStudio have created and continue to support a platform for free remote processing (ie, cloud computing) using a high-performance computing cluster that comprises 256 eight-core processors with 64GB of dedicated memory per node, and over one petabyte of hard disk storage.* In this way, the initial set-up parameters are entered into DiffeoMap on the user’s workstation, and then are automatically uploaded as a job file (along with the subject and template images) to the remote cluster, which performs the LDDMM processing. For single-channel LDDMM processing (ie, normalization with one type of image contrast), the analyses can be performed in as little as 30 minutes (depending on the server demand and backlog) for standard 1mm cubic, whole-brain, T₁-weighted anatomical images; whereas multi-channel LDDMM processing (ie, based on two or three different image contrasts) typically takes on the order of 90 minutes. Once the processing has finished, the remote server automatically emails the user a secure link to retrieve their results. Since the LDDMM analysis is not possible on conventional desktop workstations, this is an obvious example of where high-performance computing is necessary—and in this case, the free DiffeoMap software provides an intuitive and user-friendly interface that seamlessly initiates the data transfer and then automatically runs the analysis in the cloud.

In addition to tasks that cannot be run on conventional workstations (eg, due to RAM limitations, etc.), there are certain types of neuroimaging analyses that require significant amounts of processing time to complete, and running these on a cluster or cloud may therefore be an attractive solution to speed up the analyses. One such example from our own lab that falls into this category is multi-component T₂-relaxation myelin water imaging, which uses a non-negative least squares (NNLS) method to estimate the myelin water fraction, which is given by the ratio of myelin water (ie, water trapped between the myelin layers) to the total water content within each voxel.24,25 This method is relatively robust26 and has been validated with histopathology,27,28 but one drawback is that each whole-brain human analysis takes on the order of 24 hours to complete on contemporary desktop workstations with quad-core CPUs. Therefore, even after all of the preprocessing has been completed, the final step to generate the myelin water fraction maps for a moderately-sized study with 30 subjects would take an entire month, and analyzing larger or longitudinal data sets could require several months of uninterrupted computer time, which is not at all practical. Fortunately, we have been able to work around this limitation by taking advantage of free cloud computing resources because our institutional affiliation allows us to use WestGrid and Compute Canada resources at no cost. The enhanced power of these research grids allows us to analyze all of the data from different subjects and different time-points simultaneously (ie, in parallel) and retrieve our data in a fraction of the time (eg, less than one day instead of a several months), compared to running the same analyses on standalone workstations.

Cost and Other Considerations Regarding Cloud Resources

As illustrated above, there are several potential benefits of using cloud computing resources for neuroimaging data storage and analysis; however, there are also several other considerations (and a few potential barriers) that we feel warrant at least a brief discussion as well. Cost is often a major consideration in research, and will be discussed. However, in addition to economics, there are several other factors that will be touched upon, including: data security, reliability, legal issues, privacy, open standards, compliance, and the long term validity of cloud computing,29,30 as well as data transfer bottlenecks, software licensing, performance unpredictability, data confidentiality, and so on.31

Cost

In the current research funding climate, where grants are increasingly competitive and budgets are tight, one of the major considerations in terms of whether or not to use cloud resources at all (and if so, which resources to use) will be the cost.

Neuroimaging research labs have historically employed a local infrastructure model in which researchers use software and run analyses on their own workstation(s). However, these resources require: 1) large up-front costs (ie, to purchase computers, data servers and software), 2) dedicated work space (ie, for each computer system), and 3) a significant amount of low-level system administration (ie, to set up each system, manage software licenses, configure network permissions and manage users, perform routine backups and anti-virus checks, etc.).32 For these and other reasons—including the ongoing power, cooling and maintenance costs33 associated with running multiple systems—grid or cloud computing may actually offer a less expensive alternative to purchasing and supporting top-of-the-line standalone workstations.34,35

The other benefit of cloud computing is that it is almost infinitely scalable, and one only pays for what s/he uses on a publicly available cloud, and may not have to pay at all on a local cluster or research grid. For these reasons, fewer and less expensive workstations can be purchased (ie, since ordinary laptop or desktop computers may suffice, rather than purchasing high-end multi-core CPU or GPU workstations), and the computing power can then be greatly increased later, depending on each user’s specific needs. In essence, a laboratory can start relatively small and grow as it requires, instead of purchasing equipment based on what is thought will be needed a month, a year, or several years in the future. And because up-front capital costs are minimized (and replaced by operating costs that are lower and scalable), more grant money can be spent on data acquisition, additional graduate students, or hiring other personnel.

One potential pitfall, however, is that while cloud computing reduces the cost of infrastructure, the cost of data communication (ie, the cost of data transfer to and from the cloud) and the ongoing computing costs will likely be increased,36 and these issues may be exacerbated when using a hybrid cloud model in which data are distributed among multiple public/private/community clouds.37 One potential solution that avoids the high cost and bandwidth bottlenecking associated with extremely large data transfers is to physically ship the data on hard disks,38 which is the model currently being employed to distribute more than 20 TB of data resulting from the Human Connectome Project (http://www.humanconnectome.org/data/connectome-in-a-box.html). Another potential solution would be using offline file transfer services such as Globus (https://www.globus.org). Globus is a fast, reliable, and secure file transfer service that maximizes bandwidth usage, manages the security configurations, and provides automatic fault recovery in case of transfer interruption. Compute Canada has a Globus portal (https://www.computecanada.ca/research-portal/national-services/globus-portal/) where Compute Canada users can use their free Globus account to login and then transfer data within Compute Canada resources as well as between local storage and Compute Canada sites. Having said that, with the exception of extremely large data sets (eg, hundreds of GB or larger), bandwidth considerations are not likely to be a major factor for the majority of neuroimaging researchers— especially since most academic or government research institutions have robust, high-speed networks and neither monitor nor bill individual investigators for bandwidth use. Nonetheless, it should be noted that—all else being equal—file transfer (ie, upload and download) times will scale with file size, and this could become a factor in extreme cases.

Another thing to consider, if a free institutional cluster or grid is not available, is that using commercial cloud providers for sustained, long-term use may not be economically advantageous over investing in internal resources. For example, certain laboratories that anticipate running computationally-demanding tasks continuously over a long period of time (eg, protein or genetic modeling institutes) may be better served by investing in their own infrastructure or implementing a mixed-use strategy, in which some of the services and applications run internally and the rest can be hosted in the cloud.39 However, with the exception of the busiest and most prolific neuroimaging centers, this is not likely to be the case. The reality in neuroimaging research is that analyses are most often performed in relatively short and/or relatively infrequent intervals, which are exactly the types of analyses that would most greatly benefit from the scalability offered by grid or cloud computing.

In terms of cost, the lowest cost—and in many cases free—options are either: 1) Brain Imaging Analysis as a Service (BiAaaS) tools like MRICloud for specific applications, or 2) distributed computing on an institutional cluster or grid. If these resources are not available (or are not sufficient for the desired analyses), it may be necessary to use a pay-per-use commercial cloud provider like Google or Amazon (perhaps combined with NITRC-CE, see above). However, even in the latter case, cloud computing may be more economical than purchasing top-of-the-line standalone workstations for neuroimaging applications.

Data security

In addition to cost, another critical consideration before adopting a cloud service is the security of the data. As a baseline, statistics show that lost or stolen devices (ie, laptops, hard drives, and other devices) account for approximately 33% of security breaches, while a further 16% are due to internal theft;40 and although we would all like to think that our data is safe and secure, this is not necessarily the case. According to one popular report, the Medical/Healthcare industry single-handedly accounted for over 42% of all data breaches reported in 2014,41 which was far higher than any other sector (ie, Banking/Financial, Business, Education, or Government/Military). Therefore, because neuroimaging data is a type of personal health information, it should (as a rule) be de-identified or annonymized whenever and to whatever extent possible, even on standalone workstations. For example, according to guidelines set out by our institutional privacy officer and research ethics board (REB), we are not allowed to electronically store patient names or initials, gender information, or dates of birth in connection with our imaging data (including in image header files, etc.).

However, since cloud computing involves the transfer of this data over a network, and storage on external servers, it is especially recommended (and may even be subject to more stringent institutional and/or jurisdictional policies) to de-identify datasets before uploading them to any off-site locations, including cloud services. Most, if not all, cloud providers now use data encryption and network architectures that have been designed to satisfy the requirements of security-sensitive users, but that does not mean that cloud services are free from hacking and/or data theft, as evidenced by several recent high-profile cyber attacks.42 In a network, an organization’s data and software are always vulnerable to data loss, phishing, botnet attacks, etc.37 For example, hackers are using botnet that runs remotely on a collection of machines and provides cheaper options to start an attack on any cloud.43 As a result, cloud data servers are often kept in different regions and data are distributed across different servers to protect individual data from hacking. Additionally, some cloud providers offer data encryption while uploading, and this is likely one of the best solutions to prevent unwanted data security breaches.44 Currently, most researchers also use encryption to ensure that their data are kept private in a cloud.45,46 However, each potential cloud user should check with their local REB (and privacy officer, if applicable) ahead of time to make sure that their proposed cloud computing and/or data storage solutions comply with federal (eg, HIPPA: Health Insurance Portability and Accountability Act), provincial/state (eg, PHIA: Personal Health Information Act) and institutional regulations prior to transferring any data off-site.

Data monitoring and seizure

Because many cloud vendors house and maintain data servers in different countries throughout the world—in part to safeguard the data from third-party cyber attacks (see above)—there is also a possibility that, according to local laws, data stored on these servers can be legally monitored or seized by intelligence, security or other government organizations (eg, the CIA, FBI, etc.).47 For this reason, some commercial cloud providers now provide users an option to choose where (geographically) their data will be stored. For example, Google Cloud Storage provides an option for users to restrict their data to servers in specific geographic locations—eg, Asia, Europe, or North America at a continental level, or even Eastern USA, Central USA, or Western USA at the regional level (c.f., https://cloud.google.com/storage/docs/bucket-locations). Therefore, although data should always be properly de-identified anyway, these storage options are available for researchers who wish to add another level of data security (ie, by restricting where their data will be stored) to safeguard against data monitoring and seizure.

Data availability

Data availability means that the resources of a system are accessible to the user whenever they are required,48 and in the case of cloud computing solutions, security breaches48 or other issues may impact the ability to access a particular server or database, which may in-turn affect data availability.49 Also, while there is some redundancy in most cloud systems, power or network failures can cause network slowdowns and lead to interruptions in uploading, downloading or even data processing.7 However, a more common occurrence with public grids and clouds is that there are numerous service requests to access the same resources, and this not only increases the CPU demand, but also increases waiting times and may (in the worst case) even drop service requests.50 However, to tackle this problem, cloud providers are including technologies such as Adobe AIR, Google Gears and Curl, through which cloud based applications can run locally whenever there are disruptions in a network connection,49 and it is now also possible to provide accessible resources in urgent cases by using multi-cloud (aka, rain cloud) models.51

Compatibility

Compatibility refers to the interoperability of the processed data and software within different platforms. Each cloud has its own data storage model, licensing model, and security model,52 and these may render cloud services from different providers incompatible with one another.53 For example, some Google cloud products and services are incompatible with Microsoft cloud products and services (and vice versa),29 which may preclude information from being exchanged between cloud systems. Furthermore, similar incompatibility issues may also occur between private and public clouds, which may prohibit users from running one portion of their analysis on an institutional cluster and then performing another portion of the analysis on a public cloud. For these reasons, data transform-ability into a suitable form for analysis has been a hindrance in the adaption of big data.54 Two of the possible solutions are to design an interoperable platform, or to develop a standard and open application program interface (API) so that data portability or software compatibility can be ensured.37 To date, some cloud vendors have embraced others’ APIs (eg, Hewlett-Packard Helion Eucalyptus is compatible with Amazon Web Service), and the Open Grid Forum continues to develop an open standard called the Open Cloud Computing Interface (OCCI) (http://occi-wg.org/) that focuses on developing interoperable and portable tools for various management tasks. Finally, some computing grids and cloud services—including Compute Canada and WestGrid—have started to support multiple operating systems (ie, Windows, Linux, etc.) and a variety of commonly used software platforms (eg, Matlab, Python, Java, etc.) to maximize utility, flexibility and compatibility.

Data integrity

Data integrity is making sure that the modification of data is done only by the owner or other authorized parties to prevent data misuse or corruption. There are a growing number of applications that allow users to manage and store their data on cloud servers, but these data may have different formats that eventually make it difficult to integrate multiple sources for analysis.55 Also, these applications must make sure that the data integrity has been maintained,56 and that data have been preserved.57 Nonetheless, due to the remote access to data and a lack of support, verification of data integrity remains a challenge for cloud storage solutions, where it may be difficult to detect portions of missing or corrupted data in large data sets.56

Technical expertise and IT support

Although most of the aforementioned issues are relatively minor and will have little (if any) impact on most users interested in grid/cloud computing for neuroimaging data storage and/or analysis, perhaps the biggest challenge for new users is the lack of technical expertise and proper IT support. For example, first-time cloud users may experience difficulty with: 1) renaming and reformatting their files, 2) uploading their data to the remote server, 3) analyzing/processing their data, and 4) retrieving their results. Unfortunately, these and other technical challenges can lead to wasted time and extra frustration, which are ironically the two things that cloud computing is supposed to save users from. Fortunately, automated processes can improve data preparation and transport,58 and improve compatibility of software and data formats in the cloud can decrease these problems. However, the next section will go into further detail on how to get started with cloud computing and avoid some of the most common pitfalls.

How to Get Started

Due to the growing number of research grids and cloud providers, and the equally wide variety of specific user requirements, it is unfortunately beyond the scope of this or any review to provide a set of instructions that will apply in every case and fulfill everybody’s needs. However, the goal of this section is to briefly provide some general advice to new or novice cloud users and how they might be able to start leveraging the benefits of cloud-based resources.

Although it may seem daunting to first-time users, getting started with cloud computing is relatively straight forward, and if there is already an established desktop workflow for a particular type of neuroimaging analysis, the same tasks can typically be done in the cloud with few (if any) changes. As previously noted, “new tools (some commercial and even public), have made it so that dealing with the cloud and running large-scale processing can be rather easy and efficient”.59

A cloud solution usually offers an interface through which users can manage their accounts, monitor their usage, and set up new compute environment(s). Once the environment is set up (e.g., in either a Linux-based or Windows OS), the user interaction with the remote machine will be similar to regular desktop systems. An important factor to consider is the availability of software to carry out the neuroimaging tasks (ie, in terms of licensing, run-time environment, etc.). Once the proper environment is ready, typical steps involved in doing in-the-cloud computations are:

• Transferring (uploading) de-identified data and any analysis code to the cloud environment

• Performing the required image processing and data analysis (which may require installing and/or configuring the required software, but should then follow the same procedures as running the software on a standalone workstation)

• Completing any desired statistical analysis or other post-processing

• Transferring (downloading) the final results back to a local workstation

• Archiving or deleting the data and analysis code from the cloud

As previously mentioned, transferring data between the cloud and the local facilities could be a bottleneck in the case of large neuroimaging data sets. Therefore, using cloud-based storage (when possible) could enhance efficiency and simplicity, although the cost of cloud storage may factor into whether this is a suitable option.

Most cloud solutions, and all of the major research grids and cloud service providers, supply a detailed guide to get new users up and running. Some cloud solutions come with all of the tools required to carry out certain types of neuroimaging analysis (eg, BiAaaS platforms), while others simply provide the infrastructure (IaaS) or the infrastructure and platform (PaaS), in which case users are responsible for installing their own analysis software. Although Compute Canada has a cloud solution for life science researchers called GenAp (https://www.genap.ca/public/home), most of Compute Canada’s cloud solutions are of the latter type, where users need to install all of the necessary software (including the operating system). Therefore, in the absence of a BiAaaS model to perform a desired function, one possible solution would be to use an IaaS cloud model and then load NITRC-CE (discussed above) to configure the cloud environment and automatically load several of the most common neuroimaging analysis packages. Ultimately, the choice of cloud model will come down to each user, depending on their data set (eg, what types of neuroimaging data they wish to analyze), their specific analysis requirements (eg, how much data they have, what types of analyses they wish to perform, etc.), and a host of other circumstances (eg, their funding situation, whether there are suitable BiAaaS solutions available for the desired types of analyses, what institutional resources are available, etc.).

A special note about clusters

High-performance computing on local clusters, as opposed to user-friendly cloud solutions, may not provide the illusion of a private computing environment and may therefore require more user involvement and technical expertise. In many cases, users of computing clusters are aware of the shared environment and have to use non-interactive (or batch) modes to perform their analyses. A queuing system may still accept users’ job submissions and order them based on predetermined factors (ie, to ensure fairness in the system). However, since most clusters are Unix/Linux based, familiarity of users with a command-based environment is essential (if not required), and installing software on a local cluster is not always as easy as in cloud-based solutions (and it may sometimes be impossible, due to licensing restriction). For these reasons, clouds or grids with user-friendly interfaces and dedicated IT support are likely a better option than local clusters for users with limited technical expertise. However, it should be noted, especially for government institutes and research university clusters/grids, that on-site IT support may be available to assist new users transfer their data and set up their analyses.

Discussion and Conclusions

In recent years, high-performance grid/cloud computing has matured and grown considerably in popularity; as a result, there are now a relatively large number of academic, government and private computer grids/clouds that are available to various end-users. Nevertheless, while these resources have seemingly been embraced by astrophysicists, geneticists and researchers in other big-data fields,60 there are—with the exception of a handful of imaging physicists, computer scientists and biomedical engineers (ie, researchers who are developing new image processing techniques, pipelines and/or software packages)—relatively few radiologists, psychologists, psychiatrists, cognitive neuroscientists and other neuroimaging researchers have put these resources to use. The two possible explanations for this are that researchers in the latter group (ie, who are using imaging to address neuroscience questions) are:

1. generally informed about cloud computing resources and how these could potentially benefit their research, but they have made an informed choice not to use these resources; or

2. unaware of what cloud computing resources are available, unacquainted with how these could potentially benefit their research, or are uninformed (and likely intimidated) about how much it might cost or how hard it might be to make use of these resources.

Therefore, by introducing the main types of computer architectures, highlighting some of the cluster/grid/cloud resources that are available, discussing some of the considerations (including cost and some of perceived barriers) related to cloud computing, identifying when it might (and might not) make sense to use cloud resources, and then outlining how someone could get started, we hope this primer has addressed most of the questions neuroimaging researchers might have had, but were to afraid to ask about various high-performance computing options.

Although conventional desktops and/or laptop computers will continue to serve a major role in neuroimaging data analysis, we feel that a growing number of neuroimaging researchers might significantly benefit by incorporating cluster, grid and/or cloud computing solutions for certain compute-intensive applications. Especially for the many researchers who only run these sorts of computationally-intense analyses periodically, the scalable (or elastic) nature of grids and clouds offers unparalleled processing speed when it is needed, while in many cases, also offering a more cost-effective alternative compared to purchasing and maintaining high-end CPU or GPU workstations that would still pale in comparison. Based on a combination of factors (eg, cost, convenience, time, etc.), there will be some scenarios when it makes sense to run an analysis on the cloud and other scenarios when it does not. However, we advocate that being informed of both options, and therefore having the luxury to choose, is the ideal scenario.