The ISMRM Open Science Initiative for Perfusion Imaging (OSIPI): ASL Pipeline inventory

Abstract

Purpose:

To create an inventory of image processing pipelines of Arterial Spin Labeling (ASL) and list their main features; and to evaluate the capability, flexibility, and ease of use of publicly available pipelines to guide novice ASL users in selecting their optimal pipeline.

Methods:

Developers self-assessed their pipelines using a questionnaire developed by the Task Force 1.1 of the ISMRM Open Science Initiative for Perfusion Imaging (OSIPI). Additionally, each publicly available pipeline was evaluated by two independent testers with basic ASL experience using a scoring system created for this purpose.

Results:

The developers of twenty-one pipelines filled the questionnaire. Most pipelines are free for non-commercial use (n=18) and work with the standard NIfTI data format (n=15). All pipelines can process standard 3D single post-labeling delay (PLD) pseudo-continuous ASL images and mainly differ in their support of advanced sequences and features. The publicly available pipelines (n=9) were included in the independent testing, all of them being free for non-commercial use. The pipelines, in general, provided a trade-off between ease of use and flexibility for configuring advanced processing options.

Conclusion:

While most ASL pipelines can process the common ASL data types, only some, namely, ASLprep, ASLtbx, BASIL/Quantiphyse, ExploreASL, and MRIcloud, are well-documented, publicly available, support multiple ASL types, have a user-friendly interface and can provide a useful starting point for ASL processing. The choice of an optimal pipeline should be driven by specific data to be processed and user experience and can be guided by the information provided in this ASL inventory.

Introduction

Cerebral blood flow (CBF) is a key physiological parameter for assessing cerebrovascular health in physiological and diseased conditions^1–3. Arterial spin labeling (ASL) perfusion MRI provides a contrast-agent-free acquisition method for the voxel-wise quantification of CBF⁴. Its non-invasive nature and ability to quantify absolute CBF make it ideal in all settings that require repeated acquisitions. ASL has been validated by comparison with other methods that use exogenous contrast agents, such as ¹⁵O-H₂O-PET and dynamic susceptibility contrast MRI (DSC)^5,6, and has been applied in neurological, neuropsychological, and neuropsychiatric research^7–9.

Since the inception of ASL, various ASL approaches have been developed and used, which mainly differed in blood labeling^10,11, readout¹², use of background suppression¹³, and use of protocols involving multiple labeling times and post-labeling delays (PLDs). A consensus recommendation on acquisition has been formed to facilitate its use in different settings¹⁵, other ASL protocols are still used based on specific clinical populations, availability and experience in the use of the specific protocol, and compatibility with the MRI scanner. While basic processing is readily available on the MRI scanner console for the current product sequences, offline processing offers more advanced algorithms, is often required for custom sequences¹⁶, and can add features such as outlier rejection¹⁷, quality control¹⁸, or partial volume correction¹⁹. Additionally, offline processing is the only option to derive CBF values in different regions of interest or in the template space as required for statistical analysis²⁰. Proper tools are needed for offline processing, ideally with automatic batch mode, which allows processing the entire dataset with minimal manual intervention and can be easily scalable to larger datasets without additional manual effort.

For potential users it is often not technically or time-wise feasible to implement their own ASL processing software, particularly for such a wide variety of ASL types. A recent European survey noted that general awareness, technical difficulty, and lack of tools are indeed some of the main hurdles to the more widespread use of ASL and quantitative MRI in general²¹. More than twenty different toolboxes have been released for ASL data processing and analysis^{17,20,22–32} from different laboratories that specialize in ASL analysis. The selection of a particular pipeline by a new user can be complicated because of the wide variety of ASL sequences, data formats³³, and processing methods²⁰. Both new and experienced ASL users looking for suitable ASL image processing and quantification software may benefit from a comprehensive and detailed list of ASL image processing software along with their features to guide their search for a suitable pipeline.

The International Society for Magnetic Resonance in Medicine Open Science Initiative for Perfusion Imaging (ISMRM OSIPI, referred to hereafter as ‘OSIPI’) is an initiative and activity of the ISMRM perfusion study group. Established in May 2020, its mission is to create open-access resources for perfusion imaging research to improve its reproducibility, speed up the translation into tools for discovery science, drug development, and clinical practice, and eliminate the practice of duplicate development³⁴. The activities of OSIPI were divided among task forces. Here, we describe the activities and output of Task Force 1.1 (TF1.1), which is aimed at creating an inventory of the available ASL pipelines, targeting mainly novice ASL users. The inventory summarizes ASL-pipeline features that include supported ASL types, type of input and output, and requirements of software and technical expertise as reported by the pipeline developers and independently assessed by task force volunteers.

Methods

The development of the ASL pipeline inventory and independent pipeline testingwas the goal of TF1.1 for the first two years (May 2020-May 2022). Each OSIPI TF has a lead and a co-lead (JP and SD, respectively, for TF1.1), appointed in May 2020 by the OSIPI Strategy board. TF1.1 also consisted of seven other researchers (HF, UA, CB, VK, LH, HM, DT) with either technical, biomedical, or clinical backgrounds appointed by the TF leads.

Pipeline inventory

The first step of inventory creation entailed compiling a list of pipelines and collecting their basic information, features, and requirements through an online questionnaire developed by the task force and filled by the pipeline developers. The questions broadly focused on:

1. information about the developer, method of availability of the software, and license;

2. operating system and other software and hardware requirements;

3. compatibility with the type of input data, MR scanner, and ASL sequence type;

4. details of data processing steps, analysis, and output features;

5. software’s applications in animals or organs other than the brain;

6. availability of batch-mode to automate data processing and flexibility to modify pipeline configuration;

7. the approximate number of ASL studies and ASL scans processed by the pipeline, as self-reported by the developer.

The full questionnaire is provided in the Supplementary Materials.

The questionnaire and a cover letter explaining the purpose of the inventory were distributed in September 2020 through several channels: i) pipeline developers known to the TF members were emailed directly; ii) the ASL research community was contacted using several mailing lists of perfusion and medical imaging networks (OSIPI³⁴, ASL-network (https://asl-network.org), GliMR 2.0 COST Action³⁵ (https://glimr.eu), ISMRM Perfusion study group (https://groups.ismrm.org/perfusion/), Imaging Cerebral Physiology (http://www.icp-network.org), SPM (https://www.fil.ion.ucl.ac.uk/spm/support/), and FSL (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Support)). Questionnaires were returned by November 2020.

Pipeline testing

The OSIPI TF1.1 also aimed to provide a more objective and independent testing of the capability, flexibility, and ease of use of the pipelines, by running them on a variety of test datasets and scoring the pipelines on a range of features. Between September and November 2021, all pipeline developers were contacted to make their pipelines available for testing (unless already publicly available), and to assist the independent testing by answering questions of the testers if required. Moreover, the ISMRM Perfusion study group, the pipeline developers, and task-force members were contacted for volunteers to provide independent testing of the pipelines. All pipelines from the ASL inventory were tested with the following exceptions:

• The developers stated that the pipeline was not ready for independent use without the active support of the developers for one of the following reasons: i) lack of a comprehensive manual, ii) requirement of coding expertise or substantial changes in source code to configure a new project, iii) or being too narrow-focused only to a specific goal or a single sequence;

• The software was not publicly available, or it was only accessible under a commercial license and the developers did not grant us a special license.

A unified scoring system was created by TF1.1, which consisted of four categories with a total of sixteen criteria as listed below. In addition, a detailed manual for grading was provided for each criterion to allow objective and consistent scoring across testers, including:

• Ease of use

  • Free for non-commercial use or available under a commercial license only. Note that free software can still require or include third-party software with a different license. We recommend commercial users to always check the licenses of all the software used.
  • Ease of installation
  • Ease of data preparation
  • Ease of pipeline configuration
  • Availability of a graphical user interface
• Requirements

  • Requirement of third-party software
  • Instruction for third-party software installation
  • Required programming knowledge for a simple and complex setup
  • Version control of the software
• Flexibility of configuration

  • Availability of batch mode for automated processing
  • For GUIs, batch mode can be saved and reused
  • Flexibility in modifying sequence parameters, support of Brain Imaging Data Structure format for ASL (ASL-BIDS)
  • Flexibility in modifying quantification parameters
  • Flexibility in modifying processing steps
• Support

  • Availability of a discussion forum, an email, or a hotline for support
  • Software under active development

A full list of the criteria and guidance that were given to the testers for the scoring is provided in the Supplementary material “Pipeline Questionnaire”.

The independent testing also aimed at assessing the capability of the pipelines to process different types of commonly used ASL data and the flexibility to configure for different types of studies. For this purpose, six ASL datasets – acquired on scanners from the three major vendors (GE, Philips, and Siemens) – with various imaging parameters (pulsed or pseudo-continuous ASL (PASL/PCASL), 2D/3D acquisition, with/-out background suppression, single/multi PLDs, with/-out M0, with/-out high-resolution anatomical reference from T1-weighted scan). The datasets were provided in NIfTI format, with corresponding imaging parameters and file and directory structure following the ASL-BIDS definition³³. PAR/REC, Digital Imaging and Communications in Medicine (DICOM), or ANALYZE formats were provided when the pipeline did not support NIfTI input without penalizing the pipelines. The protocols were detailed in a JavaScript Object Notation (JSON) file according to ASL-BIDS. The details of the datasets are provided in Table 1. Datasets 1 and 2 were GE and Siemens product sequences, which are widely available. Dataset 3 was a repeatability study with 2 sessions and 2 runs per session, testing the ability to deal with longitudinal studies with pre- and post-task scans. Datasets 4 and 5 were advanced multi-PLD acquisitions. Dataset 6 contained a subject scanned at 7T and at 3T, testing the ability to incorporate different parameters in the same dataset. Note that only the difficulty of pipeline setup, data handling, and the presence of main outputs were assessed. The CBF maps were visually checked for the presence of major processing artifacts and having GM and WM CBF values within an expected range. See more information in the Supplementary Material “Scoring System”.

Table 1.

Summary of the details of the six testing datasets used in the pipeline evaluation. Unless stated otherwise, the datasets were single PLD with control and label pairs acquired on a single subject in a single session. Abbreviations: arterial spin labeling (ASL); Background suppression (BSup); echo-planar imaging (EPI); Flow-sensitive Alternating Inversion Recovery (FAIR); GE Healthcare (GE); Gradient and Spin Echo (GRASE); Pulsed ASL (PASL); Pseudo-Continuous ASL (PCASL); Post Labeling Delay (PLD); Quantitative Imaging of Perfusion using a Single Subtraction II with Thin-slice TI₁ Periodic Saturation (Q2TIPS).

Set	Vendor	Type	BSup	M0	T1	Relevant details
1	GE	3D Spiral PCASL	Yes	Yes	Yes	ASL signal saved as DeltaM; DeltaM and M0 saved in the same file
2	Siemens	2D EPI PASL	No	No	Yes	Q2TIPS labeling saturation, M0 calibration using the control image
3	Siemens	3D GRASE PCASL	No	Yes	No	One subject scanned in two sessions with two runs per session
4	GE	3D Spiral PCASL	Yes	Yes	No	Multi-PLD with 7 PLDs and variable labeling duration
5	Philips	2D EPI PCASL	Yes	Yes	No	Time-encoded with Hadamard matrix
6	Siemens	2D	Yes	Yes	Yes	Two subjects acquired with two sequences: ⅰ) EPI FAIR PASL on 7T; ⅱ) multi-band PCASL on 3T

Each pipeline was randomly assigned to two testers (HF, DA, KB, AC, ZL, TL, JH, BP, and MT) who had no conflicts of interest (no studies with the pipeline developers in the last two years) or any previous experience with the pipeline. The testers were junior researchers (predominantly PhD students) with basic ASL experience and diverse backgrounds, including engineering, neuroscience, biomedicine, neurosurgery, and radiology. Two testers independently scored each pipeline, and the testing coordinator (JP) checked the score agreement between the testers and, in case of significant discrepancy, resolved them by reaching a consensus with the two testers. Testers contacted the developers via official channels when they faced any issue and were unable to solve with instructions in the manual; note that a need for such intervention was part of the scoring system and led to lower scores due to incomplete manuals.

Results

Pipeline inventory

We received responses from twenty-one pipeline developers (Tables 2–5). The results are also published on the OSIPI website (osipi.ismrm.org) and will be updated when new entries are obtained [ref will be provided after the website update].

Table 2.

Author contact information and affiliation, a link for publication and download, availability: On request , commercially available only , a public download is available . ASAP: Automatic Software for ASL Processing; ASLPrep: ASL preprocessing; BASIL: Bayesian Inference for ASL MRI; ENABLE: Enhancement of automated blood flow estimate; LOFT: Laboratory of Functional MRI Technology; Quantiphyse: Visualization and analysis tool for 3D and 4D quantitative and physiological imaging data; VANDPIRE: A Novel Data-processing Pipeline for Imaging Research; MC/UMC: University Medical Center; CSIRO: The Commonwealth Scientific and Industrial Research Organisation.

Table 5.

Experience. This table approximates the actual use of the pipeline in practice. It lists how many different studies (column 2) and datasets (column 3) were processed by the pipeline; how many human brain and non-brain, and animal scans were processed by the pipeline. For studies, the developers were able to choose from options 0, 1–5, 5–10, 10–20, 20–50, 50–100, and 100+. For scans, the options were 0, 1–10, 10–100, 100–1000, 1000–10000, 10000+. Note that the numbers were self-assessed by the developers and are rough estimates only.

Pipeline Name	Studies	Human brain scans	Human non-brain scans	Animal scans
ASAP	10–20	1,000–10,000	0	0
ASL toolbox	100+	10,000+	0	0
ASL-MRICloud	N/A	10,000+	0	0
aslm	5–10	N/A	0	0
ASLPrep	100+	10,000+	0	0
BASIL	50–100	1,000–10,000	10–100	10–100
CereFlow	20–50	10,000+	0	0
Clinical ASL-CVR	5–10	100–1,000	10–100	0
ENABLE	10–20	1,000–10,000	0	0
ExploreASL	50–100	10,000+	0	0
Functional ASL (FASL)	5–10	10–100	0	0
Iris pipeline	5–10	1,000–10,000	0	0
LOFT-CBF	10–20	1,000–10,000	0	0
milxASL	5–10	1,000–10,000	0	0
MJD-ASL	20–50	100–1,000	10–100	0
nordicICE (nICE)	1–5	10–100	0	0
Quantiphyse (ASL)	1–5	10–100	0	1–10
SCRUB-ASL	20–50	10,000+	0	0
Super-selective pCASL CBF	1–5	10–100	0	0
VANDPIRE	20–50	100–1,000	10–100	0
Andrea Federspiel	10–20	100–1,000	0	0

Most pipelines are free for non-commercial use (n=18) and easily available (n=11; Table 2). While the majority of the pipelines were initially developed as research tools for in-house use, over half of the developers provide a manual or a tutorial (n=13), publicly available source codes (n=14), and a graphical user interface (n=12), making independent external use possible.

Most of the pipelines explicitly declare support for data from the three major vendors, i.e., GE (n=15), Philips (n=15), Siemens (n=16), and all three (n=12; Table 3). The majority of the pipelines run on MacOS (n=13), Windows (n=14), and Linux (n=14), while one pipeline is an OS-independent cloud-based tool. Some pipelines use software such as Matlab (n=13) and FSL (n=10); the former requires a fee for both commercial and academic use, while the latter requires a fee only for commercial uses. Other pipelines use free software such as Python (n=3) or SPM (n=8) or parts of it, although non-compiled SPM code also requires Matlab. Most pipelines work with the neuroimaging data formats NIfTI (n=15) and ANALYZE (n=9), respectively. Less than half of the pipelines support imports from the clinically used DICOM format (n=9). Only three pipelines support ASL-BIDS³³, the recently developed ASL extension of the standard for storing image data and acquisition metadata. All pipelines were developed for human brain data, and only a few were reported to be suited also for body imaging in humans (n=3) and preclinical brain imaging (n=3).

Table 3.

Requirements as reported by developers. Basic requirements on the operating system (MacOS , Linux , Windows , cloud ), third-party software (Matlab , FSL , SPM , Python ), and data input types, MR vendors (Agilent , GE Healthcare , Philips Healthcare , Siemens Healthineers , United Imaging - other vendors might work but were not tested). All pipelines were developed and tested to work with human brain scans, further tested functionality is provided for different organs or for animal use in the last column. Abbreviations: Brain-Imaging Data Structure (BIDS), Digital Imaging and Communications in Medicine (DICOM), enhanced DICOM (ENH DCM), Neuroimaging Informatics Technology Initiative (NIfTI).

All the pipelines can process single-PLD PCASL data¹⁵ (Table 4), which is the current clinical ASL standard. Some pipelines can also process the pulsed (n=13) or multi-PLD (n=9) ASL data. Only a small number of pipelines support more advanced sequences, such as Look-Locker (n=4), time-encoded (n=3), or velocity-selective ASL (n=3). All pipelines provide the basic processing steps like motion correction and co-registration with structural images. Most pipelines are modular, and either offer a choice of different algorithms or the possibility to add new algorithms. However, more advanced steps, like partial volume correction (available in n=12) or quality control measures (available n=14), are not universally available. In general, the pipelines provide a sufficiently wide range of output options, with nearly all providing the individual ASL space output (n=20) and most providing an output in the MNI and/or T1 space (n=13 for both).

Table 4.

Features as reported by developers. Main features concerning the labeling type (pseudo-continuous ASL (PCASL), pulsed ASL (PASL), single post-labeling delay (s-PLD), multi-PLD (m-PLD), Look-Locker (L-L), time or Hadamard encoded (T-enc); vessel/super selective (VS)), use of partial volume correction (PVC); output space of the resulting maps (ASL space , high-resolution T1-weighted space , MNI space ); the output of regional values (whole brain , GM , manually provided region of interest (ROI) , standard atlases ); and a presence of a quality control (QC) report or images are provided to facilitate subject exclusion.

Finally, Table 5 lists the number of studies and scans that have been processed by each pipeline as reported by the pipeline developers. Compared with the human brain scans, the number of other organ scans and pre-clinical scans is much smaller, being below 100 for all pipelines.

Pipeline testing

Out of twenty-one pipelines from the ASL inventory, nine pipelines were included in the testing. Nine testers participated. Two had no conflicts of interest or prior experience using any tested pipelines, six testers had a single conflict (previous collaboration with specific developers or experience with a specific pipeline), and one tester had two conflicts.

Based on the inventory-information provided by the developers, a majority of tested pipelines provide GUI for data processing (n=6), a batch mode for automated processing (n=7), installation instructions for the required software (n=8; Table 6), and manuals (n=8; Table 2). Furthermore, all pipelines provide user support. Only a single pipeline (MRIcloud) has a GUI for batch processing; the GUI extension of ExploreASL was not included in this independent testing as it was not available at the time of testing. Most pipelines (n=6) support all three OSes and one is available as a cloud application. As claimed by the developers (Table 3), seven tested pipelines supported NIfTI, three DICOM, three ASL-BIDS, and five multi-PLD data.

Table 6.

Results summary of non-scalar questions in the pipeline assessment. Pictograms: optional (*); instruction of required software installation ; active discussion forum ; email contact ; active software development ; notification of currently used version ; all versions available to use ; A: GUI for a batch data import from DICOM to NIfTI or BIDS; B: GUI for all processing steps on individual subjects; C: GUI for batch processing of multiple subjects; D: GUI for population analysis; E: GUI for results visualization; Batch mode is available for automated processing (Yes/No); For GUIs only, the batch-mode configuration can be saved and reused for future study (Save).

Figure 1 shows the general scores. All pipelines can be downloaded directly online (n=5) or after a simple registration (n=4) (Figure 1A). In most cases, installation (n=9), data preparation (n=5), and setup (n=8) are described in the manual and are easy to follow (Figure 1B). ASAP, MRICloud, BASIL, Quantiphyse, and VANDPIRE have a GUI. ASLprep and ExploreASL use command-line interfaces, but do not require advanced programming skills. ASLtbx and ASLM are recommended only for people with programming skills. Only MRICloud and VANDPIRE have GUIs for batch mode. ASAP, BASIL, and QUANTIPHYSE are more difficult to use in a batch mode than in a single-subject mode, see Figure 1C. Only ASLprep and ExploreASL support ASL-BIDS. ASLtbx can still accept sequence parameters on subject level, but all other pipelines do not allow individual sequence-parameter configuration, see Figure 1D.

Figure 1.

Summary of ease of use of pipelines. A) Download can be done using a direct public link, after registration, or the pipeline has a commercial license. B) Programming skills are needed to set up a simple study with a single scan or multiple scans of the same type or a complex setup with more than one scan possibly having various sequence types. This requires either no programming knowledge as a GUI is available, minimal knowledge as it has simple text-based inputs, medium knowledge of basic scripting, or advanced programming knowledge. C) Installation, preparation, and setup are either trivial to follow or clearly described in the manual (easy); the description lacks details but is manageable without additional help (medium); or difficult. D) The sequence, quantification, and processing parameters in a multi-sequence study can be configured in ASL-BIDS format on the subject- or study-level; it has to be changed in the source code; or it cannot be changed at all.

Figure 2 shows the scores of processing the test datasets and Figures 3 and 4 show examples of processed datasets 3 and 6. Three pipelines (ASAP, ASLM, and MRIcloud) required data format conversion from NIfTI to ANALYZE, though that was not penalized in the scoring system. Prior manual splitting of the deltaM and M0 volumes (dataset 1) or the multi-PLD volumes (dataset 4) was necessary for ASLtbx, BASIL, and Quantiphyse, leading to rating “consulting needed.” Dataset 5 proved difficult to process as only ASLtbx and ExploreASL have built-in decoding of time-encoded data, although BASIL and Quantiphyse can process the decoded data. VANDPIRE could only process dataset 3 as it only works with single-PLD PCASL saved as control-label pairs, though that represents the majority of current clinical ASL data. MRICloud could not process dataset 4 with varying labeling durations as it can process multi-PLD data only with fixed labeling duration. ASAP and ASLPrep only process ASL images accompanied by T1w images and hence could not process datasets 3 and 4. ASAP and ASLM were given lower scores mainly because some basic configurations were difficult to set up and important information was missing in the manual, respectively. Although ExploreASL accepted ASL-BIDS, extra processing parameters had to be set for datasets 3–5. Only a few pipelines were able to run the more complex dataset 3 (n=5) and dataset 6 (n=3) in a batch mode. ASLTbx was not given the highest scores for datasets 2–6 because the configuration file was difficult to set up and some deep-level functions needed to be adjusted to process specific datasets, however, it was able to process all datasets.

Figure 2.

Summary of results of 6 test datasets with 5 score levels: i) ASL-BIDS: Pipeline worked directly with the ASL-BIDS data, or parameters from JSONs were manually filled in the GUI; ii) manual setup: Additionally, the data format or internal parameters had to be adapted according to the description in the manual; ⅲ) Additionally, we had to consult the developers or source-code or experiment with different inputs; ⅳ) Additionally, the pipeline ran with major quality issues with the output; ⅴ) no result: The pipeline did not run, or it crashed, or it was not possible to configure such a dataset. Batch mode was tested in datasets 3 and 6, and a symbol indicates that the batch mode is available.

Figure 3.

A representative axial slice for Dataset 3 for pipelines that processed the dataset without major quality issues. The same scale is used for all images. The main differences are in the amount of regularization, noise, CBF scaling, and processing of the M0 scan. GM masking is applied by default in VANDPIRE.

Figure 4.

A representative axial slice for Dataset 6 for pipelines that processed the dataset without major quality issues. The same scale is used for all images. The main differences are in the amount of regularization, noise, CBF scaling, and use of M0. Unlike all other datasets, ASLprep result is shown aligned to the MNI space.

Discussion

In this study, we created a comprehensive inventory of available ASL pipelines, summarizing their requirements and features. A subset of these pipelines was evaluated by independent reviewers to assess their ease of use, user requirements and technical limitations. While the core functionalities of the pipelines are similar, our findings showed that each pipeline has unique properties that can be useful to different users. Therefore, the inventory and testing results can help users to select a pipeline for their specific needs.

ASL pipeline inventory

The ASL pipeline inventory was built on the self-assessment performed by the developers. As it is difficult to fairly assess the flexibility and ease of use of the pipelines by the developers themselves, we have evaluated these by independent reviewers. In addition, the majority of the pipelines applied all the relevant steps, such as motion correction, co-registration, and spatial normalization. Therefore, we decided not to report the inclusion of processing steps in the inventory. Lastly, the number of scans and studies reported as processed by each pipeline was also self-assessed without strict criteria and should be interpreted with caution. We included it as an approximate assessment of the pipeline maturity, such as the level of effort spent in debugging, validation, and ability to work with different datasets. However, it is difficult to objectively assess how many studies have been processed with a specific pipeline to date, as pipelines are not always mentioned in the methodological sections of published studies and some studies might be mentioned in multiple publications. Lastly, some pipelines do not even have a publication to reference.

For the inventory, to encourage collaborations, we listed all pipelines whose authors had filled in the questionnaire, irrespective of pipeline accessibility. Even in-house pipelines might provide unique specialized features and can be very valuable for certain research lines with data processed away from the original site. Lastly, scanner vendors offer the possibility of quantifying ASL scans directly at the scanner console. While this can be more useful for clinical use, this often does not include advanced processing and evaluation on the group level. Therefore, we have decided not to include MRI vendor pipelines in this inventory.

There are many interactions and dependencies between the twenty-one listed pipelines due to the thriving collaborations between ASL researchers or the development of research and commercial projects within the same institute. The first publicly available pipeline – ASLtbx – has been widely available for almost two decades, and many other pipelines draw inspiration from it, e.g., SCRUB-ASL uses its motion correction module. Parts of code from the LOFT-CBF pipeline were reused by Andrea Federspiel’s pipeline and later ASLM, and were the basis for the commercial spin-off CereFlow. The pipelines MJD-ASL and VANDPIRE were developed in the same institute and share parts of the code. Further pipelines – ASL-MRICloud and nordicICE (nICE) – were developed as an ASL module within a larger software package that is capable of processing structural MRI and other quantitative MRI modalities. Finally, many pipelines, including QUANTIPHYSE, ASL-CVR, ASLPrep, ENABLE, and ExploreASL, use BASIL to perform the non-linear fitting for CBF quantification,. Despite all these interactions, all pipelines have highly variable input data formats, interfaces, outputs, and settings, making them distinct.

Pipeline testing

The majority of testers were junior researchers with a technical or biomedical background and experience in ASL data processing. The testers were asked to assign scores from the perspective of inexperienced and non-technical users using the scoring protocol, which is expected to provide objective rating. We believe that this policy provided a fairer assessment than if tested directly by inexperienced users, as the latter group would not have been skilled enough to assess a pipeline’s positives and negatives critically. Moreover, it would have required a larger group of inexperienced testers for an accurate assessment.

In the testing, most low scores for test dataset evaluation were related to the difficulty of data preparation due to incomplete manuals. Unlike new ASL users, an experienced user is usually able to prepare the inputs correctly by deriving the correct configuration from the source code, or by splitting and rearranging the NIfTI files. In such cases, we scored the pipelines as “able to process” specific datasets but with “low user-friendliness”. Notably, the ASLtbx pipeline was perfectly able to process all datasets, and lower scores for dataset evaluation are only indicating that technical experience is required. ASLtbx is the ASL pipeline with the most years of experience, has high flexibility, has gone through extensive testing by users, has a very active discussion forum, and several other pipelines were partly derived from ASLtbx. Most pipelines provided CBF maps without any significant artifacts, and the CBF values were within a reasonable range for the datasets that we considered though they varied across pipelines. More challenging and poor-quality datasets can potentially differentiate the pipelines, but such comparison will require extensive datasets of variable quality and is beyond the scope of this manuscript.

ASL-BIDS, introduced in 2021, is the ASL extension of the standard for storing image data and acquiring metadata and should provide standardized tags for important ASL quantification parameters such as Labeling Duration or PLD³³. Currently, only ExploreASL and ASLprep fully support the use of ASL-BIDS, while several developers are working on implementing it in their pipelines. The commonly used dcm2nii (https://github.com/rordenlab/dcm2niix) is now in the process of fully implementing ASL-BIDS. Support of multiple formats — such as ASL-BIDS, ANALYZE, or NIfTI — is also important because they do not contain protected health information, and thus are compatible with Health Insurance Portability and Accountability Act regulations.

While most pipelines had a batch-processing mode, only a few pipelines were able to batch-process datasets that included various sequences. Although this feature is not required for small studies, it will become increasingly important in the future, given the trend of working with large multi-site datasets.

Discrepancy between ASL pipeline inventory and testing

The testers did not manage to run some pipelines on certain datasets, even though the developers reported support for these specific input formats and sequences. This resulted in some contradictions between the inventory and the pipeline testing. These were due to issues with pipeline configuration or a too-broad definition of the sequence type, rather than incorrect information in the inventory. For example, BASIL and Quantiphyse can process time-encoded data only if the raw data are decoded. Also, MRICloud works for all multi-PLD subtypes except for the sequence with variable labeling durations.

Limitations

Both the inventory and pipeline testing have several limitations. First, our inventory is limited by the willingness of the pipeline developers to list their pipelines and the pipeline availability for testing. We anticipate that missing pipelines will be included in our continuously updated online inventory. Also, the inventory contains pipelines for human brain data only, as nobody registered ASL pipelines that would be primarily non-brain or non-human; probably because they are still in an early developmental stage and not widely available. Secondly, while we tried to design the inventory and testing to answer the important questions from the anticipated user’s perspective, it is inevitably incomplete. Also, this inventory is aimed uniquely at pipeline selection and does not provide details or guidance for performing ASL processing. However, literature is available for ASL beginners for designing studies³⁶, acquisition protocol¹⁵, quantification³⁷, processing¹⁸, and clinical use³⁸. Most of the pipelines, however, handle raw data and provide CBF in different regions of interest, which can be used for statistical analysis. Third, there were only two testers per pipeline. Therefore, there is a risk of bias in the evaluation because of a possible difference in ASL processing expertise. However, we anticipate the bias to be minimum as the testers because of the objective nature of the questions. Fourth, we could not perform an extensive comparison between the pipelines in terms of the quality and accuracy of the CBF maps as the implementation of the processing steps and their configurations often differ. A fair comparison of pipeline performance would need a detailed comparison with different real and synthetic datasets and is out of the scope of this manuscript. A comparison of pipeline performances has been carried out by ISMRM OSIPI TF 6.1 (ASL Challenges). Lastly, advanced sequences, e.g., velocity- and vessel-selective ASL, were not included in the testing despite the fact that they are increasingly becoming available.

Conclusion

We created an inventory of ASL pipelines, which we anticipate to serve as a guide to new and experienced ASL users looking for a pipeline that best meets their needs. This should accelerate the adoption of ASL outside of the specialized centers. The inventory also identifies the gaps in the availability of specific processing options, and the test results point out the weak spots in the ease of use of pipelines, both of which provide useful feedback to the developers. This will help to steer future ASL pipeline development, further benefiting the users.

While it is difficult to recommend a single processing pipeline for all types of ASL data or users, we summarize the features of six pipelines that have an easier installation, more features, or support more ASL types. As a general recommendation, ASLPrep integrates complementary techniques from multiple software packages and uses container technologies that ensure easy installation and reproducibility. However, there is limited flexibility in reconfiguring the pipeline. ASLtbx is a widely available and flexible pipeline with a strong community, although coding expertise might be needed to configure the pipeline for individual studies. MRIcloud can process data in the cloud, thus lowering the hardware and user requirement, although it offers lower flexibility in configuration and ASL sequences supported. Both BASIL and QUANTIPHYSE are easy to use and able to process most of the commonly used ASL variants. While they offer the most robust quantification of the multi-PLD data, automated processing of larger or heterogeneous datasets might be more cumbersome. ExploreASL provides a reasonably easy-to-use option to process any data with a focus on automated batch processing of heterogeneous studies. ASLprep and ExploreASL support the new ASL-BIDS format for easier data sharing.

Data Availability Statement

For access to testing datasets contact the corresponding author. The complete questionnaire and scoring guidelines are available in the supplementary materials. The publicly available code of each pipeline was used for the pipeline testing and links to repositories and the exact version of the software is provided in Table 6.