Advancing RF pulse design using an open-competition format: Report from the 2015 ISMRM Challenge

Abstract

Purpose

To advance the best solutions to two important RF pulse design problems with an open head-to-head competition.

Methods

Two sub-challenges were formulated in which contestants competed to design the shortest simultaneous multislice (SMS) refocusing pulses and slice-selective parallel transmission (pTx) excitation pulses, subject to realistic hardware and safety constraints. Short refocusing pulses are needed for spin echo SMS imaging at high multiband factors, and short slice-selective pTx pulses are needed for multislice imaging in ultra-high field MRI. Each sub-challenge comprised two phases, in which the first phase posed problems with a low barrier of entry, and the second phase encouraged solutions that performed well in general. The Challenge ran from October 2015 to May 2016.

Results

The pTx Challenge winners developed a spokes pulse design method that combined variable-rate selective excitation with an efficient method to enforce SAR constraints, which achieved 10.6 times shorter pulse durations than conventional approaches. The SMS Challenge winners developed a time-optimal control multiband pulse design algorithm that achieved 5.1 times shorter pulse durations than conventional approaches.

Conclusion

The Challenge led to rapid step improvements in solutions to significant problems in RF excitation for SMS imaging and ultra-high field MRI.

Introduction

The ISMRM Challenge is an initiative to foster impactful innovation in MRI applications and methodology through a series of year-round head-to-head online competitions, in which participating teams from around the world submit their solutions to topical problems for evaluation by objective quantitative measures and/or subject matter experts (e.g., radiologists). To date, a total of three Challenges have been conducted: the 2012 Challenge on water-fat reconstruction, the 2013 Challenge on sub-Nyquist reconstruction of dynamic imaging, and the 2015 Challenge on RF pulse design. An ISMRM Reconstruction Challenge was also held in 2010, but it was a separate initiative from the ISMRM Challenge series. This article describes the design and results of the 2015 ISMRM Challenge, which aimed to address the most significant contemporary problems in RF pulse design.

The selection of problems for the 2015 Challenge was based on three considerations: 1) they address existing technological issues whose solutions would have a significant clinical impact; 2) they are associated with a near-critical mass of recent scientific developments making them good candidates for major improvements; and 3) they lend themselves to an open-competition format (i.e., they are amenable to generally objective and well-justified criteria to quantify progress). The two problems identified for the challenge are described next.

The first selected problem was the design of patient-tailored slice-selective parallel transmission (pTx) excitation pulses. Short slice-selective pTx pulses would enable multislice imaging at ultra-high field (7 T and above) with spatially-uniform contrast and signal-to-noise ratio, without sacrificing excitation bandwidth and spatial resolution in the slice dimension. Spokes pulses [1, 2] were proposed as a solution to this problem more than a decade ago and remain the standard method for slice-selective pTx excitation with uniform flip angles. However, their relatively long durations and multiple large RF peaks currently necessitates undesirable tradeoffs between RF inhomogeneity mitigation/flip angle uniformity, slice sharpness, spectral bandwidth, and SAR, and even state-of-the-art designs are too long to fit within many pulse sequences [3–5]. Furthermore, optimal pTx pulse design requires the solution of a non-convex magnitude-least squares optimization problem [6, 7], which presents challenges in terms of speed and robustness. The goal of the pTx sub-challenge was to find the shortest possible slice-selective pTx pulses that excite uniform flip angle patterns with sharp slice profiles, while meeting SAR and power constraints.

The second problem selected for the Challenge was the design of short multiband refocusing pulses for spin echo simultaneous multislice (SMS) imaging. SMS imaging with high multiband factors has the potential to significantly accelerate spin echo MRI sequences such as T₂-turbo spin echo (TSE), FLAIR, T₁, and HASTE [8, 9]. The accelerated sequences may be useful for many clinically challenging imaging scenarios, including scanning patients who have difficulty lying still (e.g., pediatric and epilepsy patients), minimizing motion sensitivity in body MRI, and achieving faster scanning for acute neurological emergencies (e.g., stroke, traumatic brain injury, and hemorrhage). SMS acceleration is also important for diffusion-weighted imaging, where it can significantly decrease the scan time required to measure a large number of diffusion directions and/or to acquire high-resolution diffusion-weighted images. The main challenge of spin echo SMS imaging with a large number of bands (i.e., high acceleration/multiband factor) is the high peak RF power/SAR of the associated multiband refocusing pulses, which currently necessitates long pulse durations not compatible with many pulse sequences. This problem is exacerbated in multiple spin-echo sequences with a large number of 180° pulses, and it limits SMS diffusion imaging in terms of repetition times (TRs), multiband factors, slice thickness, and robustness to off-resonance. Consequently, low-power multiband pulse design has become a research topic of great interest, but current solutions such as PINS and MultiPINS pulses [10, 11], inter-band phase optimization [12], time-shifting [13], and root-flipping [14] either require a tradeoff in time-bandwidth product/slice sharpness, or they reduce peak RF power but do not reduce SAR. The goal of the SMS sub-challenge was to find the shortest possible SMS refocusing pulses that meet realistic constraints on refocusing efficiency, RF power/SAR, and gradient performance across a range of multiband factors.

The Challenge ran in an open competition format from October 2015 to May 2016. A total of thirteen teams participated, and the winners of each sub-challenge were chosen based on the shortest pulse durations subject to realistic scanner hardware and SAR constraints and excitation quality criteria. Below, we describe the design of the Challenge, followed by the results, key success factors, lessons learned, and implications.

Methods

Challenge overview

The Challenge was conducted in two phases. The goal of Phase I was to engage the community by posing a small set of problems sufficient for initial development of solution approaches. Phase II posed larger problem sets including additional anatomies and geometries to encourage development of robust, broadly applicable solutions. Submission and scoring of Phase I ran on the challenge website (challenge.ismrm.org) from November 4th, 2015 until March 15th, 2016. All teams who submitted valid solutions in Phase I were admitted to Phase II. Similarly, Phase II ran from April 4th until the beginning of 2016 ISMRM Annual Meeting (May 5th, 2016), where the winners were announced. Interviews with the winners about their experience with the Challenge and their design approaches appear in a Magnetic Resonance in Medicine Highlights article [15]. Outreach was conducted to raise awareness and recruit participants before and during the challenge. This included a rotating ad slide between talks at the 2015 ISMRM Annual Meeting and the 2015 ISMRM Workshop on Simultaneous Multi-Slice Imaging. A virtual journal club meeting was held immediately after the Phase I details were posted to introduce the Challenge and walk contestants through the provided examples.

The RF pulse design challenge was fundamentally different from previous ISMRM challenges that focused on limited data reconstructions, in that there were no underlying ground-truth solutions available to the organizers, and there was a large set of constraints to which solutions must adhere. Thus, establishing a transparent and open competition platform was critical to the challenge’s success. This was achieved using a fully-specified and open scoring process, scoring criteria, field map datasets, and example codes, as well as a common platform (MATLAB; The Mathworks, Natick, MA, USA). Scoring codes and representative contemporary pulse examples that were easy to understand and met all the constraints were posted to the challenge website, along with $B_1^{+}$ maps and virtual observation points (VOPs) [16] for the pTx challenge. Those materials are now available at https://github.com/wgrissom/ISMRM-RF-Pulse-Design-Challenge, and are described further below.

Contestants submitted their pulses through a form on the challenge web site. Submission meta-data (team name, filename, and submission date) were collected in a custom database powered by Drupal (www.drupal.com), and the pulse shapes were uploaded to the ISMRM web server in the form of a MATLAB binary file. The pulses contained in the files were automatically evaluated by a MATLAB-based scoring script running on one of the challenge organizer’s local machines. The results of this evaluation were stored back into the Drupal database alongside the submission metadata for sorting and public display on the challenge’s leaderboard pages.

pTx sub-challenge design

Phase I

In Phase I of the pTx sub-challenge, contestants were asked to design one set of slice-selective excitation pulses to excite the central slice of a multislice GRE acquisition in the torso at 7 Tesla. Simulated $B_1^{+}$ maps and SAR VOPs for an 8-channel fractionated dipole transmit coil [17] were provided for this problem. Figure 1a lists the constraints that were enforced in the pTx challenge, which included hardware constraints (limits on peak RF amplitude, gradient amplitude and slew rate), quality criteria (maximum and root-mean-square (RMS) flip angle errors inside and outside the slice), and safety constraints (local and global SAR). The peak RF amplitude constraint is based on a peak power limit of 900 Watts per channel, for a 50 Ω coil impedance. Excitation error constraints were set on a problem-to-problem basis to be just above the errors produced by the example pulses (up to 20% higher); the figure lists the maximum allowed errors across all pTx sub-challenge problems. The through-slice phase constraint was defined as the maximum phase error of the unwrapped through-slice phase at each in-slice location, after subtracting off the mean. It was also required that the gradient waveforms start and end at zero, to prevent solutions needing additional ramps. Figure 1b lists the torso-specific parameters, including the resolution and field-of-view (FOV) of the $B_1^{+}$ maps, which were finely sampled in the through-slice dimension to enable evaluation of the slice profile, and more coarsely sampled in the in-slice dimension. The sequence parameters (flip angle, slice thickness, and TR) were derived from a T₁-weighted spoiled gradient echo liver imaging sequence [18]. The TR was used to calculate SAR.

Figure 1.

pTx sub-challenge constraints and parameters. (a) Constraints that applied in all scenarios. (b) Torso-specific parameters (both Phases). (c) Brain-specific parameters (Phase II).

Phase II

In Phase II of the pTx sub-challenge, contestants were asked to design slice-selective pulses for an expanded set of scenarios. The torso excitation problem of Phase I was expanded to include a total of three slices, spaced 4 cm apart. Two SMS brain excitations were added for axial and coronal slice orientations, based on an elliptical 16-channel microstrip transceiver coil at 10.5 Tesla [19]. The parameters for those excitations are listed in Figure 1c; sequence parameters were representative for an SMS-accelerated gradient echo EPI sequence for BOLD fMRI [20]. The simulated $B_1^{+}$ maps were resampled separately for each slice orientation, to obtain fine resolution in the slice dimension in each case. The pulse repetition period (the sequence TR divided by the number of slice groups) was used to calculate SAR.

Examples and scoring codes

Contestants were provided with code to generate 5-spoke pulses that met all constraints for each pulse design scenario. The example pulse for Phase II’s 5-slice coronal brain excitation is shown in Figure 2. The code implemented the small-tip-angle magnitude-least squares SMS spokes pulse design method of Ref. [21], without local spokes location or inter-slice phase optimization. It constructed pulse waveforms at full gradient amplitude and slew, and then time-stretched them as needed to meet maximum RF amplitude and SAR constraints. Contestants were also provided with the scoring code that checked if the pulses met all the constraints listed in Figure 1a. The code used the spatial-domain small-tip-angle expressions [22] to compute the pulses’ excitation pattern, with non-uniform fast Fourier transforms to accelerate computation [23]. If a submission met all the constraints, the code returned the contestant’s score as the sum of the durations of the pulses across the design problems, in microseconds.

Figure 2.

pTx pulse design example provided to the participants (coronal SMS brain excitation from Phase II). (a) RF amplitude and gradient waveforms. The pulse duration (24.4 ms) was SAR-limited, with 4 of the 70 VOP’s within 10% of the maximum SAR constraints. (b) Slice and in-plane profiles of the example pulses. The pulses had a maximum in-slice flip angle error of 17.4°, an in-slice RMS error of 2.9°, a maximum through-slice phase deviation of 0.1 radians, a maximum out-of-slice error of 0.55°, and an out-of-slice RMS error of 0.07°.

SMS sub-challenge design

Phase I

In Phase I of the SMS sub-challenge, contestants designed two multiband pulses, one for a twice-refocused or double-pulse field gradient (dPFG) diffusion sequence, and one for a TSE sequence. Figure 3a lists the constraints for the SMS sub-challenge. In the TSE sequence all the slices must be refocused, so a through-slice phase error constraint was enforced for that case, in the same way as in the pTx sub-challenge. This constraint was not enforced for the twice-refocused diffusion sequence since the phase profiles of the two refocusing pulses cancel. Contestants could thus exploit this freedom to reduce diffusion pulse durations [13, 21]. As in the pTx sub-challenge, gradient waveforms were required to start and end at zero. The peak RF amplitude constraint was representative for a 3T body coil. Figures 3b and c list diffusion- and TSE-specific profile and sequence parameters. The diffusion sequence parameters were based on a sequence with a multi-band factor of 5, with slices spaced evenly over a 12 cm axial brain volume. The TSE sequence parameters were based on a sequence with a multiband factor of 12, with slices spaced evenly over a 24 cm coronal brain volume.

Figure 3.

SMS sub-challenge constraints and parameters. (a) Constraints that applied in both sequences. (b) Parameters specific to the diffusion sequence. (c) Parameters specific to the TSE sequence.

Phase II

In Phase II of the SMS sub-challenge, the problems were expanded to include larger ranges of multiband factors and slice thickness combinations (Figure 3b,c). A SAR constraint was further added to make the solutions more realistic. SAR was calculated using the integrated pulse power, the pulse repetition period, and a representative 3 Tesla body coil SAR efficiency of 0.25 W/kg/μT² (SAR due to the sequences’ excitation pulses was assumed negligible). The diffusion pulse repetition period was based on two refocusing pulses and 120 ms per slice group. The TSE pulse repetition period was based on an echo train length of 12 and 220 ms per echo train.

Examples and scoring codes

Contestants were given code to generate conventional multiband pulses that met all the diffusion problem constraints, and PINS pulses that met all the TSE problem constraints. The conventional multiband pulses were constructed by direct summation of modulated single-slice pulses, and were scaled to the maximum RF amplitude constraint. The example conventional multiband pulse for Phase I’s 5-slice diffusion refocusing problem is shown in Figure 4. The PINS pulses were constructed as a train of hard pulses that sampled an SLR-designed envelope [24] with maximum amplitude equal to the RF amplitude constraint, which were interleaved by identical phase encoding blips designed at maximum gradient slew rate. It was not necessary to scale the example pulses to meet the Phase II SAR constraints. Contestants were also provided with the scoring code that checked if the pulses met all the constraints listed in Figure 3a. That code calculated the pulses’ refocusing profiles using a spin domain Bloch simulation [24]. As in the pTx sub-challenge, if a submission met all the constraints, the code returned the contestant’s score as the sum of the durations of the pulses across design problems, in microseconds.

Figure 4.

SMS pulse design example provided to the participants (5 slice diffusion refocusing). (a) RF and gradient waveforms. The pulse duration (24.7 ms) was peak RF-limited. (b) Slice profile of the example pulse. The maximum refocusing efficiency error was 1.5%.

Results

pTx sub-challenge

Phase I

On the first day of Phase I scoring (November 4th, 2015), a solution was submitted with a score of 9,990, improving the example pulse’s score of 16,687 by 40%. After that, the competition was quiet until late February 2016, at which point activity increased; 37 of 39 submissions were made within one month of the deadline. Figure 5a plots the first place score and the cumulative number of submissions over this period. A plateau was reached around a score of 3,000 in early March (an 82% improvement over the example pulse’s score). The first place spot switched hands several times between teams aselsan, MRinnesota, and StanfordUHF. The final Phase I leaderboard is shown in Figure 6a. The Phase I winner was StanfordUHF with a score of 2,763, corresponding to a pulse duration of 2.763 ms.

Figure 5.

Scores and cumulative number of submissions for the most active periods immediately before the deadlines of (a) Phase I and (b) Phase II of the pTx sub-challenge. Five and four teams competed in Phases I and II, respectively. Pins highlight submissions that significantly reduced (improved) the first place score.

Figure 6.

Top three submissions in each phase of the pTx sub-challenge. (a) The Phase I leader-board. (b) The Phase II leaderboard. Team StanfordUHF were the final winners of the sub-challenge.

Phase II

Four teams competed in Phase II, and all online activity took place within 9 days before the deadline. Figure 5b plots the first place score and cumulative number of submissions over this period. In this phase, the scores were the sums of the durations of the three torso pulses and the two brain pulses; the example pulses’ score was 201,931. The first place spot changed hands several times between teams Metric, MRinnesota, and StanfordUHF. The final Phase II leaderboard is shown in Figure 6b. The final winner of the sub-challenge, StanfordUHF, improved the example score by a factor of 10.6.

Winning Approach

Figure 7 shows a representative pulse designed by the winning team Stanford UHF, which was captained by Mihir Pendse, a PhD student at Stanford University (advisor Professor Brian Rutt). It is a 3-spoke pulse designed for the Phase II 5-slice coronal brain excitation with 16 channels. The pulse is almost 5 times shorter than the example pulse for the same problem (Figure 2), and made more efficient use of SAR and excitation error constraints. In an interview with Mihir Pendse [15], he described his approach as an extension of his minimum-SAR spokes pulse design method [25, 26], which optimizes both transmit channel weightings and spokes locations. To further shorten the pulses produced by that method, he applied time-optimal variate-rate selective excitation (VERSE) [27, 28]. He also replaced the linear-phase slice-selective subpulses typically used in spokes pulses with minimum-phase subpulses, but was still able to obtain sufficiently refocused slice profiles. For the brain SMS pTx problems in Phase II, he extended his design method to use multiband slice-selective pulse shapes designed by time-optimal control [29].

Figure 7.

pTx pulse design from the pTx sub-challenge winner StanfordUHF (5 slice coronal brain excitation with 16 channels). (a) 3-Spoke RF and gradient waveforms, which followed a spokes construction with 3 spokes, multiband minimum-phase slice-selective waveforms, VERSE, and inter-band phase optimization. The pulse duration (5.0 ms) was gradient amplitude- and slew-constrained, with 12 of the 70 VOP’s within 10% of the maximum SAR constraints. (b) Slice and in-plane profiles of the winning pulses. The pulses had a maximum in-slice flip angle error of 19.7°, an in-slice RMS error of 3.81°, a maximum through-slice phase deviation of 0.13 radians, a maximum out-of-slice error of 0.36°, and an out-of-slice RMS error of 0.10°.

SMS sub-challenge

Phase I

On December 21st, with a score of 10,517, team gymz became the first to submit a Phase I SMS sub-challenge pulse that beat the example pulse’s score of 37,621, and they held on to first place until February 22nd. Activity increased significantly in late February, with 38 of 55 submissions coming in within one month of the March 15th deadline. Figure 8a plots the first place score and cumulative number of submissions over this period. During that time, the first place spot switched hands several times between teams MRinnesota, gymz, and rfcontrol. Team rfcontrol took the top spot a few days before the deadline, and then firmly cemented it one day before the deadline by improving their score by more than 1,000 points which is equivalent to shortening the total duration of their diffusion and TSE pulses by more than one millisecond. The final Phase I leaderboard is shown in Figure 9a. Team rfcontrol won with a final score of 5,559, which resulted from a diffusion pulse duration of 2.4 ms, and a TSE pulse duration of 3.15 ms.

Figure 8.

Scores and cumulative number of submissions for the most active periods immediately before the deadlines of (a) Phase I and (b) Phase II of the SMS sub-challenge. Ten and seven teams competed in Phases I and II, respectively. Pins highlight submissions that significantly reduced (improved) the first place score.

Figure 9.

Top three submissions in each phase of the SMS sub-challenge. (a) The Phase I leader-board. (b) The Phase II leaderboard. Team rfcontrol were the final winners of the sub-challenge.

Phase II

Seven teams vied for leadership in Phase II, and submissions started the day after the scoring system went live on April 4th. Figure 8b plots the first place score and the cumulative number of Phase II SMS sub-challenge submissions. In this phase, the scores were the sums of the durations of 15 diffusion pulses and 16 TSE pulses, and the example pulses’ score was 520,436. The top spot changed hands several times between teams MRinnesota, gymz, and rfcontrol, but rfcontrol took and maintained it a few days before the May 5th deadline, and on the last day they solidified their position by shaving another 8,726 points from their score, corresponding to a total pulse duration reduction of 8.726 ms. Their longest diffusion pulse duration was 2.94 ms, and their longest TSE pulse duration was 6.19 ms. The final Phase II leaderboard is shown in Figure 9b.

Winning Approach

The team winning the SMS sub-challenge (rfcontrol) comprised Armin Rund (post-doctoral fellow at University of Graz, advised by Professor Karl Kunisch), Christoph Aigner (PhD student at Graz University of Technology, advised by Professor Rudolf Stollberger), and Dr. Christian Clason (Professor at the University of Duisburg-Essen). A representative pulse for 5 slice diffusion refocusing is shown in Figure 10. The pulse is more than 10 times shorter than the example pulse for the same problem (Figure 4), and made more efficient use of peak RF, gradient slew, and refocusing error constraints. In an interview [15], Armin Rund and Christoph Aigner described their approach as a new time-optimal control multiband pulse design method that was built on a previous time-optimal control method they developed outside of MRI [30]. The algorithm jointly optimized the RF and gradient waveforms, and refocusing error, RF and gradient limits were enforced as l_∞-norm constraints. Their code used semi-smooth Newton and quasi-Newton optimization, and a new strategy for globalization based on an auxiliary optimization problem to overcome the non-convexity of the design problems. Interestingly, their pulses were all real-valued, and they reported that complex RF modulation did not significantly reduce the durations of their pulses.

Figure 10.

SMS pulse design from the SMS sub-challenge winner rfcontrol (5 slice diffusion refocusing). (a) RF and gradient waveforms. The pulse duration is peak RF- and gradient slew-limited. (b) Slice profiles produced by the waveforms. The maximum refocusing efficiency error was 2.0%, and the stopband has an equiripple-like error.

Discussion and conclusions

Summary of results and impact

Open competition in the 2015 ISMRM Challenge pushed the envelope in RF pulse design, and culminated in multifold improvements of slice-selective pTx and SMS refocusing pulses compared to contemporary designs (reduction of pulse duration by factors 10.6 and 5.1, respectively), which could signify a notable advance in pulse design methodology. Several insights can be drawn based on the analysis of the challenge results. First, the organizers hypothesized that the pTx sub-challenge could inspire development of a new trajectory to replace spokes. However, all pTx submissions were based on the spokes trajectory. Contestants significantly improved on the initially provided spokes pulse design by finding ways to optimize spokes locations in excitation k-space, applying VERSE, optimizing the subpulse shapes, and making better use of safety margins. Therefore, the challenge results may further cement the spokes trajectory as the first choice for slice-selective pTx. A second insight is that, to date, few algorithms have been developed that incorporate strict SAR constraints into spokes pulse designs [4]. In contrast, these constraints were one of the challenge requirements, which may stimulate translation of the resulting algorithms into practical pTx applications. Third, all the top SMS submissions featured the use of VERSE, which may motivate more widespread adoption of that technique for SMS imaging. A fourth insight is that while most contestants used some form of inter- and intra-slice phase modulation to spread out RF energy in time, the winning team avoided it. This suggests that the winning pulses may not suffer from the drawbacks of existing short-duration phase-modulated multiband refocusing pulses (i.e., TE dispersion and the need for matched 90° excitations), and may represent a breakthrough for this problem (pending peer-reviewed publication of the method).

The challenge elicited a broad engagement from the MR community. In all, 13 out of 34 registered teams participated actively. Submissions were received from a total of ten countries (identified by IP addresses), with the most submissions coming from the US (58), Turkey (23) and Austria (15). Four countries were represented in the final leaderboards: US, Austria, France, and Turkey. Multiple contestants (including the winners) expressed intent to publish the new methods they were inspired to develop.

The transparent nature of the challenge was critical to its accessibility, as reflected in interviews with contestants. Transparency was also important to the quality of the challenge, as in two instances contestants brought critical bugs to the organizers’ attention that were then quickly resolved. Furthermore, the scoring codes, field maps, criteria, example design codes, and documentation developed by the organizing team represent a set of common tools (shared on Github). Their public availability can facilitate future work on slice-selective pTx and SMS refocusing pulse design by providing benchmarking tools to future researchers, as it is possible that shorter (better) solutions could still be developed. To further stimulate these efforts, the challenge website and leaderboard will remain open to submissions.

Limitations and opportunities for improvement

There were several tradeoffs in designing the challenge. First, off-resonance was not included in either sub-challenge. This choice was made based on the assumption that the challenges’ emphasis on reducing pulse durations would automatically promote high-bandwidth, off-resonance-robust pulses (although short pTx and SMS pulse solutions may exist with different properties). It also simplified the problems and reduced the computational burden of on-line scoring. Another tradeoff was made by emphasizing small-tip-angle pTx pulse designs, rather than the less-developed large-tip-angle design problem. Small-tip-angle designs are considerably simpler than large-tip-angle ones due to the linear relationship between the pulses and the excited magnetization. This choice was made based on the projection that offering large-tip-angle problems would limit participation and shift the focus away from the main goal of the pTx sub-challenge, i.e. improvement of slice-selective parallel excitations. In addition, similarly to 2012/2013 ISMRM Challenges, it was originally intended that in Phase II, contestants would be required to submit code that implemented their designs and adhered to provided function prototype definitions for “blinded” evaluation. However, analysis of the results of the previous challenges indicated that such an approach could have prolonged the contest beyond the intended time window due to the significant amount of development and debugging both on the part of the organizers and the contestants. Yet, following this route could have enabled more effective judging of how well algorithms worked in general, because the contestants would have been blinded to the problems their algorithms would be scored on. It would have also allowed the enforcement of practical constraints on computation time, to encourage methods that could execute while a patient lies in the scanner, and would thus be more immediately translatable. Indeed, in their interview, team rfcontrol reported that their Phase I pulse designs resulted from weeks of computation, which would not be compatible with online use. However, they also reported that the expanded problem set of Phase II did inspire them to develop faster and more automated methods that ran in a day. Though it would have been expensive, another possible implementation of Phase II would have used actual scans with the submitted pulses on real hardware. This could have further improved translatability of the developed methods by placing implicit constraints on RF amplitude slew rates [31]. It would also have required specific dwell times and introduced $B_1^{+}$ map measurement errors. Finally, to minimize any perceived bias (e.g., favoring performance of specific platforms), MR vendors were not engaged to formulate the challenge problems, although collaboration with a vendor could potentially connect the value chain end-to-end and accelerate clinical adoption of the developed methods.

In addition, a few opportunities to improve the Challenge were realized in hindsight. Contestants suggested that reducing the SMS flip angles to slightly less than 180 degrees would be more realistic and would have significantly reduced the SAR and peak power of the pulses. It was further suggested to allow any starting and ending gradient amplitudes in the SMS pulses, since the gradients could presumably be bridged with crusher gradients. Finally, in their interview with the organizers, the SMS sub-challenge winners rfcontrol suggested that a scientific or poster session dedicated to the Challenge at the ISMRM meeting would have given contestants a forum to present and share their ideas. Such an event could help stimulate further work on these problems past the end of the Challenge.

Looking ahead: Open competition and collaborative science

US Vice President Joe Biden recently advocated for team science and collaboration to dramatically accelerate the pace of progress in curing disease in his Cancer Moonshot talks [32]. Open competitions have the power to channel collective energy towards solving meaningful problems as we have seen in all three ISMRM challenges, and other competitions such as the Netflix Prize (netflix-prize.com) and Kaggle (kaggle.com). The ISMRM Challenge, along with other recent ISMRM initiatives such as ISMRMRD [33], ISMRM Unbound (http://www.ismrm.org/mriunbound/), Gadgetron [34], and others, is the first step towards encouraging a collaborative research and development ecosystem through sharing and open competition. We call on the MR community to identify the next set of impactful problems that can significantly improve the outcome-based value of MR [35] and that are amenable to an open competition format.