BrainPainter: A software for the visualisation of brain structures, biomarkers and associated pathological processes

Abstract

We present BrainPainter, a software that automatically generates images of highlighted brain structures given a list of numbers corresponding to the output colours of each region. Compared to existing visualisation software (i.e. Freesurfer, SPM, 3D Slicer), BrainPainter has three key advantages: (1) it does not require the input data to be in a specialised format, allowing BrainPainter to be used in combination with any neuroimaging analysis tools, (2) it can visualise both cortical and subcortical structures and (3) it can be used to generate movies showing dynamic processes, e.g. propagation of pathology on the brain. We highlight three use cases where BrainPainter was used in existing neuroimaging studies: (1) visualisation of the degree of atrophy through interpolation along a user-defined gradient of colours, (2) visualisation of the progression of pathology in Alzheimer’s disease as well as (3) visualisation of pathology in subcortical regions in Huntington’s disease. Moreover, through the design of BrainPainter we demonstrate the possibility of using a powerful 3D computer graphics engine such as Blender to generate brain visualisations for the neuroscience community. Blender’s capabilities, e.g. particle simulations, motion graphics, UV unwrapping, raster graphics editing, raytracing and illumination effects, open a wealth of possibilities for brain visualisation not available in current neuroimaging software. BrainPainter is customisable, easy to use, and can run straight from the web browser: https://brainpainter.csail.mit.edu, as well as from source-code packaged in a docker container: https://github.com/mrazvan22/brain-coloring. It can be used to visualise biomarker data from any brain imaging modality, or simply to highlight a particular brain structure for e.g. anatomy courses.

1. Introduction

Efficient visualisation of brain structure, function and pathology is crucial for understanding the mechanisms underlying neurodegenerative diseases and eases the interpretation of results in neuroimaging studies. This is especially important in populations studies, where two or more populations are compared for group differences in biomarkers derived from e.g. Magnetic Resonance Imaging, Positron Emission Tomography (PET) or Computer Tomography (CT). The results are best visualised as brain images, where regions-of-interest (ROIs) are highlighted based on the magnitude of the difference between the two groups. These visualisation are generally done by the same software that performs the registration, segmentation and statistical analysis. However, for traumatic brain injury or less common neurodegenerative diseases such as Parkinson’s disease and Multiple Sclerosis, visualisations of statistical results is sometimes not performed due to the inability to register images to a common template or lack of robust registration software. Therefore, many studies such as (Coughlin et al. (2015); Schoonheim et al. (2012)) only report differences between patients and controls in tables or as box plots. There is therefore a lack of visualisation tools that can highlight neuroimaging findings for these complex diseases.

When registration to a common population template is possible, e.g. in Alzheimer’s disease (AD), excellent 3D visualisation software exists which allows interactive visualisation of population differences – e.g. 3D Slicer (Pieper et al. (2004)), Freesurfer (Fischl (2012)) or SPM (Penny et al. (2011)). However, they have several inherent limitations. First, such software – e.g. Freesurfer – generally require inputs in their proprietary data format, which is usually difficult and time-consuming to create without using their pipeline. While creating these proprietary data formats is necessary when users need to display voxelwise visualisations, often users only need to highlight entire ROIs – in this simpler case the user could only provide a list of RGB colors for each ROI in a csv file, removing the need to create input data in a specialised format. Another limitation of existing visualisation software is their difficulty in highlighting complex patterns of pathology in a single slice of a 3D volumetric image. To overcome this, some authors show multiple slices (sometimes up to 8 slices, e.g. Migliaccio et al. (2015)), although this takes too much space on the academic paper being published. While Freesurfer solves this problem using a cortical surface-based representation that captures most of the complexity of pathology patterns in a single image, this surface representation is not supported for subcortical structures such as the hippocampus. Third, current visualisation software cannot be easily used to generate e.g. a movie showing a dynamic process, e.g. propagation of pathology within the human brain, as most of them have been intended for interactive visualisation and have no application programming interface (API) that allows automatic generation of hundreds of images using pre-defined settings.

We present BrainPainter, a software for easy visualisation of structures, pathology and biomarkers in the brain. As opposed to previous visualisation software, the input data is very simple: a generic .csv file containing numbers for each ROI, each number mapping to a different colour to be assigned to that ROI – such a simple input allows BrainPainter to be used in conjunction with any other neuroimaging analysis software. Secondly, BrainPainter can visualise both cortical and subcortical structures using a surface representation, removing the need to show multiple slices of the same 3D scan. Third, the images are generated automatically from pre-defined view-points, and can be easily used to create a movie showing e.g. the propagation of pathology, without the need to write any extra software code or interface with an API. Written in Python, BrainPainter is open-source and can be extended in many ways. In this paper we describe the capabilities of the software, and showcase three use cases where it was used in neuroimaging studies.

2. Design

BrainPainter has a very simple yet effective workflow. Given an input csv file with biomarkers for each region, it produces high-quality visualisations of cortical and subcortical structures. For this, it uses Blender as a rendering engine, and loads 3D meshes from a template brain (one 3D mesh for each ROI), which are then coloured according to the input numbers. Instead of providing a list of RGB colours for each ROI, we choose a simpler interface of providing one number for each ROI which maps to an RGB color using a user-defined color gradient. For example, the gradient can range from white → yellow → orange → red, as in the example from Fig. 1. In this case, the input numbers for each ROI need to be in the range [0,3], where a value of 1.3 would interpolate between colour 1 (yellow) and color 2 (orange).

Figure 1:

Given a .csv file with region-of-interest (ROI) biomarkers and a user-defined color gradient, BrainPainter can automatically generate brain images with the cortical surface (left and middle) as well as with subcortical structures (right). The input .csv file can have multiple rows, one for each set of output images. The color gradient is a list of RGB colours given by the user. Final colours are interpolated using the numbers from the input .csv file based on the color gradient – e.g. if the hippocampus has a value of 1.2, its final color will be an interpolation of colors 1 (yellow) and 2 (orange) from the gradient.

BrianPainter uses open-source software Blender as the rendering engine. We chose Blender for three reasons. First, it is open-source, allowing us to distribute it already integrated with BrainPainter, thus requiring no further installation. Secondly, Blender is a powerful 3D graphics software, which allowed us to create realistic lightning conditions and handle transparency required for the glass-brain. Third, it also supports creating movies of complex temporal processes such as pathology spread along the brain. The software also supports a variety of object formats for the brain template, including the popular .obj mesh format. As BrainPainter is written in Python, it allows interfacing with any Blender function.

The software is able to colour and visualise regions belonging to a pre-defined atlas. Currently, we support three widely-used atlases: (i) the Desikan-Killiany (DK) atlas (Desikan et al. (2006)), (ii) the Destrieux atlas (Destrieux et al. (2010)) and (iii) the Tourville atlas (Klein and Tourville (2012)). However, a custom atlas can also be used by mapping those regions to any of the three atlases currently supported, through the modification a simple mapping in the main configuration file. ¹

3. Customisation

BrainPainter can be easily customised in several ways, as shown in Fig 2. First of all, the colours assigned to each region can be changed by modifying both the control points of the color gradient and the input numbers selecting colors along the gradient. The background colour and image resolution can also be changed.

Figure 2:

BrainPainter website interface at http://brainpainter.csail.mit.edu, showing how it can be easily customised. Here, the user selects the brain type, colours and resolution, and finally uploads the input .csv file with ROI biomarkers. The server then generates the output images, which can be downloaded by the user. More customisation features will be added in future versions.

The 3D structures being visualised can also be customised. We currently support three atlases (Desikan-Killiany, Destrieux and Tourville) as well as two types of brain surfaces: inflated, which is a brain surface that is smoothed out and where no gyri appear, and also pial, the standard brain surface with gyri. The software allows one to remove some 3D structures – for example, Fig. 5 shows the subcortical structures with the cerebellum removed from the visualisation – contrast this with Fig. 4.

Figure 5:

Progression of pathology in subcortical regions within a glass brain, using images generated with our method. Images used by Wijeratne et al. (2018).

Figure 4:

Demonstration of our software for showing the temporal progression of pathology, as a sequence of snapshot at different stages. Images used by Young et al. (2018)

BrainPainter also support two types of surfaces, cortical and subcortical structures. For the cortical surface, we only show the left hemisphere (although the right hemisphere can also be added), and provide two default viewing angles (front and back). For the subcortical structures, we show them for both hemispheres and also show the right hemisphere as a glass brain, for reference.

More complex settings such as the viewing angle and luminosity can also be customised, but currently require minor modifications to the source code. In the future, we plan to enable these customisations from the main configuration file.

4. First use case: Visualising the degree of pathology

In the first use case, we want to visualise the degree of pathology in Alzheimer’s disease. During the progression of Alzheimer’s disease, some regions of the brain such as the hippocampus and temporal lobes will be more affected compared to other regions of the brain such as the occipital lobe. Visualisation of pathology in AD is important in order to understand its underlying mechanisms and generate new hypotheses.

The notion of pathology here is abstract, and can refer to atrophy as measured by volume loss or cortical thinning, white matter degradation as measured by diffusion tensor imaging (DTI) changes in fractional anisotropy (FA), or the level of abnormal conformations of proteins such as amyloid-beta or tau as measured by Positron Emission Tomography. However, BrainPainter is agnostic to the meaning of these biomarkers and can be used with any imaging modality, including markers derived from several modalities together.

Fig. 3 shows an application of BrainPainter by (Young et al. (2018)) to highlight the degree of atrophy in Alzheimer’s disease. Regions with no atrophy are coloured in white, while regions with severe atrophy are coloured in blue. The gradient on the right shows, for every color, the number of standard deviations away from controls.

Figure 3:

Demonstration of BrainPainter for showing extent of pathology on a vertical bar, where colours towards blue show increased severity. Source: Young et al. (2018).

5. Second use case: Visualising the temporal progression of neurodegenerative diseases

In the second use case, we would like to visualise the temporal progression of Alzheimer’s disease (AD). Alzheimer’s disease is characterised by a slow, continuous progression – while it’s mechanisms are still not fully understood, it is currently believed that initial abnormalities in the amyloid and tau proteins cause a cascade of events that eventually lead to axonal degradation, neural death and cognitive decline (Mudher and Lovestone (2002)). Therefore, being able to visualise the progression of these events, including their timing and speed, is crucial for understanding the mechanisms of Alzheimer’s disease.

Fig. 4, reproduced and adapted from Young et al. (2018), demonstrates the ability of BrainPainter to visualise the evolution of atrophy in two subtypes of Alzheimer’s disease – cortical and subcortical – characterised by prominent atrophy in the cortical and subcortical regions respectively. This study done by Young et al. (2018) used data from the Alzheimer’s disease Neuroimaging Initiative to disentangle the heterogeneity of AD into subtypes with different progression. Here, visualisations provided by BrainPainter were able to characterise not only the degree of atrophy in each region (white/red to blue colors), but also the timing of atrophy events. For example, even in the cortical subtype, the hippocampus becomes affected by stage 13, while similarly, in the subcortical subtype the temporal lobe becomes affected by stage 13.

6. Third use case: Visualising pathology in subcortical structures

The ability to visualise subcortical structures is crucial for neurodegenerative diseases that cause damage to these regions. Apart from Alzheimer’s disease, Huntington’s disease (HD) is also known for targetting subcortical regions (Douaud et al. (2009); Wijeratne et al. (2018)). The neurodegeneration in HD is believed to begin in the striatum and pallidum, and later followed by other subcortical and cortical regions (Douaud et al. (2009)).

Fig. 5, reproduced and adapted with permission from Wijeratne et al. (2018), shows visualisations generated by BrainPainter of atrophy progression in subcortical areas, for Huntington’s disease. The images show early involvement of the putamen, caudate and pallidum in the progression of Huntington’s disease, and demonstrate the potential of BrainPainter in visualising pathology dynamics in subcortical regions using parsimonious glass-brain images.

7. Conclusion

We presented BrainPainter, an open-source software that can be used to visualise structures, biomarkers and pathologies in the human brain. The visualisations generated by BrainPainter can be used to significantly enhance the interpretation of neuroimaging research and can be easily embedded by researchers into scientific articles. While not demonstrated here, BrainPainter can also easily generate movies showing dynamic processes, e.g. propagation of brain pathology.

Our software has several limitations that can be addressed in future versions. First, it can currently only highlight entire regions-of-interest from an atlas. However, this was a design choice, as it removes the need for users to create specialised input files with voxelwise measurements, thus increasing usability. Nevertheless, in future versions we might add the ability to highlight fine-grained patterns of pathology. Yet another limitation of BrainPainter is that it cannot visualise more complex structures such as white-matter tracts, although we plan to add such functionality in future releases.

The use of the powerful Blender engine opens numerous avenues not possible with current neuroimaging software: motion graphics can be used to generate realistic movies showing e.g. the evolution of biomarkers, particle simulations can be used to visualise toxic proteins accumulating in certain regions, soft-body simulations can be used to model brain deformations due to head trauma, while camera-based rendering allows the creation of educational videos.