Automatic brain lesion segmentation on standard MRIs of the human head: a scoping review protocol

Abstract

Introduction

Automatic brain lesion segmentation from medical images has great potential to support clinical decision making. Although numerous methods have been proposed, significant challenges must be addressed before they will become established in clinical and research practice. We aim to elucidate the state of the art, to provide a synopsis of competing approaches and identify contrasts between them.

Methods and analysis

We present the background and study design of a scoping review for automatic brain lesion segmentation methods for conventional MRI according to the framework proposed by Arksey and O’Malley. We aim to identify common image processing steps as well as mathematical and computational theories implemented in these methods. We will aggregate the evidence on the efficacy and identify limitations of the approaches. Methods to be investigated work with standard MRI sequences from human patients examined for brain lesions, and are validated with quantitative measures against a trusted reference. PubMed, IEEE Xplore and Scopus will be searched using search phrases that will ensure an inclusive and unbiased overview. For matching records, titles and abstracts will be screened to ensure eligibility. Studies will be excluded if a full paper is not available or is not written in English, if non-standard MR sequences are used, if there is no quantitative validation, or if the method is not automatic. In the data charting phase, we will extract information about authors, publication details and study cohort. We expect to find information about preprocessing, segmentation and validation procedures. We will develop an analytical framework to collate, summarise and synthesise the data.

Ethics and dissemination

Ethical approval for this study is not required since the information will be extracted from published studies. We will submit the review report to a peer-reviewed scientific journal and explore other venues for presenting the work.

Strengths and limitations of this study.

• Our study design ensures an inclusive and unbiased review while maintaining good quality of the gathered sources by proposing the requirement of a quantitative validation of the presented methods.

• We will validate our search strategy by comparing the bibliographies and citations of the most recent and most cited records with the gathered sources.

• Criteria for including studies in the scoping review may turn out to be too generous, with the number of matching papers exceeding our capacity for reviewing them.

Introduction

In clinical practice, diagnosis of brain lesions is based on the patient’s history, clinical presentation, visual assessment of appointed scans, and other laboratory examinations. MRI has become an important tool in brain lesion identification and classification due to its ability to produce images with high contrast resolution and sensitivity for abnormalities. Various conditions can give rise to such lesions. The most common causes include trauma, inflammation and autoimmune diseases, stroke, malignant or benign tumours and infections.¹ Although brain lesions tend to appear significantly different from healthy tissues on MR scans, differentiating between brain lesion causes based on visual examination can be difficult or impossible. Still, visual interpretation is the most common and trusted mode of image analysis in clinical practice. Accurate identification and delineation of lesion boundaries and classification of lesional tissue components is particularly important in treatment planning for surgery or radiation therapy in tumour patients. It is also essential for determination of disease burden, prognosis and therapy response in nearly all types of brain lesions. The process is currently commonly performed manually by an expert rater. The procedure is tedious, time-consuming and subject to inter-rater as well as test-retest variability.^2–5

Automatic image segmentation methods promise to reduce or eliminate subjective decisions in this process, facilitating fast and accurate delineation of lesions and classification of their components on MR brain images. Although many automatic brain lesion segmentation methods have been proposed, substantial challenges remain, for example the variable appearance of the lesions on MR images due to unknown, possibly biological factors; differences of image acquisition protocols between centres; and the difficulty of validating such algorithms on sufficiently large case numbers.^6–8 Taken together, these challenges explain why no single tool or approach has thus far been adopted in clinical or even in research practice. On surveying the literature on automatic brain lesion segmentation methods in an ad hoc, preliminary fashion, we recognised the need for a rigorous and comprehensive review. A formalised approach to reviewing literature in this manner is the scoping review as proposed by Arksey and O’Malley.⁹ Using this framework along with refinements by Levac et al ¹⁰ and Colquhoun et al,¹¹ as well as elements of the PRISMA and PRISMA-ScR guidelines,^{12 13} we will map key concepts, converging developments, challenges and promising new research avenues. The purpose of publishing the research plan at this stage is to document our objectives openly, to invite comments and suggestions and to enhance the rigour of our study. This open documentation will compel us to follow the plan and justify any deviation. We believe that being fastidious in this manner will enhance the value of the research once completed.

Study aims and objectives

We have identified the following aims and objectives of the scoping review on existing automatic brain lesion segmentation techniques on conventional MR images:

1. Elucidate the state of the art and provide a synopsis of competing approaches to automatic brain lesion characterisation.

2. Identify common procedures necessary for automatic brain lesion segmentation on conventional MR images.

3. Identify MR data sets with reference segmentations and/or diagnostic classifications that are publicly available for method validation.

4. Identify common challenges, weaknesses and controversies, as well as unaddressed issues which can signify opportunities for future work to improve segmentation methods.

Methods and design

General

This section describes how each of the scoping study stages identified by Arksey and O’Malley⁹ will be applied to the present study. In this study we will balance the breadth of the included studies with the depth of the analysis of reviewed methods. The nature of the researched topic imposes certain logical limitations which, together with the inclusive nature of a scoping review, will help to create a focused yet comprehensive overview of the topic.

Stage 1: identifying research questions

We have defined the following research questions that will be addressed in this study.

1. Which common image processing steps are necessary for automatic brain lesion segmentation on MR images?

2. Which mathematical and computational theories are most commonly applied in which types of brain lesions?

3. What is the efficacy of existing implementations?

4. What are the limitations of those methods and issues that should be addressed in future studies to develop a tool that is suitable for clinical use?

5. What are the most commonly used MR data sets that provide reference lesion segmentation and/or diagnostic classification?

Stage 2: identifying relevant studies

Eligibility criteria

We established the following initial criteria for the proposed segmentation methods to be eligible for inclusion in the scoping review. A method must be applied to one or a number of commonly acquired MRI sequences from human subjects investigated for a condition known to cause brain lesion(s).

The proposed methods should be validated by comparison with a gold-standard reference segmentation of the lesions. The efficacy of the method should be reported providing quantitative scores such as sensitivity, specificity, overlap, surface distance or volume error. Alternative validation approaches will be considered if they have face validity. A thorough validation is a necessary step in developing medical image segmentation methods. Even though we dispense with formal quality assessment of the included studies, we believe that our principled approach will enable us to provide a valuable report on the researched topic.

We do not define any particular study designs types as an inclusion criterion for this scoping review. This ensures inclusion of diverse approaches and designs and will potentially reveal which ones are favoured by a plurality of authors. This aligns with the generic aim of undertaking a scoping review, which is to investigate the range and type of evidence in a given field.

Initial search

The proposed search strategy was thoroughly discussed and approved by the scoping study authors. We also took advantage of services provided by our university library and consulted with a librarian on the search strategy. Eligible studies will be retrieved from peer-reviewed journal articles and conference papers. We will not impose any limitations with respect to year of publication at this stage of the study.

The search will be conducted using search terms constructed in English.

Three online databases will be searched: PubMed, IEEE Xplore and Scopus. The following search phrases will be constructed using non-controlled vocabulary to initialise the search. An advanced search in publication metadata will be conducted in all databases using the following search terms to identify potentially relevant sources (asterisk indicates a wild card character to account for variations in the spelling of the search terms):

1. automat(ic)*

2. AND brain

3. AND lesion OR tumor OR neoplasm

4. AND segment(ation)* OR identif(ication)* OR delin(eation)* OR classif(ication)*

5. AND mri OR mr.

The search results will be exported from each database. From the controlled vocabulary tags (MeSH in PubMed; IEEE terms and INSPEC terms in IEEE Explore; index keywords in Scopus), we will build frequency tables. The most common relevant terms will be used to refine the original search phrase. Combining free text and index terms ensures high sensitivity of the search. The results will be refined by applying possible limitations defined in the eligibility criteria depending on the availability in the search engine, such as document type (journal and conference articles) or species (human).

A separate search will be conducted in PubMed to identify potentially eligible papers that have not been indexed with MeSH yet. MeSH terms are assigned by specialists at the National Library of Medicine after a variable delay, meaning that some recent papers lack them. We will modify the search phrase and look for the search terms as well as MeSH terms in all fields, and an additional status criterion will be added to exclude MeSH indexed papers.

We will screen bibliographies of the most recent papers as well as citations of the most cited papers and compare it with the existing sample to evaluate inclusiveness and validate the proposed search strategy. We will identify the most cited studies by dividing the number of citations of a given paper by the number of months since the publication. If there is a substantial mismatch between the existing set of selected studies, and the bibliographies and citations, we will identify additional sources by screening the bibliographies and citations of the identified set.

Screening

The records found in the search phase will be exported to a reference management software (Zotero) to scan for and remove duplicated items. The screening of the identified records after duplicate removal will be conducted in two phases using web based application for systematic review—Rayyan QCRI.¹⁴

First, we will rapidly screen the titles to exclude papers that evidently do not match the selection criteria. We will exclude any papers that deal with imaging modalities other than MRI, or where the title suggests that the study does not propose an automatic brain lesion segmentation method.

In the second phase, abstracts of the papers that passed the previous phase will be screened to identify and exclude irrelevant or otherwise ineligible items. The screening will be performed by finding the key terms or their synonyms in the publication title or abstract and determining if the publication is relevant. Articles that do not name in the title or the abstract or refer to any proposed method of any form of identification of any type of brain lesion will be excluded under the assumption that those papers either do not contain enough evidence for the method to be eligible for the synthesis, or do not propose a lesion segmentation method. The following key terms (and their synonyms) will be considered:

1. method

2. identification

3. brain lesion

4. magnetic resonance imaging

At the screening stage, records will be excluded if a given study has previously been reported or any of the study characteristics stated in the abstract clearly do not match the eligibility criteria for this study.

Stage 3: study selection

For items selected during the screening stage, full-text articles will be retrieved. The following criteria will be considered as a reason for exclusion of a paper from the review:

• Full paper not available.

• Paper written in a language other than English.

• Modality other than standard MRI used.

• No quantitative validation found.

• Semiautomatic method proposed.

The terms automatic and semiautomatic segmentation do not have a widely accepted definition. For our purposes, a semiautomatic method shall be one that relies on expert’s decisions during the segmentation process, while an automatic method is one that requires a user to provide (possibly preprocessed) images and launch the programme, after which all decisions regarding lesion segmentation and characterisation are made without human interaction.

Stage 4: data charting

In the data charting phase, the following study information will be extracted from every eligible record: author(s), year of publication, country of origin and funding information (if available). We will note the clinical diagnosis of the patients and MRI sequence(s) used in a given study as well as the type of brain lesion(s) studied. Based on our knowledge in the field of medical image segmentation we expect to find information on the following main categories in the method description⁸:

1. Image preprocessing methods and procedures (eg, registration, skull stripping, intensity inhomogeneity correction, noise reduction, intensity normalisation).

2. Segmentation methods and particular computational and mathematical theories applied.

3. MR database used.

4. Validation (eg, amount and types of reference data; accuracy and reliability measures).

5. Efficacy of the methods in terms of applicability to a predefined task.

6. Limitations of the method stated by authors.

Following the suggestions included in the PRISMA checklist for scoping reviews,¹³ we will critically appraise the information in the gathered studies. Together with the information collected in the data charting phase, it will help us identify the common patterns, efficacy and limitations of the studies and presented methods. The elements we will appraise will include method and material description, preprocessing description, robustness of the method and validation procedure including reference segmentation information and segmentation evaluation measures.

To test the proposed approach we will conduct a pilot charting and appraisal on a subset of recent studies to evaluate and refine the charted variables. Implementation details will be decided during a pilot phase of data charting and critical appraisal.

Given the extent and variability in reporting the information to be extracted, we may have to update the charting form in an iterative manner even after conducting a pilot charting.¹⁰ Levac et al ¹⁰ note that it is nearly impossible to design an adequate charting form at the outset, and recommend iterative refinement.

Stage 5: collating, summarising, and reporting the data

Once the eligible studies have been collected, we will develop an analytical framework to collate, summarise and synthesise the data. We will make use of summary counts and tables to provide quantitative information on the body of research on automatic brain tumour segmentation methods. While analysing the data, common procedures for the methods, types of lesions and their outcomes will be identified. We will also present an inclusive comparison of methods and their performance that use the same MR datasets and segmentation evaluation measures, if their study design allows for making such a comparison. We will identify discrepancies between the analysed methods and use that information to address Objective 4 (cf. Study aims and objectives). The consultation stage of the scoping review, described in the following section, will contribute to fulfilling that objective. We allow for the possibility to adjust or expand the initial analytical framework after the consultation stage to present the gathered information according to the stakeholders’ requests.

Stage 6: consultation

Although the consultation stage is currently considered optional in scoping reviews, we see advantages in including this stage in our study. We will use this opportunity to share preliminary findings and refer to potential stakeholders to gain more insight into our data from different perspectives. The consultation will be conducted using a questionnaire or through interviews, however a detailed design of the consultation process will be created after finishing collating, summarising and internal reporting the data.

Study limitations

While we aim to conduct a well structured and reproducible study, we are aware that our approach has limitations. In the defined inclusion criteria, we limit the eligible sources to journal articles and conference papers. We presume that any other sources, such as posters, books, theses and so on, will contain ineligible evidence or duplicates of included papers. We are aware of that our presumption may not be true in all cases and thus we risk excluding some eligible studies.

In the study identification phase, we attempt to construct a very sensitive search phrase with high precision. Our strategy may, however, turn out to be insufficiently inclusive, and a substantial number of additional, potentially eligible studies will be identified by screening the references and citations of selected papers. Since the process is time consuming, we will have to limit the number of papers we will consider for reference and citation screening. This may introduce subjectivity and we may not be able to identify every relevant study.

Since the nature of a scoping study is to identify evidence in a given field without particular assumptions about the designs of the sampled studies, creating an optimal framework for data charting, appraisal and synthesis is a complex task. Evaluating efficacy of the gathered studies poses a particular challenge, and we may not be able to provide an objective comparative assessment of segmentation methods’ efficacy.

We attempt to ensure reproducibility of our study through rigorous planning and thorough documentation of the study methodology. In addition, a single investigator (EAG) will be responsible for identifying and selecting eligible papers, as well as extracting and summarising the relevant information for the study. We thus avoid inter-rater variability in the protocol implementation, accepting as a trade-off that we may have to constrain the sample to account for the amount of time available to the investigator.

Patient and public involvement

Neither patients nor public were involved in the development of this study design.

Ethics and dissemination

The study will contain information gathered from already published papers therefore it does not require ethical approval. We will distill the project diary (cf. General) into a review report for submission to a peer-reviewed scientific journal. In addition, we will seek to present the study at scientific conferences. Following on from the work done at the consultation stage, we will identify stakeholders outside of academia and seek to disseminate the results to them in appropriate formats (trade journal articles, lectures, laypersons’ summaries, press releases).