GestaltMatcher Database - a FAIR database for medical imaging data of rare disorders

Abstract

The value of computer-assisted image analysis has been shown in several studies. The performance of tools with artificial intelligence (AI), such as GestaltMatcher, is improved with the size and diversity of the training set, but properly labeled training data is currently the biggest bottleneck in developing next-generation phenotyping (NGP) applications. Therefore, we developed GestaltMatcher Database (GMDB) - a database for machine-readable medical image data that complies with the FAIR principles and improves the openness and accessibility of scientific findings in Medical Genetics.

An entry in GMDB consists of a medical image such as a portrait, X-ray, or fundoscopy, and machine-readable meta information such as a clinical feature encoded in HPO terminology or a disease-causing mutation reported in HGVS format. In the beginning, data was mainly collected by curators gathering images from the literature. Currently, clinicians and individuals recruited from patient support groups provide their previously unpublished data. For this patient-centered approach, we developed a digital consent form. GMDB is a modern publication medium for case reports that complements preprints, e.g., on medRxiv. To enable inter-cohort comparisons, we implemented a research feature in GMDB that computes the pairwise syndromic similarity between hand-picked cases.

Through a community-driven effort, we compiled an image collection of over 7,533 cases with 792 disorders in GMDB. Most of the data was collected from 2,058 publications. In addition, about 1,018 frontal images of 498 previously unpublished cases were obtained. The web interface enables gene- and phenotype-centered queries or infinite scrolls in the gallery. Digital consent has led to increasing adoption of the approach by patients. The research app within GMDB was used to generate syndromic similarity matrices to characterize two novel phenotypes (CSNK2B, PSMC3).

GMDB is the first FAIR database for NGP, where data are findable, accessible, interoperable, and reusable. It is a repository for medical images that cannot be included in medRxiv. That means GMDB connects clinicians with a shared interest in particular phenotypes and improves the performance of AI.

Introduction

Although next-generation sequencing (NGS) has significantly increased the diagnostic yield in rare diseases, it is still challenging to analyze a considerable amount of variants to reach the diagnosis, which is often called the “diagnostic odyssey”¹. With the recent advance in artificial intelligence (AI), next-generation phenotyping (NGP) technology which can analyze the patients’ frontal images, has significantly progressed over the last few years^2–14. Facial analysis applications such as DeepGestalt, which runs behind Face2Gene (https://www.face2gene.com), are increasingly used by geneticists and pediatricians to diagnose patients with facial dysmorphism⁹. The recent approach, GestaltMatcher¹¹, can not only predict the ultra-rare and novel disorders but also delineate the facial phenotypes of the diseases^15–22. Moreover, the facial analysis can be further integrated into the exome variants prioritization to facilitate the diagnosis^23,24.

However, despite the increasing interest and technological advances, properly labeled training data is still the biggest bottleneck in developing NGP applications²⁵. Furthermore, the existing data are often siloed, so curation is usually done repeatedly²⁶. In recent years, principles have been developed to support the reuse of scientific data, thus meeting the increasing need to make data findable, accessible, interoperable, and reusable (FAIR) for humans and computers²⁷. Nevertheless, most current databases, e.g., the London Medical Database (LMD) used to train Face2Gene’s DeepGestalt algorithm, are commercial and not freely accessible to the scientific community, thus building yet another data silo²⁸, even though it has long been clear that patients with rare diseases benefit enormously from data sharing²⁹. Due to the mentioned diagnostic odyssey, patients are increasingly interested in data sharing. A recent study showed that over 73.7% of surveyed parents are willing to allow the storage of their children’s facial images in a secure database that other researchers and doctors can access without general public access³⁰. Therefore, the need and the will to improve the data situation are present, but these efforts often fail due to time-consuming recruitment processes. A faster and easier way for the patient recruiting process is needed.

Therefore, we proposed GestaltMatcher Database (GMDB; https://db.gestaltmatcher.org), a web framework that addresses the needs of human syndromologist first and yields data curation for AI as a by-product. On the one hand, it serves as a modern publication medium that enables updating case reports after the first publication and improves the accessibility from the former image databases such as LMD. With the implementation of digital consent, we also provide a more efficient way of patient recruitment. We have collected 9,863 frontal images from 7,533 patients with 792 different disorders, including 1,018 unpublished frontal images uploaded by 26 clinicians worldwide. At the same time, the data compiled through this community-driven effort was not only made available to other clinicians and scientists but could also be used to train and test AI, in accordance with the FAIR principles. With this incentive, we were able to curate thousands of case reports in a short time and to enhance collaboration amongst researchers all over the world. We envisioned that GMDB could be further connected with other databases with the protocols, such as Beacon³¹ and Phenopackets³².

Results

Overview of GMDB

The aim of the GMDB is, on the one hand, to satisfy the needs of clinicians for a reference database and a more modern publication medium while also achieving data curation with crowdsourced labeling for machine learning. With this incentive, we have curated 9863 frontal images of 7,533 patients with 792 disorders (Figure 1 and Figure 2a). Our curators gathered images from 2,058 publications, and 1,018 frontal images from 498 cases that had not been published before were uploaded by clinicians or patients (Figure 2b).

Figure 1:

Overview of GestaltMatcher database. a) GMDB recruited patients from the publication, clinicians, and patient support group; b) GMDB stored the patient metadata such as sex, ethnicity, age, phenotypic and genomic information; c) GMDB provided two types of data: public and private patients; d) Public patients can be accessible by searching in gallery view. For example, users can search for “Coffin-Siris syndrome” in the gallery and visualize the patient images of Coffin-Siris syndrome. Both public and private patients can be used in the research platform for the patients’ similarity analysis and the training of GestaltMatcher approach.

Figure 2:

Overall statistic of data in GMDB. a) distribution of the public images (images from publication and unpublished image); b) distribution of the unpublished images; c) distribution of age; d) distribution of age; e) distribution of ethnicity; f) distribution of asian ethnicity; g) distribution of african ethnicity; h) distribution of the counts of HPO for each image.

The male patients are slightly more than the female patients (2.6% higher). The age distribution is imbalanced (Figure 2c). 42.8% of images were taken below five years old, and 23.9% were taken between five to ten years old (Figure 2d). The distribution of ethnic backgrounds is also imbalanced. Most images (68.3%) are from Caucasian patients (Figure 2e). The other ethnic groups are under-represented (Asian: 15.9% and African: 6.1%). When we look into the composition of Asian patients, the majority of images have no specific ethnicity labels, and the combination of Chinese, Japanese, and Korean patients is only 8% of the total Asian patients (Figure 2e). African patients are mainly recruited from North Africa, especially Egypt (Figure 2f). We also stored the clinical description encoded in HPO terminology. For the images with HPO terms, 62.1% of images have more than ten HPO terms (Figure 2g).

Reference Database and modern publication medium

In finding the correct diagnosis for a rare genetic disease, comparing the patient’s face with images of other patients with a known diagnosis can sometimes be helpful. A literature search is often tedious due to data siloing and takes much time, which is rarely given in the everyday clinical routine. The “Gallery view” feature in the GMDB makes it easy to search for the candidate gene or disease and visualize all the images relevant to the search, collected by our curators from the literature and the previously unpublished cases, at a glance. Moreover, the user can search for HPO terms³³ or even for PubMed ID (PMID) and DOI for a specific publication. A GMDB user can also easily find his or her cases using a personal reference. This search allows clinicians to compare their patients to the images in the database and get an overview of the heterogeneity of the individual facial characteristics of many diseases. In this context, having many patients of various ethnicities due to our numerous international collaborations is also helpful; these patients are historically underrepresented in other resources³⁴.

In addition to its function as a reference database, the GMDB can also be seen as a new publication medium that shows the most recently published cases in the GMDB. Unlike conventional publication media, which are usually static, a case in the GMDB can be updated at any time, for example, after a follow-up patient consultation in which new symptoms were reported or after new laboratory results were obtained.

Facilitating the NGP development by open database

To address the idea of FAIR principles, we make the dataset of “publicly” uploaded patients in GMDB available to other researchers. They can then use the labeled images and the collected data to train and test their AI model. For example, the disorder prediction tool can be helpful to clinicians in finding a diagnosis, and it is possible to get suggestions for suitable syndromes and the GestaltMatcher score (https://www.gestaltmatcher.org/api.html). Brand et al. describe how the results of analysis with the GestaltMatcher disorder prediction tool helped to solve a typical phenotype of Koolen-de Vries syndrome (KdVS) with an unusual disease-causing mutation, by revealing a high gestalt-score for KdVS¹⁷. Moreover, the direct matching of patients via similarity is possible. Thus, not only matches with already solved cases can lead to a diagnosis. For instance, Marbach et al. describe two patients with a previously unknown genetic disorder caused by the same de novo mutation in LEMD2. GestaltMatcher was used to demonstrate the similarity of the two cases to each other, which supports the assumption that they represent a new phenotype³⁵.

Several other research groups have already achieved promising results by sharing the data. For example, the AI Bone2Gene uses our dataset of hand x-rays to determine bone age and in the future should also learn to reliably predict genetic diseases that manifest on the bone based on the image data³⁶.

Worldwide collaboration

The FAIR database “GMDB” aims to reduce the time to diagnosis for patients with rare genetic diseases by acting as a reference database and as a training set for AI. By matching individual patients instead of classifying diseases, this approach enables clinicians to network based on the phenotype of their patients and enables collaborations around the world.

We currently collaborate with clinicians in Europe, Asia, Africa, and America who have published and analyzed their patients in the database. This globalization can also help to ensure that all ethnic groups are represented in the database and AI training sets, and thus also improve the diagnostic rate in ethnic minorities. At the same time, the community-driven effort worldwide also rapidly increases the number of patients in the GMDB.

The patient-centered approach also allows patients from patient support groups to contribute. We have now established collaborations with seven patient support groups, from which many patients have volunteered to share their photos in the database to shorten the time to diagnosis for others.

Research platform

Beyond the function of the disorder prediction tool, entire patient cohorts can also be analyzed within the Research platform in the GMDB. This research platform can therefore meet three known needs of clinician-scientists in genetics. First, it is possible to quantify the similarity of the individual patients in the cohort by generating a similarity matrix. This approach can detect clusters and see if, for example, cases with an identical variant cluster together. For example, Ebstein et al. showed that facial dysmorphism is more heterogeneous among their PSMC3 patient cohort, but more similarities were found in patients with identical variants²². Using this method in the research platform, similarity has already been quantified in 15 cohorts, of which seven have already been published^{15,16,18–22}.

Discussion

GMDB is a web framework designed to empower the scientific community to compile and query labeled data for deep learning applications in the field of next-generation phenotyping. At the same time, it is also a modern publication organ that can be used as a searchable reference source by clinicians and researchers all over the world.

The ultimate goal is to drive research in rare genetic disorders and shorten the time to diagnosis of these disorders. This is achieved, in part, through GMDB function as a modern publication medium. It displays the latest submitted case reports in the Gallery view, which offers endless scrolling through the gallery. This feature allows phenotypes to be compared, but it can also be used to train residents and students to better and more quickly recognize these disorders based on facial dysmorphisms.

What is remarkable about this form of publication are the possibilities it offers. The previous static publication system is being replaced by a modern, more dynamic approach. Dynamic means updating a case at any time after a further patient consultation or new findings is possible. Often, specific symptoms of a disease do not all appear at the same time but develop over time. When describing a new phenotype, it is possible that further symptoms, which may also be relevant for patient surveillance, occur after publication. These cases can be better represented by a dynamic publication organ such as the GMDB.

It is well known that ethnicity can be a significant bias in detecting rare dysmorphic disorders - not only for AI but also for clinicians. Lumaka et al. also showed that European clinicians are worse at recognizing dysmorphic features in patients of African origin than in patients of European origin³⁷. In the GMDB, clinicians get images of patients’ diseases of all ethnicities available in the database at a glance, and there is no need for a time-consuming literature search anymore. If the database is fed with images of many different ethnicities, the GestaltMatcher AI would also improve its performance for these ethnicities, as Lumaka et al. were also able to show using the example of Face2Gene³⁷.

The research platform and the disorder prediction tool also aim to shorten the time to diagnosis. They offer the possibility to get suggestions for the underlying disorders, for direct matching of patients - with already diagnosed patients, and for matching with still unsolved patients. Tools for matching patients on genotype level with sequencing data already exist (e.g., GeneMatcher³⁸ and DECIPHER). They are connected through the MatchMakerExchange Network API^39,40. In the GMDB research platform, it is now possible to collect cohorts based on phenotype and to examine them to see whether the same or a related genetic mechanism underlies the disorder. Thus, the GMDB can also be seen as a photo version of GeneMatcher and will become a node of the MME network^38,41.

Due to the interoperability of the data in the GMDB, it is possible to connect it in the MME network and may also be used for other tools. For instance, the phenotype also plays a particular role in interpreting genomic variants. It is also possible to link the GMDB with, for example, PEDIA, an AI-based approach that uses portrait images to interpret clinical exome data, improving the performance of bioinformatics pipelines for exome analysis²³.

This database also addresses the problem of preprint servers like medRxiv. So far, all medical photography has to be removed because their board cannot check whether appropriate consent has been obtained. That also means the photos in case reports cannot be made accessible, and GMDB can be used as a repository for such data.

There are also several limitations of this database. Thus, the curation of data by different clinicians and even patients can also lead to different quality of image data. A standardized portrait image without irritating confounding factors, such as different facial expressions, camera angles, or even patients’ items such as glasses, can lead to a distortion of the information for the AI. An example is the problem we have encountered with images from previously published papers, which often use black bars over the patient’s eyes to prevent identification. Moreover, it is not possible to form a completely balanced data set, not only because the disorders vary so much in frequency. Other biases, such as the age and ethnicity of the patients shown in the images, can also affect the performance of AI. To positively influence these limiting factors, an attempt can be made to build up a data set that is as diverse as possible and collect images from patients of different age and ethnicity groups. Due to the dynamic form of publication, follow-up over the years is always possible.

However, the ever-growing numbers of images and cases in the database show how the GMDB can foster global collaboration and finally stop the data siloing by making data findable, accessible, reusable, and interoperable not only for humans but also for computers.

Methods

Curation process:

The curation process can be roughly subdivided into three phases. First, we started having medical students annotate cases from the literature, mainly by searching Pubmed and google scholar for publications with images of patients with facial dysmorphism and monogenic molecular diagnosis.

Second, we started to recruit solved patients from patient support groups. As we aimed to develop a patient-centered platform and strengthen patient autonomy, we collected feedback from the recruited patients during this phase to provide patients with a user-friendly experience. Patients are allowed to upload images and findings autonomously and access their data at any time. To facilitate the retrospective recruitment of patients, we have also implemented digital consent, which allows patients to decide under which conditions they consent to store their data in the database and enables direct signature online. We also further developed this feature in close cooperation with the German Smith-Magenis Syndrome patient organization Sirius e.V. to cover the patients’ requests precisely.

In the last phase, we expanded our database through international collaborations with clinicians from different continents. In the beginning, we also focused on the already solved but not yet published patients to improve the AI’s performance. However, as we progressed, more clinicians shared their unsolved cases with the scientific community. With the increasing number of database users, we also received feedback that other genetic mechanisms should be represented in the database.

Implementation and stored data

We first built an online platform with Ruby on Rails to allow users to input images and other patient data. For the back end, we set up a database by MySQL to store the patient data. An entry in GMDB consisted of a medical image such as a portrait, X-ray, or fundoscopy and machine-readable meta information such as age, ethnicity, and sex. We further collected the clinical features encoded in HPO terminology, the disease-causing mutation reported in HGVS format or ISCN nomenclature, and the test method and zygosity^42–44. Despite the facial photograph, no other patient identifying information is stored in the GMDB. The diagnosed disorders will be stored based on the disorder list from OMIM⁴⁵. The clinician can choose whether the patient was clinically diagnosed or molecularly diagnosed. It is also possible to add potential differential diagnoses. For cases curated from the literature, we collected the DOI and PMID as well as the contact details of the corresponding author. We then clarify whether reuse is possible while respecting intellectual property rights. The privacy settings of the patient, uploaded documents (laboratory findings or letters from the doctor), images from the unaffected family members, and information about the relationship between the cases were all stored.

Clinicians are also asked to provide their expert opinion about the distinctiveness of a phenotype: They have to score whether the medical imaging data was supportive (1), important (2), or key (3) in establishing the clinical diagnosis. Computer scientists can then use this information for the interpretation of the performance of their AIs.

GMDB started focusing on facial portraits of patients with rare monogenic diseases and is currently mainly populated by those cases, but not limited. Later in the curation process, we also annotated cytogenetic disorders with facial dysmorphism. Additional data under curation are X-rays documenting skeletal malformations and photos of the fundus of the eye documenting retinal diseases.

The GMDB architecture

The user interface of the database web service offers three main functions: The Gallery search, data download for developing NGP approaches, and the Research platform.

For diagnosing rare diseases, the face, if it shows dysmorphic features, can give clues to the underlying disease. In the case of very well-known and distinct phenotypes, experienced physicians can rely on their knowledge. Especially in the case of ultra-rare diseases this is often impossible because the physician has not seen a patient before, especially in the case of ultra-rare diseases. Here, sharing patients in publications can expand a physician’s wealth of experience. For this reason, GMDB can also be seen as a novel publication medium. The most recently published cases are displayed in the gallery view. Each case is initially represented only by the portrait image of the patient. After hovering over the image with the mouse, the affected gene and the disease are also displayed. However, the clinician can open the individual data sheet of the patient and thus also view all collected data of the patient. A clinician publishing a case can also select which uploaded images will be displayed in the Gallery.

To enable the analysis of patient cohorts with GM AI, we have also integrated a research platform into GMDB. This allows quantifying and comparing the similarities of all patients stored in the GMDB (from the literature or those publicly uploaded to the GMDB). Patients can be easily selected from the Gallery or “My Patients” section, and additional patients can be added to the research. As a result of the analysis, the Research platform provides a pairwise distance matrix, a pairwise rank matrix, and a tSNE projection⁴⁶.

Data Governance and ELSI

The GestaltMatcher Database (GMDB) is hosted physically in the university hospital of Bonn and guarded by Arbeitsgemeinschaft für Gen-Diagnostik e.V. (AGD) which is a non-profit organization for genomic research. The service is free of charge for the users and is financed by membership fees of the AGD.

The GMDB is only accessible to the scientific community. To protect the patient data, the database underlies strict access control. Registration is only possible after receiving an invitation link from an existing user.

Beyond the initial purpose of GMDB as a search engine for medical imaging data for rare disorders, the framework is, in principle, also suitable for contributing health data to research as a referenceable micro-contribution. Clinicians who have already obtained appropriate informed consent to publish images can use this consent to upload their cases to GMDB. However, a case will only be visible to other users if he or she confirms this again by checking a checkbox and the patient’s image was set to be “public”.

“Public” means that the case is displayed in the Gallery and is also available to other users for further similarity comparisons and can be used for training and testing the GM AI and other AIs. A publication in the GMDB is comparable to a publication in a medical journal. So, when the case was not published before, appropriate consent for publishing the patient in GMDB is acquired. A “private” case, on the other hand, is only visible to the uploading clinician and to the patient himself. The data is included in the training set of the GMDB but is not available to other researchers. This means the case is not displayed in the Gallery and cannot be used to train other AIs. It is also possible to add cases from the existing literature. For this purpose, a PubMed ID or a DOI must be given and the paper’s corresponding author. In this way, these cases are also made reusable in the Research platform.

However, it is usually time-consuming to acquire consent in paper form. Therefore, we also implemented patient-centered digital consent. Patients can directly sign the consent in the database, for example, during a doctor’s visit on a tablet, or they can sign from home by receiving a safe invitation link via email. After creating a password, with this link, they can access their data at any time without having access to the other cases in the GMDB. The consent enables the patient to decide whether their case will only be used for training purposes of the AI or if they are also willing to show their facial image in the gallery and make it public in GMDB and sharable and visible to the database users.

After application with an IRB-approved research proposal, the dataset can be downloaded by other scientists for the training of other AIs.

Advisory Board

All the applications for acquiring the access to download GMDB public data for developing NGP approach will be reviewed by the advisory board consisted of the following coauthors: Koen Devriendt, Shahida Moosa, Christian Netzer, Martin Mücke, Christian Schaaf, Alain VERLOES, Christoffer Nellåker, Markus M. Nöthen, Gholson J. Lyon, Aleksandra Jezela-Stanek, and Karen W. Gripp. Once the majority of the board agree with the application, the applicant will be granted to the download access.