The Impact of Structural Genomics: the First Quindecennial

Abstract

The period 2000–2015 brought the advent of high-throughput approaches to protein structure determination. With the overall funding on the order of $2 billion (in 2010 dollars), the structural genomics (SG) consortia established worldwide have developed pipelines for target selection, protein production, sample preparation, crystallization, and structure determination by X-ray crystallography and NMR. These efforts resulted in the determination of over 13,500 protein structures, mostly from unique protein families, and increased the structural coverage of the expanding protein universe. SG programs contributed over 4,400 publications to the scientific literature. The NIH-funded Protein Structure Initiatives (PSI) alone have produced over 2,000 scientific publications, which to date have attracted more than 93,000 citations. Software and database developments that were necessary to handle high-throughput structure determination workflows have led to structures of better quality and improved integrity of the associated data. Organized and accessible data have a positive impact on the reproducibility of scientific experiments. Most of the experimental data generated by the SG centers are freely available to the community and has been utilized by scientists in various fields of research. SG projects have created, improved, streamlined, and validated many protocols for protein production and crystallization, data collection, and functional analysis, significantly benefiting biological and biomedical research.

Introduction: A Brief History of Structural Genomics

The idea of a large-scale, high-throughput approach to protein structure determination arose in the late 1990s, spurred in part by the spectacular success of the Human Genome Project, and rapid progress in DNA sequencing technology [1]. This was accompanied by construction of synchrotron light sources and other structural biology facilities. Several large-scale structural genomics (SG) programs were established at the beginning of the 21^st century, with the intention of achieving comprehensive structural coverage of the protein universe. The original goal envisioned making the 3-D structure of most proteins easily available, either directly from the Protein Data Bank (PDB), or through computational modeling methods. At the outset of SG, it was estimated that by solving ~16,000 structures of “carefully selected proteins” it would be possible to obtain structural coverage of “the vast majority of all proteins” (~90%) [2]. However, it was soon realized that this goal might be very challenging due to the continued exponential growth of the number of known protein sequences [3,4]. Later, structural genomics programs refocused on protein families with high biological importance.

In 2000, the US-based National Institutes of Health (NIH) created the Protein Structure Initiative (PSI), “a national effort to assemble a large collection of protein structures in a high-throughput operation” [5]. In its 5-year pilot phase, PSI included nine structure determination centers, with a total funding of 270 million dollars [5]. During this initial phase, it was shown that the high-throughput approach was feasible and could be applied to many protein classes. Several protein production and structure determination pipelines were created using both X-ray crystallography and NMR. In 2005, PSI entered its “production” phase, in which 325 million dollars were budgeted for four large-scale structure determination centers and six specialized centers targeting more specific biology areas [6]. The funding also supported homology modeling centers, a material repository, a knowledgebase, and several synchrotron beamlines. These investments expanded access of the biology community to structural data and models, materials, and technologies and attempted to convert the data and results of PSI into new knowledge. In 2010, PSI evolved into the PSI:Biology program, which associated high-throughput structure determination centers with “High Throughput-Enabled Structural Biology Partnerships.” The price tag for PSI:Biology was 290 million dollars [7] and included funding for four high-throughput centers, nine membrane protein centers, fifteen Biology Partnerships, a material repository, the PSI knowledgebase with homology modeling, and support for several synchrotron beamlines. Before PSI officially ended in July 2015, its combined programs deposited around 7000 protein structures into the PDB. Nearly 90% of these were determined by X-ray crystallography, the rest by NMR.

In 2002, Japan started its own 5-year structural genomics program, Protein 3000, aiming to analyze structural-functional relationship of about three thousand proteins with biomedical importance [8]. The program was implemented by the RIKEN Structural Genomics/Proteomics Initiative (RSGI), with a yearly budget of 45 million dollars, an additional 45 million at collaborating sites [9], and additional funding from partnerships with the pharmaceutical industry [10]. The project set up robotized protein production and sample screening systems and generated a total of 2743 PDB deposits, of which approximately half were determined using NMR, and half using X-ray crystallography.

In 2003, the Structural Genomics Consortium (SGC) was established by researchers from Canada and Great Britain to “focus explicitly on less well-studied areas of the human genome” [11]. At its outset, it secured nearly 100 million dollars funding from the Canadian research organizations, the Wellcome Trust charity, and from pharmaceutical companies. An additional 50 million was committed in 2011 [12]. SGC has grown to include researchers from Sweden, Brazil, Germany, and the US, and so far has contributed around 1400 protein structures to the PDB.

In 2008, the National Institute for Allergy and Infectious Diseases (NIAID) started a structural genomics program targeting the emerging and re-emerging (drug-resistant) infectious human pathogens. The program established two centers, and emphasized target submissions from the wider biology community. NIAID invested $72 million during the first 5 years of the program, and in 2012 budgeted an additional $50 million for the next 5 years [13]. These SG centers took advantage of the technologies, protocols and structure determination pipelines developed by the PSI. As of December 2015, the two NIAID centers have deposited around 1600 protein structures from pathogens into the PDB, including complexes with inhibitors and potential drugs.

There have been other large-scale programs similar to the SG model. A prominent example is the Enzyme Function Initiative (EFI), which operated from 2010 to 2015 and integrated bioinformatics, structural biology, enzymology, genetics, and metabolomics to develop strategies for discovering enzymatic and physiological functions of uncharacterized enzymes [14]. While it was not a SG program itself, the EFI made use of the technologies and protocols developed by the PSI, such as the large-scale pipelines for enzyme production and structure determination, and contributed around 340 deposits to the PDB.

In addition to the major programs described above, there have been around twenty smaller centers in Europe, Asia, and North America that have also contributed PDB deposits classified as “structural genomics.” This classification has been essentially self-assigned by the depositors, and is indicated by the content of the set of fields in the category “pdbx_SG project” [15] in the mmCIF files of the PDB deposits. However, the smaller centers accounted for a relatively small percentage of the SG deposits to the PDB, with the five major initiatives – PSI, Protein 3000, the NIAID centers, SGC, and EFI contributing more than 95% of SG deposits to the PDB. The smaller centers include several programs funded by the European Union and its member countries, the largest of which was the 3-year program SPINE (Structural Proteomics in Europe), which began operations in 2002 and determined over 100 protein structures [16,17]. The Tuberculosis Structural Genomics Consortium, originally formed as part of PSI, has been continuing as an international collaboration comprising almost 500 researchers [18,19]. The “taxonomy” of the all currently living and extinct SG projects is presented in Table 1.

Table 1.

The taxonomy of Structural Genomics: list of the different SG centers, according to the contents of the pdbx_SG_project fields in the mmCIF files in the PDB (curated manually), as of December 31, 2015. Joint deposits for two or more centers are included in the total for each center.

Center	Country	Funding	# Structures
Accelerated Technologies Center for Gene to 3D Structure (ATCG3D)	USA	PSI-2 specialized center	29
Assembly, Dynamics and Evolution of Cell-Cell and Cell-Matrix Adhesions (CELLMAT)	USA	PSI:Biology partnership	2
Atoms-to-Animals: The Immune Function Network (IFN)	USA	PSI:Biology partnership	31
Bacterial targets at IGS-CNRS, France (BIGS)	France	EU	4
Berkeley Structural Genomics Center (BSGC)	USA	PSI-1	103
Center for Eukaryotic Structural Genomics (CESG)	USA	PSI-1,2	218
Center for High-Throughput Structural Biology (CHTSB)	USA	PSI-2 specialized center	6
Center for Structural Genomics of Infectious Diseases (CSGID)	USA	NIAID	811
Center for Structures of Membrane Proteins (CSMP)	USA	PSI-2 and PSI:Biology specialized center	28
Chaperone-Enabled Studies of Epigenetic Regulation Enzymes (CEBS)	USA	PSI:Biology partnership	6
Enzyme Discovery for Natural Product Biosynthesis (NatPro)	USA	PSI:Biology partnership	66
Enzyme Function Initiative (EFI)	USA	NIH “glue grant”	338
GPCR Network (GPCR)	USA	PSI:Biology membrane protein center	48
Integrated Center for Structure and Function Innovation (ISFI)	USA	PSI-2 specialized center	28
Israel Structural Proteomics Center (ISPC)	Israel	EU	45
Joint Center for Structural Genomics (JCSG)	USA	PSI-1,2, Biology large scale center	1601
Marseilles Structural Genomics Program @ AFMB (MSGP)	France	EU	12
Medical Structural Genomics of Pathogenic Protozoa (MSGPP)	USA	US	35
Membrane Protein Structural Biology Consortium (MPSBC)	USA	PSI:Biology membrane protein center	3
Membrane Protein Structures by Solution NMR (MPSbyNMR)	USA	PSI:Biology membrane protein center	5
Midwest Center for Structural Genomics (MCSG)	USA	PSI-1,2, Biology large scale center	1872
Mitochondrial Protein Partnership (MPP)	USA	PSI:Biology partnership	10
Montreal-Kingston Bacterial Structural Genomics Initiative (BSGI)	Canada	Canadian research organizations	126
Mycobacterium Tuberculosis Structural Proteomics Project (XMTB)	Germany	German government	36
New York Consortium on Membrane Protein Structure (NYCOMPS)	USA	PSI:Biology membrane protein center	58
New York SGX Research Center for Structural Genomics (NYSGXRC)	USA	PSI-1,2 large scale center	1042
New York Structural Genomics Research Consortium (NYSGRC)	USA	PSI:Biology large scale center	344
Northeast Structural Genomics Consortium (NESG)	USA	PSI-1,2, Biology large scale center	1212
Nucleocytoplasmic Transport: a Target for Cellular Control (NPCXtals)	USA	PSI:Biology partnership	5
Ontario Centre for Structural Proteomics (OCSP)	Canada	Canadian research organizations	35
Oxford Protein Production Facility (OPPF)	UK	EU	33
Paris-Sud Yeast Structural Genomics (YSG)	France	EU	6
Partnership for Nuclear Receptor Signaling Code Biology (NHRs)	USA	PSI:Biology partnership	3
Partnership for Stem Cell Biology (STEMCELL)	USA	PSI:Biology partnership	20
Partnership for T-Cell Biology (TCELL)	USA	PSI:Biology partnership	48
Program for the Characterization of Secreted Effector Proteins (PCSEP)	USA	PSI:Biology partnership	17
Protein Structure Factory (PSF)	Germany	EU	2
RIKEN Structural Genomics/Proteomics Initiative (RSGI)	Japan	Protein 3000 project	2743
Scottish Structural Proteomics Facility (SSPF)	UK	EU	11
Seattle Structural Genomics Center for Infectious Disease (SGCID)	USA	NIAID	801
Southeast Collaboratory for Structural Genomics (SECSG)	USA	PSI-1	121
Structural Genomics Consortium (SGC)	International	Industry, charities, Canadian research organizations	1404
Structural Genomics Consortium for Research on Gene Expression (SGCGES)	USA	PSI:Biology partnership	2
Structural Genomics of Pathogenic Protozoa Consortium (SGPP)	USA	PSI-1	71
Structural Proteomics in Europe (SPINE)	EU	EU	119
Structural Proteomics in Europe 2 (SPINE-2)	EU	EU	8
Structure 2 Function Project (S2F)	USA	US	54
Structure-Function Analysis of Polymorphic CDI Toxin-Immunity Protein Complexes (UC4CDI)	USA	PSI:Biology partnership	3
Structures of Mtb Proteins Conferring Susceptibility to Known Mtb Inhibitors (MTBI)	USA	PSI:Biology partnership	36
TB Structural Genomics Consortium (TBSGC)	International	PSI-1; international	285
Transcontinental EM Initiative for Membrane Protein Structure (TEMIMPS)	USA	PSI:Biology membrane protein center	2

SG programs, like many other “big science” initiatives, have encountered a variety of often polarized reactions from the scientific community, from a very strong support to, in some cases, strong criticism [8,20–22]. In particular, the target selection process in some SG centers was criticized. The list of “carefully selected proteins” envisioned at the outset [2] has never been actually compiled, and over the years the list of proteins targeted by SG grew to over 330,000 entries. Critics also argued that the funding expended for SG would have been more effective if it were instead directed to traditional, individual programs (e.g. R01). Proponents of structural genomics emphasized that the scientific community at large benefitted from access to publically available technologies, data, structures, materials, software, synchrotron beamlines, NMR facilities etc. developed by the SG programs. Although discussion of such policy and philosophical questions is beyond the scope of our paper, we will briefly summarize the economic aspects of SG.

Although it is difficult to know the budgets of all individual centers, a reasonable estimate of the total costs of the worldwide SG to date is $2.0 billion in 2010 dollars. This amount has been obtained by the summing up the known expenditures of the four major programs (inflation-adjusted to 2010) and extrapolating the result to the other programs. Around 60% of SG funding was provided by the US governmental agencies. While these are significant amounts, they should be put in perspective. The PSI program represented a mere 0.2% of the total NIH budget during the 15 years when PSI was operating. In comparison, the cost of the Human Genome Project was $5.6 billion in 2010 dollars [23,24]. As an investment, the Human Genome Project was a resounding success, driving an estimated $800 billion of worldwide economic impact by the year 2010 [24].

There have been no comparable studies of the economic impact of SG programs and, in our opinion, it would be premature to try to determine it now. It should be emphasized that the two megaprojects were significantly different. The Human Genome Project concentrated its resources on the development of new, focused technologies and methodologies (DNA sequencing and analysis of the data), and the resulting sequence of the human genome was its only end goal and the proof of its success. SG centers have concentrated their resources on a process that is orders of magnitude more complex - from gene sequence to three-dimensional structure. It involved genome analysis and classification of protein families, recombinant technologies for cloning thousands of genes, large-scale production of unique proteins, crystallization, data collection using X-ray crystallography and NMR, de novo structure solution of novel proteins and structural analysis. The integration of many steps and different kind of data creates an enormous amount of complexity. SG programs generated large amounts of experimental data/results produced in a highly reproducible manner; in particular, they cloned over 220,000 protein genes. DNA sequencing programs are not directly comparable to SG efforts due to the differing natures of the problems addressed. Sequencing of large genomes is largely a single step process, whereas structure determination is significantly more complex experimentally, requiring a high level of integration in a multi-step process with a large number of experiments that have to be often replicated.

As of the time of the writing, funding for SG projects has been greatly reduced - only two of the major SG initiatives (NIAID and SGC) are still active. It is an appropriate time to ask: what has been the gain for humanity for the two billion dollars invested in SG so far? In this work, we try to answer this question by discussing the impact of SG in five specific areas: (1) structures and modeling, (2) publications, (3) experimental data, (4) structure determination technologies, and (5) function determination technologies.

Structures, Modeling Leverage and Modeling Methods

The most often used metric of the outcome of SG is the total number of structures deposited into the PDB. As of December 31, 2015, there were 13,606 PDB deposits (including 11,337 solved by X-ray crystallography, 2267 solved by NMR, 1 solved by electron microscopy, and 1 by combination of NMR and solution scattering) that were attributed to SG centers (as indicated by the mmCIF fields “pdbx_SG_project,” manually curated), constituting 11.8% of all PDB deposits. The PSI alone accounted for slightly over half of all SG deposits. The SG contribution was especially high in terms of “unique” deposits–defined as those for which there was no structure in PDB with ≥ 30% sequence identity at the time of the deposition. As of December 2015, SG contributed around 5100 such structures. The SG contributions constituted about a quarter of all the unique X-ray crystallography deposits in the PDB and nearly a third (29.7%) of the unique NMR deposits.. The growth of SG deposits is shown in Fig. 1.

Fig. 1.

The structural output of SG programs. The vertical axis represents the number of structures deposited to the PDB by SG programs worldwide. Structures were assigned to a particular SG program based on the contents of mmCIF files (field _pdbx_SG_project) with manual curation in the PDB as of December 31, 2015.

These unique deposits are important in terms of “novel modeling leverage,” as they provide templates to build comparatively reliable homology models for domain families that previously had no structural data [25]. Increased structural coverage has also benefited crystallography, as non-redundant structures can be used as templates for structure determination using molecular replacement (MR). This means that crystallography workflows can be simplified and significantly accelerated, facilitating determination of more 3D structures.

SG target selection strategies, particularly at the beginning of SG, focused on selecting targets from the largest structurally uncharacterized families in an effort to maximize the structural coverage of “protein sequence space” [26]. However, the rapid growth in known protein gene sequences due to revolutionary advances in DNA sequencing technology has made this goal more challenging. In 2000, the entire known protein sequence universe consisted of about 300,000 sequences, as measured by the size of the UniProt and TrEMBL databases [27]. In January 2016, the UniProtKB/TrEMBL database had nearly 60 million “non-redundant” entries comprising nearly 20 billion amino acids [28]; the NCBI Reference Sequence (RefSeq) database (release 73) contained a similar number of proteins. The growth of the total number of structures has not been able to match this pace: in recent years, the number of known proteins has been growing exponentially at a rate of about 60% per year, while the number of structures in the PDB has been growing by about 10% per year. Furthermore, the highly skewed, power law distribution of the sizes of protein domain families means that whereas there are a few very large families with many sequences, there are also many “orphan” ones with only one or a few members [29,30].

Despite these challenges, there is evidence to suggest that SG efforts have led to significant progress in increasing the modeling leverage of protein sequence space (the set of proteins for which structures may be obtainable through homology modeling). One study demonstrated that the PSI’s per structure novel coverage rate was four times higher than the equivalent rate for structures solved by non-PSI centers [31]. A more recent study has established that while the rate of growth of sequence databases is still increasing, they are also changing in taxonomic composition (a relative increase in the number of prokaryotic sequences) and are becoming increasingly redundant [32]. Overall, the total structural coverage of the GenBank non-redundant set of protein sequences increased from 30% in 2001 to 40% in 2011, and the increase in the degree of coverage in organisms of greater relevance to human health, such as E.coli, was even greater [33].

To effectively utilize the structural coverage provided by SG, structural and computational biologists developed many comparative modeling methodologies. In particular, PSI laboratories developed modBase–a database of comparative protein structure models [34] and the Protein Model Portal [35], which allows users without expert knowledge to download precomputed models or compute them on demand. The methods of target selection employed by SG, which were designed to maximize structural coverage, were also beneficial to initiatives like the Critical Assessment of Structure Prediction (CASP) [36], in which SG laboratories participated by providing novel structures for blind test cases. The influx of new structures with low sequence similarity to existing models allowed for systematic evaluation of modeling software by challenging them with very difficult comparative and ab initio modeling problems. The CASP competition, which heavily utilized novel SG targets, provided an incentive for software developers to improve both automatic and human-assisted structure prediction.

For some smaller proteomes, SG efforts have succeeded in providing extensive coverage. The proteome of the Thermotoga maritima bacteria consists of 1852 confirmed and predicted proteins (UniProt database release 2016_01, entry up000008183) of which 387 have their 3D structures determined. Nearly 60% of these (227) were first contributed by SG centers. An additional 451 Thermotoga proteins have comparative models available from the Protein Model Portal. Overall, SG provides structures or structural models for close to 37% of the proteome. Successful processing of T. maritima proteome initiated by SG [37] generated enough data to construct a working model of the metabolic networks of this organism [38], which permits simulations of metabolism in the context of whole cell.

The human proteome contains around 20,000 proteins encoded in the genome [39], comprising 11.3 million amino acids (neXtProt database release 2015-09-01) [40]. In the PDB there are currently around 28,500 Homo sapiens entries representing 5485 unique human proteins annotated in neXtProt (3.3 million amino acids) whose structures have been experimentally determined. For nearly a quarter of these (1359 proteins), SG projects have contributed the earliest PDB deposit: RSGI alone determined the first structure for 633 human proteins, the SGC for another 543. The SG contributions correspond to 0.9 million amino acids – 27.9% of the total number of amino acids in human proteins with experimentally determined structures. Structural models for around 14,000 additional human proteins (obtained by automatic modeling methods) have been made available by the Protein Model Portal. Interestingly, nearly all the Gene Ontology (GO) terms occurring in the human proteome have been already structurally characterized, in the sense of having at least one protein with that annotation in the modeling leverage of the PDB [41].

Around 20 to 30% of most eukaryotic and eubacterial proteomes consists of integral membrane proteins (IMPs) [42] for which determination of structure is notoriously challenging. SG efforts have made significant contributions to the progress in this area. Specialized SG centers have targeted over 15,000 IMPs, succeeding in “plucking” a number of “high-hanging” novel membrane targets which increased the structural coverage of membrane proteins by 7% (in terms of providing the first structure for an IMP Pfam-family) [43]. The impact of SG membrane centers, in particular the specialized PSI centers extends beyond the structural coverage: over 2000 purified IMPs have been made available [43]. Combining insights gained from the experimental pipeline and modeling of IMPs, SG has spurred novel approaches to structural characterization of the entire human alpha-helical transmembrane proteome [34].

Publications

The number of peer-reviewed publications and the number of times they are cited is often used as a measure of the impact of research projects. Until the 1990s, almost any new protein structure warranted a publication, but in the SG model of high volume structure determination, this was simply not possible. All SG consortia were supposed to immediately deposit the structure into the PDB, and not enough time and effort was allotted to obtain “publishable” results such as the determination and validation of function. The difference between the rate of publications associated with SG and non-SG deposits is striking (see Fig. 2). For “traditional” structural biology, the percentage of deposits without a primary citation (marked “to be published” in the PDB deposits) has remained below 15% throughout the last 15 years, though this percentage has been trending upwards. In contrast, around 80% of recent PSI deposits remain “to be published.”

Fig. 2.

Percentage of PDB structures with primary citation “to be published” in the period 2000–2015 in function of the year of deposition for the PSI deposits (red line) and traditional (non-SG) deposits (blue line).

Nevertheless, SG programs have produced a large number of scientific publications. It is difficult to determine the exact numbers of publications for the worldwide SG as a whole, as there are no clear criteria that would allow classifying a particular scientific paper unambiguously as a “SG publication.” As of September, 2015, a total of 4419 scientific publications (the vast majority in peer-reviewed journals) have been reported by the five major initiatives – PSI (2280, [44]), RIKEN Structural Genomic/Proteomics Initiative (1010, [45]), SGC (891, [46]), the two NIAID centers (169, [47,48]), and EFI (69, [49]) Some structural and methodological publications represented very high impact as measured by the number of citations, e.g. [50–54].

We will focus our analysis on the publications from the PSI, which has set up a database of peer-reviewed publications produced and self-reported by its members ([44], [55]). As of August 24, 2015, it contained 2280 publications credited to the PSI. Of these, 1008 were methodological, 1058 structural, 95 functional, and 119 other (reviews, computational studies, etc.). The growth in the number of PSI publications is shown in Fig. 3. These publications have been cited over 93,000 times in other scientific publications (excluding datasets cited in the Data Citation Index). The average number of citations per publication was around 56 for methodological papers, and around 32 for structural ones (excluding structural notes), which are very respectable numbers.

Fig. 3.

Number of publications produced by the PSI, based on data from the PSI Publications Portal database as of August, 24, 2015.

An often used measure of a set of scientific publications (e.g. a journal) is “cites per document in an n-year window”), defined as the average number of citations received in a given year by papers from the set that were published within the n preceding years. For n=2, this metric is known as the impact factor (a trademark of Thomson Reuters). For the set of all PSI publications, the 2014 impact factor-like parameter was 8.6 (calculated using data from the PSI Publications database). If the set of all PSI publications were a journal, it would have ranked in the top 5% of the more than 1800 journals in the category of Biochemistry, Genetics and Molecular Biology according to data compiled by the SCImago Journal and Country Rank (http:/scimagojr.com).

Fig. 4 presents the average number of citations that were received in 2014 by PSI papers as a function of the date of publication. Generally, earlier papers are cited less often than the more recent papers. On average, papers published after 2004 attracted more than 5 citations in 2014. The distribution of N(x), the number of papers with x citations is presented on a log-log scatter plot in Fig. 5. Similarly as for other sets of scientific publications studied in the literature [56] it is characterized by a “heavy tail” of a small number of highly cited papers and the majority of papers with very few citations. Such distributions were suggested to follow a power law distribution, N(x) ~ x^-α, with α ~ 3 [56], especially in the region of the most highly cited papers [57,58]. The theoretical model that appears to fit well to the dataset of the PSI publications has a dual character [59], with a direct mechanism describing the papers with few citations, and an indirect mechanism (citing a paper based on citations already existing) describing the highly cited papers. The indirect mechanism yields a non-universal power-law distribution: publications that reach the tipping point where the indirect mechanism dominates tend to become even more highly cited and become “classics”—i.e. “the rich get richer.” Using the method of Clauset and coworkers [60] to analyze the dataset of PSI publications shown in Fig. 5, we found the tipping point to be around 69 citations. Above this tipping point, the citation distribution can be modeled by a power law, with an estimated value of α ~ 2.4. This region includes about 330 (15%) papers that have crossed the tipping point, and thus are potential “classics.”.

Fig. 4.

The average number of citations received in 2014 by PSI paper, as a function of the year of publication.

Fig. 5.

Distribution of the number of citations of PSI publications. The horizontal axis represents the number of citations, and the vertical axis represents number of publications with a given number of citations.

On the other end of the distribution, about 7.6% of PSI publications have not been cited at all to date. While this is potentially a disappointing number, it reflects the policy to concentrate on structures, not on functional experiments. Papers that are “structure notes,” which contain a description of the structure but little or no functional data, attract few citations. However, this problem is not unique to SG, as a significant fraction of papers published in most journals do not attract any citations. For example, Acta Crystallographica Section F publishes short communications on “any aspect of structural biology.” According to the SCImago Journal & Country Rank service, nearly 50% of papers published in this journal have no citations, even after 10 or more years.

Data, Databanks, Databases and Other Resources

From the outset, SG centers have emphasized data management in order to optimize the structure determination pipeline and disseminate results (including negative results). The primary mechanism for reporting experimental data in the PSI (and adopted by other SG centers) was the TargetDB database and its successor TargetTrack [61,55]. Many SG centers developed their own systems for aggregating and reporting experimental data to the TargetTrack system via XML-formatted reports. Many centers also developed laboratory information management systems (LIMS) that provide tools to track many aspects of the structure determination pipeline, from cloning to structure determination. Some of the LIMS originally developed by SG programs were made available to the broader researcher community, such as PiMS/xtalPiMS [62,63] and SESAME [64], and others, such as LabDB [49], are under active development in preparation for public release.

One measure of the completeness of experimental data related to determination of a protein structure is the number of missing data items (“NULLs”) in the headers of PDB deposits. The PDB file format was first specified in 1971 and included around 400 data items [65]. At the outset of the Protein Data Bank, an average PDB deposit header had over 90 data items set to a value of “NULL”. In subsequent years, due to progress in crystallographic software, the average number of NULLs in a given deposit decreased, falling to about 30 in 2006 (Fig. 6). However, since then the average number of NULLs has stayed relatively constant. As not all data items are relevant to all experiments, some NULLs will always be present.

Fig. 6.

The average number of missing data items (“NULLs”) for all PDB deposits using X-ray crystallography (including both traditional structural biology and SG) as a function of the year of deposition from 1972 to 2014 is shown in blue. The average number of NULLs in X-ray crystallography PDB files from the PSI initiative (2000–2014) is shown in red.

The quality of a crystallographic structure and the number of NULLs in the corresponding PDB deposit appear to be correlated. As shown in Fig. 7, for a wide range of resolutions, the average values of the R_free factor for “high-completion” deposits (defined as those with 20 NULLs or less) were significantly lower (p=0.05) than the average R_free values for “low-completion” deposits (defined as those with 50 or more NULLs). One possible explanation for the observed correlation is the “human factor,” namely the experience and knowledge of the crystallographer who determined each structure. An experienced crystallographer presumably is better informed of the procedures to ensure a structure is refined and validated in an optimal way, and also has better knowledge of (and appreciation for) what data items to collect and submit related to the process of solving the structure. Experienced crystallographers are also more likely to be aware of tools, such as PDB_EXTRACT, that can help collect relevant data from pertinent log files.

Fig. 7.

Average R_free by resolution bin (with a width of 0.2 Å) for X-ray crystallography structures deposited to PDB from January 2001 to March 2015, divided into two groups by the number of missing data items (“NULLs”) in the PDB file. The means for “high-completion” deposits (20 NULLs or less) are shown in blue, and the means for “low-completion” deposits (50 or more NULLs) are shown in red.

SG has been producing large amounts of data and the dissemination of this data has been a long-standing goal of many SG centers. In addition to releasing experimental information in resources such as the PDB and TargetTrack, another approach pursued by several SG initiatives was to find alternative mechanisms to disseminate information about structures that were not accompanied by peer-reviewed papers. Two of these are wiki-like annotation systems—The Open Protein Structure Annotation Network (TOPSAN) [66] and Proteopedia [67]—which permit scientists to collaboratively collect data and annotate structures online. SG centers have also developed novel ways to interactively render and propagate information about structures and their annotations. For example, the Structural Genomics Center (SGC) and the Center for Structural Genomics of Infectious Diseases (CSGID) have developed tools to generate interactive electronic presentations making use of ActiveICM technology (Molsoft LLC). In the case of CSGID these presentations are generated semi-automatically [68].

An important resource presenting SG data is the PSI Structural Biology Knowledgebase [55,69] (SBKB; http://sbkb.org), which makes available protocols created for the purpose of protein structure and function determination, papers highlighting new findings in structural biology work, and recent technology or methodology developments created by SG programs. Recently, a significant fraction of the X-ray diffraction data collected by the US-based structural genomics programs has been made publicly available by the new Integrated Resource for Reproducibility in Macromolecular Crystallography (http://proteindiffraction.org).

SG projects which pursued large-scale NMR studies, such as the RIKEN Structural Genomics/Proteomics Initiative and the Northeast Structural Genomics Consortium (NESGC) have collected large amounts of high-quality protein chemical shift information and other relevant data. RSGI has developed an automated system for annotation of datasets for resonance assignments [70] and their deposition to the BioMagResBank (BMRB) [71]. Some SG centers have been also depositing the original time-domain NMR spectra to BMRB, allowing for recalculation and reinterpretation of structural models. Overall, SG data have significantly increased the quality of the BMRB.

Resources developed by SG programs are not limited to providing data, but also include materials such as expression clones, purified proteins, and chemical probes. In particular, the PSI:Biology Materials Repository was established for storage, annotation, and distribution of genetic constructs designed by PSI centers. As of October 2014, it had nearly 90,000 PSI plasmids containing gene inserts and 130 empty vectors that have been made available to the community [72]. The Seattle Structural Genomics Center for Infectious Disease (SSGCID) has amassed a repository of over 3,000 purified proteins; about half of which have been distributed to requesters [73]. SGC has been developing a library of open access chemical probes – high-quality inhibitors for epigenetic proteins [74].

The emphasis of the SG projects on collecting as much experimental data as possible, focusing on data quality, and inclusion of not only positive but also negative results is of particular importance in the context of recent studies, showing that a significant fraction of experimental results published in the scientific literature cannot be reproduced by groups different from the original researchers [75]. Procedures used by laboratories involved in SG, relying on databases [68] and related tools have been successfully used in identifying problems that may affect reproducibility of experimental results, e.g. mislabeling of a protein sample, contamination with a foreign protein, and other issues [76]. These approaches to detect and correct potential reproducibility problems are directly applicable in many other areas of biomedical research.

Structure determination technologies

To meet their proposed goals, SG centers have had to develop streamlined pipelines expediting processes from target selection, through protein production and crystallization, to structure determination and data management. SG centers have developed techniques for high-throughput cloning, protein expression and purification [77], tools for crystal production, and numerous software packages for X-ray and NMR data collection, structure determination and refinement.

A single protocol can be applied to generate multiple and varied genetic constructs to screen for one that most efficiently expresses the desired protein. SG has improved systems for semiautomated expression and purification of proteins of interest that allow for streamlined testing and searching for the best conditions for large scale production of protein for research applications. The requirement for large quantities of active protein complexes for structural and functional studies was met by the development of efficient pipelines that incorporated numerous technical advances, including tools for co-expression and co-purification of protein partners in E. coli cells [77], as well as eukaryotic expression techniques [78]. Methods for testing the stability of proteins for crystallization experiments have been developed and optimized [79]. Based on their experience crystallizing target proteins, SG centers have developed their own crystallization screens (for example the MCSG suite and JCSG+) [80,81]. The compositions of these screens have been optimized after analysis of all crystallization conditions applied to hundreds of protein targets, and identification of the conditions that produced the most crystals. This would not be possible without extensive data on a large number of targets including both successful and failed crystallographic experiments. The systems developed by SG centers for systematic, wide-scale collection of experimental data described above were crucial for this research.

Parallel to the development of methodologies for efficient protein production and crystallization, bioinformatics tools were also developed for sequence analysis and predicting protein stability, folding, and crystallizability. SG provided bioinformatics servers for the rational design of protein constructs [82] as well as predicting protein folds, interfaces, and protein-protein interactions [83].

SG programs have been a leader in the development of new methodology for structure determination by X-ray crystallography [84,85]. In this century, nearly half of all PDB deposits determined by the single wavelength anomalous dispersion (SAD) method were submitted by SG centers. The large number of diverse protein crystals produced by SG programs created the need for efficient methods of screening for the best diffracting crystal. The unprecedented large number of projects led to the development of programs for fast (albeit not always efficient) data collection and structure determination [51,86–88] and “pushed” protein crystallography synchrotrons to improve both the hardware and the software installed on beamline stations. For example, equipment for sample handling and data collection became more advanced, including the development of robotic equipment for sample mounting [89] and crystal alignment [90]. Microfocus beamlines were developed to permit effective data collection on very small crystals [91]. The technical advances spurred by SG and close proximity of advanced wet laboratories created a basis for interactive crystallography [92]. On many beamlines, data can be collected remotely; however, data quality may occasionally suffer on beamlines that do not allow for remote sample annealing [93,94].

SG projects have also improved technologies for high-throughput NMR data collection and analysis [95]. RSGI has built a robotized NMR pipeline, including both cell-based and cell-free protein expression systems and sample screening [96]. RSGI not only pioneered cell free expression systems but also conducted many workshop to popularize this technique. NESG has developed a protein NMR pipeline, utilizing efficient screening with micro-cryoprobes [97]. This has led to creation of a unique resource that provides experimental data for proteins whose structures have been solved by both NMR and X-ray crystallography [98]. The resource will likely assist further developments of NMR structure determination methods.

Function determination and function determination technologies

Recently, many SG centers pursued the development of methodologies to ascertain a protein’s function, and therefore established a number of computational and bioinformatics strategies to predict possible functions of proteins based on sequence-structure or sequence-function relationships [99,100]. The Fold and Function Assignment Server (FFAS) [101,102], which uses profile-profile algorithms for prediction of fold and function based on sequence, has been consistently ranked as one of the best fold recognition methods.

The mission of the third phase of PSI, i.e. PSI:Biology, was to determine the structures of proteins of biomedical importance and characterize their biological function. Over 100 PSI publications pertain to discovery of protein function (Fig. 3). To understand the structure-function relationships, new methods for functional analysis of proteins with unknown function or methods for analysis of unknown substrates for a protein with determined function have been developed [103,104]. In parallel, tools for protein function prediction via bioinformatics and ligand docking were created to leverage the continuously growing number of 3D structures [105].

The EFI has been pursuing a systematic program of “discovery of in vitro enzymatic and in vivo metabolic and physiological functions of unknown enzymes.” It has focused on large, functionally diverse enzyme superfamilies (amidohydrolases, enolases, glutathione transferases, haloacid dehalogenases and isoprenoid sythethases type I). These efforts contributed to the development of the Structure Function Linkage Database (SFLD) [106], which classifies enzymes by relating chemical/functional features to specific characteristics of sequence and/or structure. Activities of the EFI have resulted in over 60 publications [49].

Conclusions

By providing new methodologies, technologies, training, and knowledge that make determination of difficult and novel structures possible, SG has had a significant impact on structural biology and biomedical sciences. The pipelines for structure determination via X-ray crystallography and NMR has been are a unique resource for the biomedical community worldwide. Before its first quindecennial, SG centers determined more than 13,500 structures. Some members of the crystallographic community may suspect that such productivity is only possible for “low hanging fruit.” In reality, many of these structures were extremely difficult and required the development of advanced molecular biology and structure determination techniques. For that reason, the expertise of the people that built the PSI pipeline has been of benefit to many scientists worldwide for determination of ultra-difficult, biologically important structures. This collected expertise in structure determination as well as other areas of biomedical research (protein production, SAXS, NMR, ligand screening, etc.) has resulted in collaborations with hundreds of laboratories around the world. On the other hand, the initial vision of SG, namely the ability to determine an arbitrary protein structures in an automatic way, is still on the horizon. In particular, coverage of many important pathogens and of the human microbiome is still poor.

At the moment, in light of the extinction of the PSI centers, structural genomics efforts are reduced. It is, however, too early to write an epitaph for SG. Some SG programs continue, most notably the Structural Genomics Consortium and the two programs funded by NIAID. These programs also provide structure determination services to biomedical community. Our observations show that the SG approach is somewhat continued by many individual laboratories, but without taking advantage the economy of the scale, and for obvious reasons, in slightly disorganized fashion. Smaller labs usually cannot afford expensive robotic workstations and rarely can maintain complex pipelines. Unfortunately, some very important projects have been significantly hindered by the end of PSI. We mention only two below—protein production facilities and the development of sophisticated, comprehensive databases that track protein production and related information.

Several centers have developed sophisticated protein production “factories” that could potentially serve as resources for the whole biomedical community [78,107]. In particular, the New York Structural Genomics Research Consortium (NYSGRC) has developed a system for large-scale production of proteins from eukaryotic and anaerobic organisms [108,109]. All details of protein production are harvested into a sophisticated database that is integrated with structural and functional databases into one universal system. The integration of many processes through data management and databases is a key factor for reproducibility of scientific results. Data management in a large modern laboratory has become paramount for coordinating and tracking the vast amount of data generated across multiple experiments, timeframes and centers—not to mention the potential for data mining to extract even more useful and interesting information. Successful data management requires a system with a well-planned, cohesive, and flexible framework. How to best achieve this coordination and level of detail is currently being addressed in different ways, but the measure of success comes back to “data in, information out.” Clearly, gathering details on millions of experiments on thousands of proteins and making them publically available for analysis—even after the projects themselves have ended—may turn out be one of the most important benefits of SG.

The closure of the PSI may result not only in the abandonment of most of the data collected during the 15 years of the initiative, but it may also affect the development of data management tools. The most sophisticated databases developed for PSI have one very uncommon “feature”: the data are harvested automatically and therefore cannot be “censored” by experimenters. We define “censorship” in this case as an omission of unsuccessful experiments, mainly because the researcher did not see the value of negative results. The inclusion of negative results greatly improves the conversion of data into biomedically useful information.

Converting data into information is a general difficulty of modern science. Researchers are swamped in experimental data and extraction of useful information is quite often a Sisyphean task. Addressing this task effectively requires either very substantial manual labor or the implementation of “knowledge-based systems,” with comprehensive tools for efficiently summarizing and mining experimental data, and in some cases, implementation of machine learning methods. Processing structural information, particularly when combined with functional, experimental, and sequential data, while maintaining the context of interaction networks with other bio-macromolecules or bioactive chemical compounds, increasingly requires the use of Big Data paradigms for effective data management, as well as for checking data integrity and accuracy. The absence of documentation regarding which protein preparations were used in functional studies may partly be responsible for reports that a significant number of biomedical experiments cannot be reproduced in other laboratories, even though the changes between protocols seem at first glance to be insignificant [110]. According to a recent study, the annual cost of irreproducibility in life science amounts to a staggering sum of $28 billion for the US alone [111]. By emphasizing and enabling comprehensive data management, SG programs have been directly and indirectly promoting improvement in the reproducibility of research results in structural biology and in biomedical sciences in general. In our opinion, it is likely that the indirect economic benefits of SG stemming only from reducing the irreproducibility in biomedical research have already surpassed the $2 billion investment.