Continuous Validation Across Macromolecular Structure Determination Process

Abstract

The overall quality of the experimentally determined structures contained in the PDB is exceptionally high, mainly due to the continuous improvement of model building and structural validation programs. Improving reproducibility on a large scale requires expanding the concept of validation in structural biology and all other disciplines to include a broader framework that encompasses the entire project. A successful approach to science requires diligent attention to detail and a focus on the future. An earnest commitment to data availability and reuse is essential for scientific progress, be that by human minds or artificial intelligence.

1. Introduction

The Protein Data Bank (PDB)¹⁾ was formed over 50 years ago and initially contained only seven macromolecular structures. The PDB founders realized that access to macromolecular models is essential for crystallographers, students, and researchers who might use structural information for their research. Initially, every deposit contained only the coordinates of the atoms. Even then, every new structure was carefully checked to see whether the 3-D model agreed with known chemical and physical properties. Since 2007, every submission to the PDB has had two additional components: 1) information about the sample (crystal in X-ray crystallography), a rough description of the experimental setup, and other metadata (header), and 2) intensity amplitudes (usually called structure factors) of all diffraction spots measured during the experiment. The deposit header also contains additional information like authors, connections to other databases, software used to determine the structure, etc. These three components allow for calculating the electron density map and checking whether the macromolecule model, including ligands, nucleic acids, and solvent, agrees with experimental data. This checking procedure is called validation of the macromolecular model and was usually performed by the experimenter at the end of the structure determination process. The PDB also routinely performs validation on all deposited structures.²⁾ Moreover, scientists who find disagreement between their experiments and PDB deposits can perform validations for themselves. Sometimes their validation triggers a re-refinement and leads to the deposition of a new, improved model to the PDB.³⁾ Occasionally, new biomedical interpretations arise from improved structural models.⁴⁾

2. Project validation

The classical procedure described in the introduction is only a part of the validation or series of validations that must be performed during the entire structure determination process. The first step is project validation. Inexperienced scientists or graduate students should start with project validation, i.e., they should answer a series of the following questions. Is my project important enough to pursue? Do I have access to the resources necessary to complete my project? Do I have somebody to ask for help in case of problems? If all of the answers are positive, scientists need to check whether the project has already been done by somebody else, but checking this once is not enough. There are many cases where researchers depositing a structure found out they had been scooped months before because they failed to check the PDB or published communications about structural progress consistently. One of the most common ways to track research progress is PubMed (https://pubmed.ncbi.nlm.nih.gov/) and the PDB. Since the publication titles tend to be misleading or insufficient, a more in-depth analysis of keywords, MeSH terms, or abstracts is often required. Less than 10% of PDB deposit titles have at least 80% similarity (calculated using the Gestalt approach) to the title of their “primary citation,” i.e., the journal article about the structure. Despite more than 200,000 PDB deposits and over 35 million abstracts in PubMed, regular querying of these resources are very fast. Recently, anybody can deposit manuscripts in the two most popular archives, bioRxiv (https://www.biorxiv.org/) and medRxiv (https://www.medrxiv.org/). These manuscripts are not peer-reviewed and are usually not reported in PubMed. Moreover, two features of these archives occasionally turn into their flaws: i) authors may have published many versions and often differ substantially from each other; ii) only 70% of the manuscripts are eventually published in peer-reviewed journals. In addition, it is worth checking Web of Science (https://www.webofscience.com/) or Google Scholar (https://scholar.google.com/) as book chapters are not reported in PubMed. Most of these resources have Application Programming Interface (API) that allows querying them in a reproducible and efficient manner.

3. Validation of sample production

Currently, the determination of structure can take hours, or even just minutes, when an optimal diffraction experiment is performed on a well-diffracting single crystal. The production of good samples is still an art; it usually requires many attempts to obtain well-diffracting crystals (Fig.1). The first step is the expression and purification of protein. Typically, protein expression requires using a host cell, such as a bacteria, yeast, or mammalian cells, to produce the protein. Once the protein is expressed, it undergoes purification, which is a multistep process. The number of steps can vary depending on the specific protein. However, several common steps are included in a typical protein purification protocol.^5),6)These steps include solubilization, pre-purification, chromatography, buffer exchange, and concentration.

Fig.1.

Multiple clones, expressions, purifications, crystallization drops, crystals and diffraction experiments for target IDP91778. Darker colors indicate pathway that resulted in structure determination. Screenshot from CSGID data management system.

Production of pure and active protein may require optimizing the expression conditions and carefully controlled purification steps performed according to the protocol to avoid unforeseen mistakes.⁷⁾ Once the procedure to produce the pure and active protein has been found, it is essential to use the same protocol for all subsequent experiments. Ideally, the same batch of protein should be used for all planned experiments. Changing batches of proteins between experiments may introduce variations affecting results and their interpretations. For example, if two different batches of protein are used for two separate experiments, and the obtained results are contradictory, it may not be clear whether these differences result from the differences in protein batches or slightly different experimental conditions. Afterward, it is hard to conclude the correctness of either of these results. Using the same batch of the protein across all planned experiments ensures that obtained results can be compared against each other, and any discrepancies between them are the effect of the experimental conditions. When using the same batch of proteins is not possible, it is essential to take all necessary steps to minimize potential confounding effects and include proper experimental controls. Even slight deviation from the protocol may lead to the production of some contaminants and, in extreme cases, even a wrong protein. It is necessary to run a set of control experiments for all batches and ensure they lead to the same results.

Quality control procedures should be applied for each protein expression and purification step. Appling quality control procedures maximize the experiment’s credibility and reproducibility. Several methods can be used to validate protein purity after the purification procedure. One of the most popular and cost-effective methods to analyze protein is SDS-PAGE, which separates protein based on size and charge.^8),9) Purified protein should appear on the SDS-PAGE gel as a single band, indicating that it is pure. The presence of other bands will suggest that there are other proteins/ contaminates in the tested sample.

Immunoblotting¹⁰⁾ can be used in addition to SDS-PAGE. Another method is analytical size exclusion chromatography (SEC)¹¹⁾ to confirm a protein’s size or determine the oligomeric state of a protein. SEC is frequently used in conjunction with other analytical techniques, such as mass spectrometry, to confirm the identity and purity of the proteins.¹²⁾ Biochemical activity assays are frequently used to measure the activity of a purified protein. Circular dichroism (CD) spectroscopy¹³⁾ and nuclear magnetic resonance (NMR) spectroscopy¹⁴⁾ can help determine the quality of the sample.

Finally, all these and other validation methods are not always available. However, one should use as many techniques as possible to fully characterize the protein sample while optimizing and establishing a purification protocol. For example, mass spectrometry is exact and sensitive and can confirm the sample’s identity and purity, but it is often skipped because it is expensive. Unfortunately, sometimes not using the best technique can be much more expensive and time-consuming when explaining inconsistent results.¹⁵⁾

The presence of affinity tags or expression vector artifacts can completely alter the results of structural and functional studies.¹⁶⁾ His-tags and other fusion partners are considered benign because they usually do not affect the activity of the protein, but the possibility of complications should always be considered. This is especially true as the conditions get further away from physiological conditions, like the high concentrations necessary for crystallization. We even witnessed a case where a His-tag form occupies the active site of a neighboring protein.¹⁷⁾ Therefore, the inability to remove a supposedly cleavable His-tag indicates problems that should be carefully studied because it can indicate improper folding.

The ultimate validation of the crystallization process is a diffraction experiment. Beautiful crystals may not diffract, and some turn out to be salt or another small molecule in the crystallization drop. Sometimes excellent crystals are destroyed during the harvesting process or flash cooling. Like all other lab activities, every step of the crystallization process should be carefully recorded in a database system.^18)-20) Complete and accurate data maintenance is part of the validation process in modern science.^19),21),22) Data cannot be on removable disks sitting somewhere in the lab on shelves or scattered over hundreds of spreadsheets. They have to be accessible with a click of a button. At the start of the Covid-19 pandemic, dexamethasone was the only medication that increased survival rates for severe infections. When we checked our reagent database for dexamethasone, we found that we had already used dexamethasone in the past. Further, we discovered that we had collected diffraction data of dexamethasone bound to albumin in 2011. Because all the pertinent information had been documented adequately in our lab’s data management system, we could complete this structure rapidly. We combined it with initial hospital data to explain the role of albumin in Covid-19 treatment and drug delivery.²³⁾ This is a great example of the benefits that a modern data management system^19),20) can bring to biomedical sciences.

4. Validation of diffraction experiment

Fast Pixel Array Detectors (PAD) allow the collection of complete diffraction data sets in minutes. Some synchrotron stations offer automatic data reduction, which is slightly slower than data collection. In those cases, validation is sometimes skipped, and researchers occasionally later realize that they collected terabytes of unusable data. Below, we describe best practices used by the Minor Lab that led to over 450 macromolecular structures deposited to PDB that resulted in almost 100 publications. First, we prefer to be present during data collection. We check the temperature of the nitrogen stream at the place of the sample. The examination of PDB shows that most people report 100 K, which is the nominal temperature. Our experience shows that some synchrotron beamlines do not check the temperature routinely, and we once found the actual temperature was 140 K. Higher temperatures result in a faster sample decay.²⁴⁾ The distribution of the reported temperature of the crystal during diffraction experiments shows that these data, to a large extent, are taken from the sky (Fig.2).

Fig.2.

The distribution of reported crystal temperatures during diffraction experiments. Over 127,000 deposits reported 100 K.

Before attempting to collect the entire data set, we collect two groups of 2–3 images 90 degrees apart to check whether we are using a single, well-diffracting crystal and check the crystal’s mosaicity. If this is the case, we collect the entire data set using a sweeping step of 1/3–1/4 of crystal mosaicity. We collect data on the first reasonably diffracting crystal as several challenging projects in the Minor Lab produced suboptimal crystals. Even if a mounting robot can, in theory, reproducibly reposition the crystal, accidents and complications have destroyed crystals that were not great but turned out to be the best we had. In such cases of a suboptimal sample, the processing has to be exemplary,²⁵⁾ and data reduction should be done on the fly. It is incredibly disappointing to get back and realize that you should have collected data better. Data collection depends on the type of experiment. For molecular replacement experiments, one should check whether the low-resolution data are as complete as possible and whether there are overloads. When data are collected for refinement, one should collect as complete as possible high-resolution data, and the number of rejected outliers should be below 0.5%. If the number of rejections is higher than 0.5%, it is essential to check crystal decay by checking the increase of B factors during the data collection or checking the experimental setup performance.^26)-28) It is strongly recommended to apply a correction for absorption and radiation decay. Sometimes it is necessary to remove frames strongly affected by radiation decay. The experimenter should remember that data collection is the LAST experiment in the structure determination process. During processing, there are ways to improve data slightly, but there is no substitution for a well-done experiment.

5. Validation of macromolecular structure

Iterative model building,^29)-33) refinement,^34)-37) and validation recently became a single process. The solvent molecules should be added when the model is somewhat advanced.³⁸⁾ This process is reasonably automated but not fully automatic. At that point, validation is relatively straightforward. The PDB produces an excellent structure validation report which should be carefully examined. It may be necessary to return to refinement and correct problems. It is beneficial to read carefully a paper describing the state-of-the-art ³⁸⁾ refinement procedure. Less experienced scientists may also submit their model to use PDB-REDO ^39),40) and examine other areas that can be improved. The validation of the macromolecular part of every deposit is very well advanced except for regions where the electron density map is very weak or just not visible. One should realize that invisible regions of maps can indicate flexible parts of the protein which can be functional. Thus the variability of resolution observed in Cryo-EM may not be a feature of the methodology but a manifestation of the flexible parts.

6. Ligands validation

Over 80% of PDB deposits include at least one small molecule (ligand). Studies of ligands bound to biological macromolecules are essential for modern drug discovery.^41)-44) The validation of ligands is still challenging, and many X-ray deposits contain misidentified and/or poorly refined ligands.^45),46) The use of an artificial intelligence (AI) approach for ligand identifications recently became successful ⁴⁷⁾ but not as successful as we would like to see. However, AI only works well when it starts with a high-quality, large training set. So, the tools for validation and curation process are critical to making AI perform well. For the foreseeable future, these training sets must be processed by tools that must be reviewed to ensure their quality. This cannot be done by simply removing all non-ideal features because binding sites in proteins often deviate from ideal geometry due to constraints imposed by the conformation of the macromolecule; therefore, some deviations from ideality are acceptable. Some of these abnormalities are cases where a single molecule from the crystallization conditions has been incorrectly modeled as a dual conformation of a different ligand. ^47),48)

It would be easy to think that the validation of metals should be more straightforward than the validation of small molecule compounds, but those who think this way would be surprised. Metal ions play various roles in macromolecules, from providing structural integrity to active participants in catalysis, where the metal ions must interact with the macromolecule and other small molecules. Additionally, the modeling of metal ion binding sites is at the interface of organic and inorganic chemistry, often just outside the experience of many researchers. The most common mistake is the misidentification of the metal ion. Misidentification will result in poor refinement and low quality electron density maps. Errors in modeling metal ion binding sites may critically affect the conclusions drawn in the publication and trigger an avalanche of ultimately unjustified speculations in subsequent publications and research. Several experimental techniques allow the identification of the metal ion bound to a protein. The ultimate is the diffraction experiment below and above the metal’s absorption edge.⁴⁹⁾ There are others, like PIXE.⁵⁰⁾ CheckMyMetal (CMM)⁵¹⁾ analyze parameters of the binding site like ligand distances, the geometry of coordinating sphere, valence, and the metal ion–environment B-factor. CMM, especially its new version, became a potent tool for identifying and refining metal ion binding sites.^52)-54)

One of the main sources of errors is a human cognitive bias. Many researchers tacitly assume that the metal ion at the binding site comes from the solution used in crystal soaking or co-crystallization or is simply an integral component of the studied protein. However, it is essential to consider all ions involved in every step, i.e., protein expression, purification, crystallization, etc. Below is an example of zinc-binding protein zinT (PDB id 5XM5). The deposit includes six zinc ions bound to both chains. Two binding sites, A201 and B202, have a surprisingly low occupancy value for the zinc ions (0.39 and 0.37, respectively). If the occupancy of these ions is set to one, these two metal sites have a significant negative density in the difference map (Fig.3a-b). Moreover, these two ions have the Valence⁵⁵⁾ marked in CMM as Dubious, much closer to one than expected Valence of two for the divalent Zn²⁺ ion (Table 1). The analysis of crystallization conditions shows the presence of Na⁺ in the buffer. Refining the structure within CMM after replacing Zn²⁺ with Na⁺ and setting the occupancy to one renders the reasonable values of all the validation parameters (Fig.3c and Table 1).

Fig.3.

Different models of the metal binding site A201 in PDB 5XM5 deposit. Only the difference map is shown. (a) the binding site with Zn²⁺ (occupancy: 0.39), (b) the binding site with Zn²⁺ (occupancy: 1.0), (c) the binding site with Na⁺ (occupancy: 1.0).

Table 1.

Validation results of the A201 and B202 metal ion binding sites in the 5XM5 PDB deposit.

Residue		Metal	Occupancy	B-factor (env.)	Atom Contacts	Valence	nVECSUM	Geometry	gRMSD	Vacancy
A201	Original	Zn	0.39	35.6 (31.1)	O 5	0.9	0.14	Octahedral	13.5°	16%
	Re-refined	Na	1	28.8 (32.2)	O6	1.3	0.052	Octahedral	12°	0
B202	Original	Zn	0.37	25.1 (25.9)	O 5	0.9	0.13	Octahedral	9.2°	16%
	Re-refined	Na	1	21.6 (24.7)	O5	1.2	0.14	Octahedral	9.6°	16%

7. Conclusions

Within crystallography, validation has evolved from a single task performed at the end of the structural refinement into an iterative process occurring throughout model building and refinement. This has resulted in a continuous improvement of the quality of structural models within the PDB. Although some aspects of macromolecular validation still need improvement, the overall state of the field is strong. AI is rapidly evolving and changing our expectations about the quality of theoretical models. Still, for the sake of the accuracy of their predictions, AI resources need high-quality training sets, and predicting binding site details requires validated sets of experimentally determined ligands. Identifying anomalies within the PDB is part of the worthiness of validation software, and powerful validation platforms are still needed.

It is now time for researchers to realize that validation needs to be pervasive throughout all steps of the project, and it begins with honest project evaluation. The most successful labs have already incorporated this concept. It has recently been noted that scientific progress would greatly benefit from a “continuous and ubiquitous” approach to FAIR principles of data management. ⁵⁶⁾ All of our scientific endeavors would equally benefit from a continuous and ubiquitous approach to validation. Experiments should not be considered complete until the results and data are available for others to use and sometimes verify. Scientists should be proud to have other researchers reuse their data and not be offended if they sometimes get better results. Science would not exist without validation, and it is our job to keep it in mind.

Biographies

Vanessa BIJAK, PhD

Department of Molecular Physiology and Biological Physics, University of Virginia, Charlottesville 22908, USA

Biochemistry and Biophysics

e-mail: vanessa@iwonka.med.virginia.edu

https://www.linkedin.com/in/vanessa-bijak/

Michal GUCWA

Department of Molecular Physiology and Biological Physics, University of Virginia, Charlottesville 22908, USA

Department of Computational Biophysics and Bioinformatics, Jagiellonian University, Krakow, Poland

Biochemistry and Biophysics

e-mail: michal_g@iwonka.med.virginia.edu

https://minorlab.org/person/michal_g/

Joanna LENKIEWICZ

Department of Molecular Physiology and Biological Physics, University of Virginia, Charlottesville 22908

Computer Science

e-mail: joanna_len@iwonka.med.virginia.edu

https://www.linkedin.com/in/joanna-lenkiewicz

Krzysztof MURZYN, PhD

Department of Computational Biophysics and Bioinformatics, Faculty of Biochemistry, Biophysics and Biotechnology

Jagiellonian University, Krakow, Poland Biophysics and Structural Bioinformatics

e-mail: krzysztof.murzyn@uj.edu.pl

https://www.researchgate.net/profile/Krzysztof-Murzyn

David COOPER, PhD

Department of Molecular Physiology and Biological

Physics, University of Virginia, Charlottesville 22908 Structural Biology

e-mail: dcoop@virginia.edu

https://minorlab.org/person/dcoop/

Wladek MINOR, PhD

Department of Molecular Physiology and Biological Physics, University of Virginia, Charlottesville 22908

Structural Biology

e-mail: wladek@iwonka.med.virginia.edu

https://en.wikipedia.org/wiki/Wladek_Minor