How to Triage PAINS-Full Research

Abstract

Nonspecific bioactivity and assay artifacts have gained increasing attention in recent years. This focus has arisen primarily from the publication of a set of chemical substructures, termed pan assay interference compounds (PAINS), which are associated with promiscuous bioactivity and assay interference in real and virtual high-throughput screening (HTS) campaigns. Despite an increasing awareness in the HTS and medicinal chemistry communities about the liabilities of these compounds, articles featuring PAINS and PAINS-like compounds are still being published. In this perspective, we describe some of the factors we believe are driving this resource-sapping trend. We also provide what we hope are helpful insights that may lead to the earlier recognition of these generally nontranslatable compounds, thus preventing the propagation of PAINS-full costly research.

Five Years of Pains

It has been 5 years since the original work on pan assay interference compounds (PAINS) was published by Baell and Holloway.¹ This article seems to have resonated with many in the medicinal chemistry and high-throughput screening (HTS) communities, having been cited over 350 times in the scientific literature to date. PAINS and the intimately related concepts of assay artifacts, false positives, and promiscuous compounds have received considerable coverage.² It has been our experience that the majority of our colleagues are now aware of PAINS and other poorly tractable chemical matter. Importantly, there have been several recent articles describing various chemical mechanisms of assay interference from both PAINS and other related compounds.^3–8 However, has anything really changed? Based on the number of published and submitted articles that we (and others) observe containing PAINS and indiscriminately reactive compounds, the answer is probably a resounding no.^9,10 Herein, we propose some simple rules to help researchers, reviewers, editors, and other members of the drug discovery communities triage time- and resource-wasting interference and/or nonspecific bioactive compounds from the literature. It is hoped that such efforts will direct resources to more promising endeavors in the understanding of biological processes and the treatment of human diseases.

We define PAINS as compounds that are recognized by the substructure filters reported by the Baell and Holloway article.¹ The activities of PAINS are typically caused by their reactivity rather than noncovalent binding and they typically interact nonspecifically with proteins in a high percentage of bioassays. This definition is important because the term PAINS is sometimes used interchangeably with other related terms such as false positives, artifacts, and promiscuous compounds. PAINS substructure filters will not recognize every compound that could be an interference compound or frequent hitter because they were created using a well-designed library of screening compounds. In other words, some reactive compounds were not in the screening library that was used to originally define the PAINS substructures. Additionally, only a single assay method (i.e., AlphaScreen) and HTS campaigns against six targets were used to define substructure promiscuity. Since not every PAINS substructure has a fully characterized or even singular mechanism of interference, some substructures could have been enriched by the nature of the assay method or target selection in the original PAINS research. Other screening methods, for example, those based on fluorescence or cellular phenotypes, might have led to the creation of a different set of PAINS substructures. However, our experience and other anecdotal evidence from many of our colleagues in industrial and academic HTS strongly support the robustness and general use of these filters as a first-pass strategy to recognize problematic structural classes that will turn up using multiple assay formats.

We have proposed further classes of compounds that also have a high potential for assay interference and that are not flagged as PAINS.³ These classes of compounds show structure–interference relationships (SIRs) that suggest that some members of these compound classes could be reasonable starting points for further investigations against certain disease targets. In the case of these specific compound classes, the SIRs and underlying chemical mechanisms of assay interference were determined using multiple experimental approaches and by testing both active and inactive chemical analogs. However, such compounds should always be approached with caution because the liabilities of basing a project on noninterfering PAINS or PAINS-like compounds are not yet known. Therefore, it is our opinion that all screening actives containing PAINS substructures or potentially reactive functionality must be approached with skepticism and only reported if a fearless exploration of chemical interference is performed and if experimental evidence confirms any claims of useful or specific biological activity.

The Problem with Pains

PAINS are prevalent in the literature. Hundreds of publications and patents claim them and other assay interference compounds as real actives with useful properties.⁹ Despite some exciting claims, they typically turn out to be nonprogressive for any useful purpose. This suggests that follow-up on these compounds may have cost the research community millions of dollars on dead-end research and hundreds, if not thousands, of hours of research time. Worse yet, their inadvertent publication in the scientific literature suggests to other researchers that they are worthy of follow-up. Thus, a PAINS-full self-perpetuating cycle begins. These chemical con artists are reported in subsequent publications and made commercially available. This often leads to their unquestioned use in other medicinal chemistry studies, signaling pathway analyses, and in the development of flawed in silico models and pharmacophores. A similar situation has recently been described in the broad use of so-called chemical probes.¹¹ Surprisingly, many articles that report PAINS do not even refer to articles in which these and closely related chemical structures have been published previously as though this literature precedence does not exist. In genetic screens, it is routine to check for previously described functions of over- or underexpressed genes. We advise that chemists and screeners do the same with their compounds.

Articles touting PAINS as potential chemical probes or therapeutic lead compounds continue to be published for a variety of reasons, but at least five primary reasons were recently outlined in an American Chemical Society (ACS) webinar facilitated by Jonathan Baell.¹² The five primary reasons cited were (1) ignorance of the PAINS structures, (2) lack of medicinal chemistry expertise, (3) budgetary (lack of a quality screening library or the screening of too small a library because of HTS costs), (4) pressure to publish data into which money has been sunk, and (5) dynamics of some academic teams in which egos can get in the way of a reasoned evaluation of scientific data. We support this analysis and would like to offer our additional perspectives on this state of affairs.

Ignorance of the PAINS substructures can no longer be excused. Many cases of PAINS-full research could be avoided simply by knowledge of the top PAINS substructures (Fig. 1). Moreover, there are now free services available to flag compounds with PAINS substructures.^13,14 The interference potential of compounds not recognized as PAINS can be gauged using data freely available from The PubChem Project (pubchem.ncbi.nlm.nih.gov),¹⁵ and BADAPPLE (bit.ly/1EoRP2l)^16,17 offers a convenient window into a curated version of public activity data from BARD¹⁸ (BioAssay Research Database; bard.nih.gov). As an example of the utility of these kinds of data, note that five of the six most promiscuous compounds in PubChem are flagged as PAINS (Fig. 2).¹⁹ The sixth compound is not identified as PAINS because it most likely was not in the library from which the PAINS filters were devised. It is also useful to note that even the most acute PAINS do not have activity in every assay or against every target. Given this, a scientist who understands the potential reactivity and interference mechanisms of compounds is a critical member of any serious project team. This is a standard operating procedure in the pharmaceutical industry. Medicinal chemists and screening scientists with experience in HTS triage are trained to recognize reactive functional groups and other compound features that might confound assays. Along with experts in assay technologies, they also know that many assays will have a high rate of false positives and may only generate a handful of compounds worthy of the commitment of further time and resources.^20,21

Fig. 1.

Chemical substructures of the worst pan assay interference compounds (PAINS) and potential interference compounds. These substructures form part of the full chemical structure of PAINS and suggest a high potential for that compound to interfere with biological assays. Not every compound containing one of these substructures will be an interference compound, but proceed carefully if an active compound contains one of these substructures.

Fig. 2.

Six promiscuous compounds from PubChem. Some of the most promiscuous bioactive compounds from the PubChem database (pubchem.ncbi.nlm.nih.gov). Note that these PAINS examples are not active in all assays or against all targets. However, their off-target activities do not bode well for their development as specific chemical probes or target-based therapeutics.

The experimental reality is that HTS (or virtual HTS [vHTS], followed by assays) will not always turn up active compounds that are publication worthy or that can progress into useful probes, drugs, or other products. This is as true in industry as it is in academia and needs to be recognized as one of the potential outcomes of HTS.²² Contingency plans should be in place to deal with this eventuality. It is certainly reasonable to start with small screening libraries (<50,000 compounds) as libraries of this size are relatively less expensive to screen, and these screens can be used to demonstrate assay robustness, find true known actives, and obtain preliminary results to obtain funding for full-deck screening. The realities of budget and time constraints in modern drug discovery (especially in academia) often make HTS against a target a one-shot deal. That is, once the HTS has been completed, there is often no going back to repeat the primary HTS assay. Since the success of drug or probe discovery will be dependent on the chemical matter identified in this initial HTS, we believe it is best to rigorously optimize and validate the screening approach on small library subsets, and then screen as large and high quality a chemical library as possible.

With respect to the pressure to publish, a principal investigator (PI) invested in the results of an HTS should have a clear publication strategy mapped out with his or her management team. Venturing into translational medicine is risky from this standpoint and department and academic administrators ought to acknowledge this risk in the review of outcomes related to academic advancement.²³ Some people cannot handle the truth, but it is better to catch PAINS early than to spend time and money on dead-end compounds.

Team dynamics are perhaps the most difficult to address and can be especially challenging in academia. Drug discovery by HTS requires expertise in assay development, medicinal chemistry, pharmacology, and biology. However, it also requires teamwork. We believe the best collaborations are those in which the PIs respect each other and the expertise of each team member. If this does not come naturally, it may most effectively be enforced on collaborative projects by establishing well-defined funding milestones (decision points) that are PI independent and that even might reward a fail-fast outcome.²³ That is, a well-conceived and executed HTS should be rewarded regardless of its outcome, although how this attitude can make its way into the evaluation processes of academic administrators still remains a puzzle. Review and evaluation of such projects by external experts is a strategy worthy of consideration.

Therefore, the reasons offered for the continued publication of the worst PAINS can no longer be readily excused. However, it appears that no amount of education or publication can deter the uninitiated from following up on these generally useless compounds. If this is the reality, what then can the scientific community do to break the cycle of PAINS-full research?

Most importantly, researchers, reviewers, editors, and readers of the literature must learn how to recognize reactive moieties, PAINS, and other interference compounds. PAINS have increasingly well-described liabilities. Articles and proposals that present these compounds should be evaluated carefully and the burden of proof that the compounds are, or could be, useful should reside with the PIs or authors. This may appear to be unfair or draconian. However, the experiments that would likely weed out most PAINS, such as demonstrating SAR, ruling out assay interference and known chemical liabilities, confirming the purity and identity of compounds, determining selectivity, and showing evidence of meaningful target engagement, should already be expected as best practices in early drug or probe discovery. This is simply good scientific practice.

Researchers

Researchers need to be keenly aware of the challenges of compound-mediated assay interference and target promiscuity, as well as the potential costs of following up on such dubious chemical matter. We consider several articles must-reads for HTS researchers (Box 1). In addition, researchers should be keenly aware of at least the “greatest hits” PAINS (Fig. 1).¹ These are about 20 chemical substructures that encompass the most prevalent assay interference compounds. We recommend that researchers run their active compounds through the free PAINS filters and tread very carefully in the chemical space that is defined by these compounds. Most importantly, researchers should consult with scientists who understand compound structure and reactivity (e.g., medicinal chemists), as well as scientists who know how to experimentally determine compound mechanisms of action and assay interference (e.g., molecular pharmacologists). It is true that small molecules in the biological milieu do not necessarily undergo reactions like they would in the organic laboratory. However, the challenge of PAINS is that they often do react or interfere nonspecifically with proteins and assay reagents as if they were well-characterized reagents in the organic chemist's toolbox. Many of these modes of interference and nonspecific bioactivity are well characterized and researchers should be generally aware of these broad categories and some common practices for their prevention and identification (Box 2).

Box 1. Six High-Yield Articles Regarding PAINS, Assay Interference, and Bioassay Promiscuity (Alphabetical Order).

• • Baell JB, Holloway GA: New substructure filters for removal of pan assay interference compounds (PAINS) from screening libraries and for their exclusion in bioassays. J Med Chem 2010;53:2719–2740. The seminal article on PAINS. The supporting information alone is worth a thorough read.

• • Dahlin JL, Inglese J, Walters MA: Mitigating risk in academic preclinical drug discovery. Nat Rev Drug Discov 2015;14:279–294. Recommended for academic HTS groups and those interested in drug discovery and development.

• • Dahlin JL, Walters MA: The essential roles of chemistry in high-throughput screening triage. Future Med Chem 2014;6:1265–1290. A detailed description of why it is important to include knowledgeable medicinal chemists in all HTS campaigns.

• • Ferreira RS, Simeonov A, Jadhav A, et al.: Complementarity between a docking and a high-throughput screen in discovering new cruzain inhibitors. J Med Chem 2010;53:4891–4905. Required reading for those utilizing (v)HTS in any form. A highly illustrative example showing the prevalence of false positives in both HTS and vHTS.

• • Thorne N, Auld DS, Inglese J: Apparent activity in high-throughput screening: origins of compound-dependent assay interference. Curr Opin Chem Biol 2010;14:315–324. A comprehensive description of assay interference sources.

• • Walters WP, Namchuk M: A guide to drug discovery: designing screens: how to make your hits a hit. Nat Rev Drug Discov 2003;2:259–266. Discusses REOS filters, substructures, and reactive moieties typically discarded during HTS triage.

Box 2. How to Address the Major Mechanisms of Compound-Mediated Assay Interference.

Aggregators: Test compound activity ± detergents. Examine dose–response relationships (steepness of Hill slopes). Perform chemical aggregation counter-screens.

Chemical reactivity: Use LC/MS to determine reactivity with GSH, cysteamine, or protein target. Check for decomposition in assay buffer using LC/MS.

Redox activity: Check for hydrogen peroxide formation ± reducing agents.

Impurity interference: Insure compound purity and identity by analytical methods. Activity before and after purification should be similar.

Fluorescence interference: Obtain fluorescence spectra of test compounds in context of assay readouts.

Assay-specific interference: Perform orthogonal assays.

Strategies HTS researchers can employ to prevent publishing PAINS-full articles are actually straightforward. An HTS assay should ideally be validated by reference compounds that serve as positive controls (ideally these controls are not PAINS themselves). These controls should show up as actives in a well-designed HTS assay. Of course, if the target is particularly novel, this recommendation can present challenges in terms of finding good positive controls, in which case the next best option would be to find compounds that have some validated activity against related targets. It may also be useful to characterize the readout of the assay using a small library of compounds that are known to interfere in assays by well-defined mechanisms. This will help in assay optimization and prime the project team to watch out for similar interference in compounds identified as actives.²⁴

Since almost all libraries will contain interference compounds, and since even the most robust assays will be subject to some rate of false positives, the importance of post-HTS triage cannot be overemphasized. Every screening project ought to feature a well-defined screening tree and a strategic plan that will weed out these good actors.²⁵ This screening tree should include orthogonal assays, selectivity testing, and counter-screens for common sources of assay interference, such as aggregation, fluorescence interference, compound stability, compound reactivity, and redox activity. The medicinal chemistry of active compounds must be understood, including their potential reactivity and physicochemical properties. Actives that are reported must be well characterized for identity and purity (Fig. 3). The natural histories of actives should be determined. That is, where have they or related compounds shown up in the literature or in previous HTS campaigns? How do these compounds fit into the context of the scientific literature? If the compounds contain PAINS substructures, then their activities should be confirmed in an orthogonal assay (different readout, same target) and counter-screens (same readout, unrelated target) and reasonable explanations given for why they should not be considered assay artifacts. It may seem obvious, but the chemical structures of the best compounds found in an HTS assay should be revealed in the main body of any article or proposal. Extra care needs to be taken especially if probe status¹¹ is claimed for a compound, if the SAR for the compounds tested is flat, or if no chemistry follow-up (analytical or synthetic) can be performed on the compounds.

Fig. 3.

Levels of confidence for chemical and activity characterization. Increasing confidence in the chemical matter should be part of any screening strategy.

Editors

Journal editors should be PAINS-fully aware and reject articles that contain PAINS or other well-documented assay interference compounds if the authors show a lack of awareness of the literature and the risks of operating in this area. The burden of proof for reasonableness of true activity and noncovalent target engagement for these compounds should be required of authors. Editors and journals may find it useful to publish a checkbox list of the basic requirements for acceptance of articles describing HTS hits with bioactivity claims. These would be the biological equivalent of the checkbox list of spectroscopic and analytical characterization rules that many ACS publications such as the Journal of Medicinal Chemistry use to insure the identity and purity of compounds that are reported in their pages.

What is reasonable for editors to expect in an article that reports an HTS campaign and actives? (1) A discussion of the criteria for selecting HTS actives and specifics regarding the triage process—in other words, a sufficiently detailed description of the HTS assay and triage protocol that another HTS group could conceivably identify the same compound(s) had they run a similar independent screen and triage on the same library. (2) The chemical structures of the screening actives. (3) The spectroscopic and analytical data supporting the purity and identity of the featured compounds. (4) A summary of the results of screening the actives using the freely available filters (PAINS, etc.). (5) A description of the results of literature and database searches on the active compounds, along with an appropriate discussion thereof. (6) As appropriate, data supporting compound stability, nonreactivity, selectivity, activity in orthogonal assays, inactivity in counter-screens, initial SAR, and target engagement.

Other, more general, steps can be taken by publications to guard against PAINS publications. For example, the editors of Journal of Medicinal Chemistry (and some other journals) have begun requesting structural data files from authors to enable reviewers to rapidly check for PAINS.²⁶ We recommend that medicinal chemists be assigned as reviewers on articles that claim new bioactivity of chemical substances.

Reviewers

However, the burden of preventing PAINS publications does not rest entirely on researchers and editors. Reviewers must also do their part. Reviewers of manuscripts and proposals should make themselves aware of PAINS and be on the lookout for them and also more subversive chemical matter. In our experience, these compounds will often appear in articles that may arise from an academic vHTS exercise, will have the usual suspect structures, and will feature primarily flat SAR (very little change of activity with significant structural changes) with the activities of compounds ranging from 1 to 10 μM or greater. Some esthetically pleasing graphics of docked compounds (often performed post hoc) will likely be included to validate target engagement and explain the apparent SAR. Typically, no other published activity data on the commercially available compounds will be reported, nor will any other articles that report the activity of the compounds be cited. These typical hallmarks of a PAINS publication were recently codified in a webinar hosted by the ACS by Dr. Baell (Box 3).¹²

Box 3. Hallmarks of PAINS-Full Publications.

• • Highly active compounds primarily from virtual screening or from limited HTS.

• • Limited or no medicinal chemistry involvement (no resynthesis, purification, or quality control of purity and identity of actives).

• • Flat or limited SAR (all compounds show activity in the μM range of activity with usually <10-fold difference in activity).

• • Molecular modeling of actives at putative target site with extensive analysis of all of the ligand–target interactions as though discussing a cocrystal structure.

• • Lack of natural history of actives. The activities of structurally similar compounds that have appeared in the scientific literature are largely ignored.

Reviewers should also be aware of two of the most common arguments that authors may make when ignoring PAINS. These counter-arguments are true as stated: (1) some FDA-approved therapeutics contain PAINS substructures, and (2) many FDA-approved therapeutics are reactive covalent inhibitors. These facts seem to run counter to the argument that PAINS and reactive compounds need to be avoided. However, if the actives are false positives, then it does not matter what the FDA has approved. Furthermore, many of the modern covalent inhibitors have been developed with a clear mechanistic hypothesis, extensive medicinal chemistry optimization and selectivity testing, and detailed mechanistic studies to determine the reversible and nonreversible binding components. Covalent inhibitors are typically not sought out as starting points for probe or drug discovery unless they are mechanistically relevant.

Readers

Unfortunately, readers of the literature still need to be careful consumers when reading articles that claim meaningful on-target activity for compounds without offering evidence commensurate with these claims. Just because something is published, even in the most reputable journals, it does not mean that it is entirely true or the data appropriately interpreted. As always should be the case, readers need to critically appraise the original literature. We advise readers to ask some basic questions: What evidence do the authors present showing this compound engages the proposed target specifically? How were possible off-target activities tested in the experiments? Are the compounds pure and have their identities been confirmed? If in doubt about what the data really show, we recommend consulting expertise before incorporating the reported results into one's own research.

Summary

The underinformed follow-up of PAINS and other assay interference compounds continues to dilute the scientific literature and represent a waste of time and money. The good news is that PAINS-full research is easy to triage. The tools are in place to make such research easier to recognize, but it will take the concerted efforts of researchers, editors, reviewers, and readers to insure that this happens.

Abbreviations Used

ACS

American Chemical Society

GSH

L-glutathione

HTS

high-throughput screening

IC₅₀

half-maximal inhibitory concentration

LC/MS

liquid chromatography/mass spectrometry

PAINS

pan assay interference compounds

PI

principal investigator

REOS

rapid elimination of swill

SAR

structure–activity relationship

SIR

structure–interference relationship

vHTS

virtual HTS

Acknowledgments

J.L.D. was supported by a National Institutes of Health predoctoral fellowship (F30 DK092026-01), a Pharmaceutical Research and Manufacturers of America Foundation predoctoral pharmacology/toxicology fellowship, and the Mayo Foundation for Medical Education and Research. The authors thank Drs. Jonathan B. Baell (Monash University, AU) and Daniel A. Erlanson (Carmot Therapeutics) for helpful discussions.

Disclosure Statement

The authors claim no institutional or commercial affiliations that might pose a conflict of interest regarding the publication of this article.