An early indicator of drug success: Top Journal Selectivity Index

Abstract

The Top Journal Selectivity Index (TJSI) is a scientometric index reflecting the potential importance of a new drug. It represents the ratio of the number of all types of articles on a particular drug in the top 20 journals relative to the number of articles in all (>5,000) biomedical journals covered by Medline over the 5 years since the drug’s introduction. The TJSI can be an indicator of a drug’s potential for sustained use: a higher score increases the probability of continuing success.

Introduction

Scientometric methods can be used to show potentially fruitful areas for drug discovery, based on the assumption that higher levels of scientific activity, including publication rates, will point to specific targets for novel therapies.1 Bordons et al demonstrated the usefulness of bibliometric analyses to detect trends in the research of a therapeutic drug including the evolution of drug studies over time.2 Can a scientometric indicator reflecting initial drug “coverage” in biomedical journals be used to assess a specific drug’s potential?

The first attempt to predict the clinical success of drugs by using bibliometric data was made by Windsor in 1976.3 Later he wrote, “Just as rabbits leave rabbit tracks and squirrels leave squirrel tracks – successful drugs leave different bibliometric tracks than do unsuccessful drugs. Sometimes these track records can be used to make predictions.”4 He used bibliometric traits of the journal literature on levodopa over a 14-year period to identify predictors of the drug’s success. He concluded that the bibliometry of single-author papers may have promise in this regard.

In 2011 a scientometric indicator, the Top Journal Selectivity Index (TJSI), was suggested for use in the assessment of therapeutic drugs.5 It represents the ratio (as percentage) of the number of all types of articles on a particular drug in the top 20 journals relative to the number of articles in all (>5000) biomedical journals covered by Medline over the 5 years since the drug’s introduction. Five years is a relatively long period of time, but the accurate assessment of a drug’s value takes much longer. The true assessment of a drug usually comes with the confirmation of its effectiveness by meta-analyses (the process of combining results of many clinical studies to draw conclusions about the therapeutic value of a drug). The conclusion on a drug’s value reached via this process can take 10–20 years.6 However, even this period may not be sufficient, because the comparison of various drugs in meta-analyses is usually indirect, using a placebo as a common comparator.7 As a result such comparisons are not often reliable; only direct head-to-head comparisons between drugs can provide dependable information on drugs’ relative efficacy Thus an accurate conclusion on a drug’s value via this process can take longer than 20 years. The relatively rapid (3–5 years) publication response to a new drug may have an important predictive power related to the whole period of a drug’s protracted evaluation.

In the assessment of the success of new analgesics over the past 50 years, we observed a difference in the publication response to a new drug between biomedical journals in general and in the top journals: the number of published articles on a drug increased (or declined) more rapidly in the top journals.6 This observation prompted the introduction of TJSI as an early indicator of drug success. This feature of TJSI can probably be explained by the high-caliber experts involved in the assessment of manuscripts evaluating new drugs in the top specialty journals.

TJSI determinants

One of the TJSI determinants is the number of all types of journal articles with or without abstracts (including editorials, case reports, reviews, and letters to the editor) covered by the Web site of US National Library of Medicine – PubMed (see http://www.ncbi.nlm.nih.gov/PubMed). PubMed comprises over 21 million citations for the biomedical literature and covers approximately 5000 journals. The PubMed Web site was used to count the number of articles in English published in all types of journals (original research and review journals). Specific keywords were selected according to the name of a selected drug. Boolean operations were used in which variables were the selected keywords and years of publications.

Another TJSI determinant is the number of articles published in the 20 top journals (also counted using the PubMed Web site). The selection of the Top 20 Journals was based on two factors: the rank of a journal sorted by (1) the Impact Factor and (2) the journal specialty area related to the drug’s specific pharmacological class. Sixteen journals were common for all selected drugs. They represent medicine in general (10 journals) and pharmacology (six journals): Ann Intern Med, Annu Rev Pharmacol Toxicol, BMJ, Clin Pharmacol Ther, J Clin Invest, J Pharmacol Exp Ther, JAMA, Lancet, N Engl J Med, Nat Med, Nat Rev Drug Discov, Nature, Pharmacol Rev, Proc Natl Acad Sci U S A, Science, and Trends Pharmacol Sci. Four additional items were selected depending on each drug specialty area. For example, in the case of drugs used by neurologists the selected journals were: Ann Neurology, Brain, Nat Neurosci, and Trends Neurosci. Journals were sorted by the rank of their Impact Factor presented by Journal Citation Reports, Science Edition (see http://science.thomsonreuters.com). The choice of the specific journals is more important than the total number of top journals. When we used 100 top journals instead of 20, the difference in the values of TJSI was not distinct. It should be noted that the chosen determinants are not mutually exclusive; one relies on a subject of the information used to create the other. Figure 1 gives an example of TJSI calculated in this manner for drugs in six pharmacological classes when the first-in-class drug (FICD) had multiple follow-on drugs (FOD). The duration of 5 years assessment period since drug introduction can be changed. The assessment period could be decreased to 3 years if the increase in the rate of publications in all journals is sufficiently high from the very beginning.

Figure 1.

Example of Top Journal Selectivity Indices (TJSI) of first-in-class drugs (FICD) with multiple follow-on drugs (FOD) in six pharmacological classes.Note: Columns represent TJSI: the ratio of the number of all types of articles on a particular drug in the top 20 journals to the number of articles in all (>5000) biomedical journals covered by Medline (PubMed), both over 5 years since the drug’s introduction.Abbreviations: ACE, angiotensin-converting enzyme; PPI, proton pump inhibitors; NRTI, nucleotide reverse transcriptase inhibitors.

Relationship between TJSI and success score for analgesic drugs

Ten analgesics approved by the US Federal Drug Administration (FDA) during the period 1986–2009 were selected for analysis.5 They included new molecular entities (NME) developed as analgesics and also drugs developed for nonpain indications with subsequent FDA approval for the treatment of pain as an additional indication: sumatriptan, tramadol, remifentanil, gabapentin, zolmitriptan, celecoxib, ziconotide, pregabalin, topiramate, and duloxetine. In counting the numbers of articles published over the 5 years since a drug’s introduction, specific keywords were selected according to the name of an analgesic in addition to using PubMed Medical Subject Heading [MeSH] terms. Terms added to the name of a drug were “pain” OR “headache disorder” [MeSH] OR “migraine”.

The analgesic success score was determined based on the following criteria: (1) novelty of molecular target (completely novel target, novel modification of existing target, or incremental improvement of existing drug); (2) analgesic efficacy (magnitude of pain relief, strength of evidence regarding effectiveness, and universality in use); and (3) response by the pharmaceutical market (based on the number of NME similar to the initial drug). The contributions of these three criteria to the total success score were weighted differently: much more weight was given to analgesic efficacy (up to the maximum of 5 points) than to the two other criteria (up to the maximum of 2 points each). The strength of the relationship between TJSI and success score was quite impressive: r = 0.876, P < 0.001 (Figure 2). Even if the “outlier” sumatriptan is not taken into account, the correlation coefficient continues to be high at 0.694 (P < 0.05).

Figure 2.

The relationship between TJSI and success score in the development of new analgesics.Note: The correlation coefficient (r) for TJSI and success score is 0.876 (P < 0.001).Abbreviations: TJSI, Top Journals Selectivity Index; To, topiramate; D, duloxetine; P, pregabalin; Zo, zolmitriptan; Tr, tramadol; R, remifentanil; Zi, ziconotide; C, celecoxib; G, gabapentin; S, sumatriptan.

TJSI was compared with two other bibliometric indices: the number of all types of articles presented in PubMed (AJI) and the number of articles covering only randomized controlled trials (RCT). It was found that although there was some tendency for positive relationships between the “all articles” index (or “RCT articles”) and the success score, these relationships did not reach statistical significance.

The TJSI approach is purely statistical. There are other approaches trying to find relationships between the success rate of drugs and different factors of drugs research performance. The most interesting in this report is the study by Koenig.9 He defined the Composite Drug Output Index, which served as a good predictor for past (or) future success. His finding of a high correlation between the number of all published articles and expert judgment of the quality of pharmaceutical research is of particular interest. His data also indicated that the number of highly cited clinical articles correlated with drug research success (assessed by expert judgment) even better than the number of all clinical articles. Because the top biomedical journals selected for our index calculations have a much higher citation rate than the rest of the journals, the previously mentioned better correlation rate for highly cited clinical articles compared to all clinical articles could service as an indication going in the same direction as our conclusion.

TJSI detects difference between breakthrough and follow-on drugs in various pharmacological classes

Two groups of drugs were selected to test the hypothesis that the difference between the most successful (breakthrough) and less successful drugs of the same category can be detected by TJSI.8 The FDA Web site for approved drug products (see http://www.fda.gov/Drugs) was used for selection of drugs approved between 1980–1989. One group of 10 agents included the most successful (breakthrough) drugs belonging to different pharmacological classes. The other group of 10 drugs was selected using a paired design: the paired drug belonged to the same pharmacological group with a mechanism of action as similar as possible. The selection of drugs for both groups was based on two principles: the degree of advancement in the new drug development and the level of commercial success. The drug in the first group had a novel mechanism of action and also a highly competitive market status (six or more pharmaceutical companies had applied to the FDA for a generic alternative). The drug in the second group provided only an incremental improvement on the existing drug and had a lower marketing status as a generic drug (one to five pharmaceutical companies applied to the FDA). The following breakthrough drugs were included: acyclovir, buprenorphine, Captopril, diltiazem, enalapril, fluoxetine, gemfibrozil, lovastatin, omeprazole, and zidovudine. The paired group of drugs included: abacavir, alfentanil, felodipine, fenofibrate, fluvastatin, foscarnet, lansoprazole, maprotiline, moexipril, and Perindopril.

The two groups were compared using three publication indices: the TJSI, the number of all types of articles on a drug in journals covered by PubMed (AJI), and the number of articles presenting only randomized controlled trials (RCT). It was found that TJSI can detect the difference between the two groups of drugs better than the two other indices. The results indicated that the mean TJSI of the breakthrough drugs was much higher than that of similar, but less successful, drugs (18.6 ± 2.9 vs 7.4 ± 4.2, difference of 11.2 ± 5.2; P < 0.0001). For the AJI the mean difference between the groups was 423 ± 457 (P < 0.02), and for the RCT index the mean difference was statistically insignificant (8 ± 42; P > 0.5).

Table 1 indicates that the higher value of TJSI was always that of the breakthrough member of a pair. With AJI, the value of a breakthrough member was higher, in only eight of 10 pairs. With RCT, the value of an index was higher with a breakthrough member in only four of 10 pairs. The percentage of times that an index for the breakthrough drug was higher than that of its corresponding pair, and related 95% confidence interval (CI) is presented in Table 1. The table indicates that the TJSI was distinguishable from random chance was 100% of the time with a 95% CI at 69% and 100%. For the AJI, the value was 80% of the time with a 95% CI at 44% and 98%. For the RCT index, the distinguishability from random chance was 40% of the time with a 95% CI at 12% and 74%. Thus, TJSI will detect the breakthrough drug at least 69% of the time with 95% confidence. For the other two indices, their CIs contain 50%, and thus they are not distinguishable from random chance (50%) in selecting the breakthrough drug. Thus, TJSI can detect the difference between a breakthrough drug and a less successful drug from the same pharmacological class.

Table 1.

Comparison of paired drugs

n	Paired couples	TJSI	AJI	RCT
1	Acyclovir vs doscarnet	+	+	+
2	Buprenorphine vs alfentanil	+
3	Captopril vs moexipril	+	+	+
4	Diltiazem vs felodipine	+	+
5	Enalapril vs Perindopril	+	+	+
6	Fluoxetine vs maprotiline	+	+	+
7	Gemfibrozil vs fenofibrate	+
8	Lovastatin vs fluvastatin	+	+
9	Omeprazole vs lansoprazole	+	+
10	Zidovudine vs abacavir	+	+
Distinguishability from random chance		100%	80%	40%
95% Cl		69%–100%	44%–98%	12%–74%

Note: Plus indicates when the index for the breakthrough drug was higher than that of the compared drug in a couple.

Abbreviations: Cl, confidence interval; TJSI, Top Journals Selectivity Index; AJI, All Journals Index; RCT, randomized controlled trials.

TJSI and “me-too” drugs

Usually the introduction of a FICD is followed by the development of many similar drugs, referred to as FOD. While some FOD have pharmacological properties that distinguish them from the FICD, others do not. Drugs without distinguishing features are often called “me-too” drugs, because they offer no significant benefits (including safety) over the previous agents.10 The marketing of each new drug without distinguishing features becomes questionable, especially when the sequential number of the market entry is as high as five or more.

It was shown that TJSI can provide help in the assessment of late market entrants without distinguishing features.11 To prove that there is a relationship between the TJSI of “me-too” drugs and the order (sequential number) of their market entry, the 43 FOD without distinguishing features were assessed. The study was based on drug classes approved by the FDA for marketing between the 1960s and early 2000s. The eight pharmacological classes that had four or more drugs were included for analysis. Five specific indicators were used to identify FOD with distinguishing pharmacological properties. The analysis of the remaining 43 FOD without such distinguishing properties demonstrated (Figure 3) that the relationship between TJSIs for FODs and the order of the drugs’ market entry had a negative correlation (r = −0.372; P = 0.014): the higher the order (sequential number), the lower the TJSI.

Figure 3.

The relationship between TJSI of follow-on drugs (FOD) without distinguishing features (“me-too” drugs) and the order (sequential number) of drugs’ market entry.Notes: Along vertical axis – TJSI: Top Journal Selectivity Index presented as a fraction of first-in-class drug (FICD), calculated for each class separately. Along horizontal axis – sequential number of market entry for FOD starting with number 2 (number 1 is left for FICD). Circles indicate various follow-on drugs without distinguishing features (43 drugs).

The negative correlation between FOD without distinguishing features and the order of market entry can be used for the identification of “me-too” drugs. It was found that if TJSI is less than 0.5 of FICD and market entry order is 5 or higher, the FOD is a “me-too” drug (with false negative rate of only 9.1%.) The FICD and the first three FODs that followed it were excluded from this analysis to eliminate the possibility of counting as “me-too” drugs those that might be involved in the process of competition for being the first drug in a new class.

To underline the notion that the determinants of the TJSI are not mutually exclusive one can indicate that with each new ‘me-too’ drug a reasonable increase in the number of publications on the related NME could occur. This would increase both the TJSI and the AJI for the NME. Among quantitative studies regarding the effect of FODs order-of-entry on the drugs’ commercial success there was a publication indicating that despite the average success of pioneers over later entrants, there are examples of opposite phenomenon; later entrants have gained dominant market share.12

The development and the marketing of “me-too” drugs has been criticized as duplicative, wasting resources that should be used instead to develop more innovative products.13,14 The view that investment in the development of “me-too” drugs is not an effective use of community resources dates back to the 1959 Kefauer US Senate committee review.15 Since that time many arguments have been made on both sides of the debate regarding how many “me-too” drugs is too many.16 Identifying which FOD offer no significant benefits over previous agents, ie, how to distinguish “me-too” drugs among FOD is a difficult question, and the TJSI could confirm the presence of a ‘me-too’ drug among FODs.

Conclusion

TJSI is a scientometric index reflecting the potential importance of a new drug and can be an indicator for sustained use of a drug: a higher score increases the probability of continuing success.

The universality of the TJSI has important limitations. Studies on drugs with narrow, limited use do not have many publications; some of such studies are not published. In addition, many drugs do not have studies published in top-ranked journals. For example, among the 14 analgesics analyzed previously5 only 10 had enough publications to calculate the TJSI. Four drugs (ketorolac, pentosan, topical lidocaine, and valproate) were excluded from the calculations, either because they did not have sufficient number of publications in the All Journal category (more than 10 articles) or not a single article in the Top 20 Journals category.

The other TJSI limitation is that this index does not differentiate between publications characterizing a drug in a positive and negative way. Sometimes after a drug introduction (due to the possible discovery of unexpected adverse effects) a significant number of articles can be devoted to the serious problems associated with its administration. For example, in 2001–2008, publications on the cardiovascular risk related to cycloxygenase-2 (COX-2) selective inhibitors reached up to 12% of all publications on COX-2 inhibitors.6

The evaluation of the TJSI is based on the three presented examples. The real-life utility of this index for various groups of drugs and different situations should be further explored.