An Automated Approach for Ranking Journals to Help in Clinician Decision Support

Abstract

Point of care access to knowledge from full text journal articles supports decision-making and decreases medical errors. However, it is an overwhelming task to search through full text journal articles and find quality information needed by clinicians. We developed a method to rate journals for a given clinical topic, Congestive Heart Failure (CHF). Our method enables filtering of journals and ranking of journal articles based on source journal in relation to CHF. We also obtained a journal priority score, which automatically rates any journal based on its importance to CHF. Comparing our ranking with data gathered by surveying 169 cardiologists, who publish on CHF, our best Multiple Linear Regression model showed a correlation of 0.880, based on five-fold cross validation. Our ranking system can be extended to other clinical topics.

Introduction

Since medical errors have become a leading cause of death^1,2 and the healthcare costs and complexity of the diseases have increased, Clinical Decision Support (CDS) shifted its focus from an auxiliary technology to a practical necessity.³ Due to the sheer amount of information available on clinical topics and the limited time of the user, it is important to deliver context-specific natural language answers. Manual approaches to CDS is cumbersome for content maintainers. Therefore, an automated approach, like text mining, would be desirable.

Our previous research has demonstrated the feasibility of automatically extracting relevant sentences from MEDLINE® abstracts⁴ to aid clinicians in finding relevant information quickly. But, much of the relevant information is in the full text article, not the abstract. Taking two randomly selected articles from UpToDate®⁵ (a clinical knowledge system) titled “Overview of the Therapy of Heart Failure Due to Systolic Dysfunction” and “Clinically Isolated Syndromes Suggestive of Multiple Sclerosis”, we can illustrate the shortcomings of abstract only processing. By counting the number of citations in the abstract and the full-text, we found that only 49.55% (56 out of 113) of the references in the Heart Failure related article and 68.11% (47 out of 69) of the citations in the Multiple Sclerosis related article refer to information in abstract text. Combining the two, only 56.59% (103 out of 182) of references in clinical articles refer to abstracts, suggesting the importance of processing full-text for providing clinical decision support.

Today, there are over 2500 journals that have at least one article indexed in Medline with the Major MeSH® Term – “Congestive Heart Failure”. Therefore, it is important to prioritize the journals not only to assign a manageable task for text mining, but also to give confidence to clinicians by providing content from the journals they trust. In addition, journal ranking can be used as a feature in article prioritization task where systems rank⁶ or classify⁷ articles. There has been research focused on prioritizing individual articles based on quality, specifically for clinical decision support⁸, but few focused on prioritizing journals first. Prioritizing journals first aids the clinical knowledge system’s decision on the number of full-text articles to process. In addition, directly measuring an article’s relevance to a topic is a manageable task using standard text categorization and machine learning techniques^6,7; however, obtaining the full-texts of the articles for all the 2500 journals and processing each article is not.

Abridged Index Medicus (AIM)⁹, also known as core clinical journal titles, is an existing resource that ranks top journals, which has been available since 1970.¹⁰ However, this list is not suitable for identifying the journals from which articles about a specific topic or even a branch of medicine could be extracted. For example, the list misses cardiology journals such as Circulation: Heart Failure, JACC: Heart Failure and European Heart Journal that are perceived to be important by cardiologists. The McMaster Plus project¹¹ collaborates with the BMJ Group to provide citations, rated for quality by hand, from over 120 top journals. However, this list of top journals also misses journals important for a specific topic such as heart failure (example: Circulation: Heart Failure, JACC: Heart Failure and Journal of Heart Failure).

In this study, we identified the top journals for “Congestive Heart Failure” and the factors that favored their positive perception among cardiologists. We then developed a regression model to prioritize automatically journals for any topic.

Methods

A. Survey Procedure

All the Medline abstracts indexed with the MeSH Major Topic – “Congestive Heart Failure” were downloaded in XML format using the PubMed® interface.¹² The email addresses of the corresponding authors and the countries of their affiliation were extracted using a set of rules and regular expressions.¹³ To decrease the variability of the survey participants, we limited them to US organizations.

To reduce outliers and selection bias, we took into consideration that cardiologists are busy and an individual follows only a few journals. Therefore, we first ranked the journals by the number of their articles indexed in Medline by the MeSH Major Topic – “Congestive Heart Failure”. The top-100 journals presented in random order were rated by eight cardiologists locally on a 1–5 scale based on their value for clinical decision-making (same criterion as actual survey). Sixty journals that are above a threshold agreed after looking at the distribution of the ratings were considered for the final survey. Based on our experience with the Mayo Clinic cardiologists, we asked the survey participants to rank at least 20 journals among these. The survey was sent to all the participants on March 8, 2013. They were reminded to participate on March 15, 2013 and finally again on March 22, 2013.

Informed consent was obtained for each participant by asking at the beginning:

We want to extract automatically sentences from journals that are relevant to clinician information needs. For the sake of building a prototype, we are asking cardiologists to provide a subjective opinion by rating few journals (presented in random order) in relation to congestive heart failure. By clicking the next button, please confirm that you are consenting to take part in the survey and that you are very familiar with cardiology and congestive heart failure in particular.

If they provided informed consent, we asked them the mandatory questions listed below:

• For how many years have you been practicing medicine?

• How many publications do you have in cardiology?

• How many journals related to congestive heart failure do you follow regularly?

• Choose all your current roles.

• Please rate how valuable information in each journal is to your clinical decision making.

Once they finished the mandatory questions, we asked them two optional questions:

• When choosing a rating for the journal, what are the factors important to you?

• Any comments about the survey?

B. Metric Analysis

With the help of a librarian, we searched existing medical literature to know what factors of a journal such as impact factors, aims & scope and download counts are important in determining whether clinicians follow that journal or use the information at point of care. The metrics mentioned in the articles^14–28 were used as choices in the optional questions.

In addition, the journal-related numeric metrics were compared to the rating average of the corresponding journal using Multiple Linear Regression²⁹ algorithm:

1. Impact Factor (IF)

2. H-Index

3. SCImago Journal Ranking (SJR)

4. Number of articles (Total Docs)

5. Number of references in the articles (Total Refs)

6. Number of articles for 3 years (3yr Docs)

7. Number of citations to articles for 3 years (3yr Cites)

8. Number of cited-articles for 3 years (3yr Cited)

9. Number of references per article (Ref/Doc)

10. Number of CHF-indexed Medline abstracts (CHF count)

11. Broad Journal Heading – cardiology (0/1) (BJH)

12. Core clinical journal (0/1) (AIM)

The metrics 1–9 are freely available every year from the online resource for journal ranking provided by SCImago Journal & Country Rank³⁰ from Scopus. The metrics 10–12 are also available freely from the NLM.

The impact factor (IF) of a journal is the number of times that articles published in the past two years are cited by indexed journals during current year, divided by the total number of articles published by that journal in those two years.³¹ H-Index of a journal, similar to that of a scientist, is defined to be the maximum possible h such that h number of articles in the journal have at least h citations.³² SJR is an adaptation of the PageRank metric³³ used by Google©, where the nodes are the individual journals and the edges have weights corresponding to the amount to the citations between the journals representing the nodes. Metrics 4–9 are self-explanatory.

The “Number of CHF-indexed Medline abstracts” is obtained from PubMed interface by limiting the search to the name of the journal AND the MeSH Major Topic – “Congestive Heart Failure”. The “Broad Journal Heading” is extracted from the journal descriptions³⁴ provided by NLM®. For example, “American journal of cardiovascular drugs: drugs, devices and other interventions” has the following broad journal headings: Cardiology, Drug Therapy and Vascular Diseases. When one of the headings is Cardiology, we assign a score of ‘1’; otherwise, the score is ‘0’.

Cardiologists’ survey rating averages were used as the response variable (Y) and the 12 journal metrics were used as random independent variables (X) for Multiple Linear Regression. We tried all the combinations of the independent variables ( ${{\displaystyle \sum }}_{k=1}^{12}\left(\begin{array}{c}12\\ k\end{array}\right)=4095$). Variables 11 and 12, which are ordinal, were also compared with the rating averages using the non-parametric (rating averages unlikely to be normal in distribution and the journals in at least one group is anticipated to be less than 25) Wilcoxon’s Rank Sum Test.³⁵

Results

A. Survey Output

Among all the Medline abstracts with the MeSH Major Topic – “Congestive Heart Failure”, there were 32,702 abstracts with an affiliation sentence. Parsing them revealed 14,525 affiliation sentences with at least one email address and these are from multiple countries. Among them, 5,584 sentences correspond to the USA. In these affiliation sentences, we found 3,443 unique email addresses. Among these, 1,088 addresses expired or the recipient is out of office. Of the successful recipients, 169 accepted the eligibility criteria with informed consent and 161 finished the survey. Of these, 19 participants had 0–5 years of experience. For comparing the rating average of journals with the journal metrics, we considered the 142 participants with six or more years of experience.

Table 1 describes the characteristics of the participants. The majority of them have been in practice for over 15 years and published more than 20 cardiology articles.

Table 1.

Characteristics of survey participants.

Participant characteristic (N=161)	No. (%) of respondents
Experience (years)
0–5	19(11.8%)
6–10	20(17.4%)
11–15	22(13.7%)
16–25	45(28.0%)
>25	47(29.2%)
Cardiology publications
0–5	24(14.9%)
6–20	41(25.5%)
21–50	31(19.3%)
51–100	23(14.3%)
>100	42(26.1%)
CHF related journals following
0–5	118(73.3%)
6–10	35(21.7%)
11–15	7(4.3%)
16–25	0(0.0%)
>25	1(0.6%)
All current roles
Clinician	130(80.7%)
Researcher	132(82.0%)
Instructor	98(60.9%)

We presented to them 60 randomly ordered journals preselected by the eight Mayo Clinic cardiologists, with an average score of at least 1.5. They were asked to rate these journals by their value in information about CHF in a 1–5 scale (1=least value, 5=most value). The rating averages of the 142 participants with at least 6 years of experience are right-skewed with a median of 2.17 (25% quartile =1.87 and 75% quartile = 2.54). Figure 1(a) shows the Stem and Leaf plot for the distribution of rating averages. It illustrates how a majority of journals got an average rating of 1.5–2.5. Figure 1(b) shows the Stem and Leaf plot for the distribution of response counts. It illustrates that each journal is rated by at least 96 participants.

Figure 1.

Stem and Leaf plots for the ratings from 142 experienced participants: (a) individual rating averages for each of the 60 journals (b) individual responses for each of the 60 journals.

The top-10 journals with their rating averages are in Table 2. The highest rating average is 4.35 (The New England Journal of Medicine) and the lowest rating average is 1.58 (Basic research in cardiology). Table 2 lists the rating averages with the number of responses for 10 journals with highest rating averages. The journal titles in bold are those that are NOT indexed in AIM (core-clinical journal list).

Table 2.

Top journal rating averages for 142 participants (at least six years of experience).

Journal names	Rating Average	Response Count
The New England journal of medicine	4.35	132
Circulation	4.35	127
Journal of the American College of Cardiology (JACC)	4.13	127
JAMA : the journal of the American Medical Association	3.86	124
Circulation. Heart failure	3.79	124
Lancet	3.74	126
JACC. Heart Failure	3.52	104
European heart journal	3.21	112
Annals of internal medicine	3.14	118
Journal of cardiac failure	3.04	117

Figure 2(a) shows the correlation coefficients of the first 10 variables, which are continuous along with the respective p-values. The rows with p-values in bold indicate that the correlation of the variable with ranking average is statistically significant (alpha=0.05). It should be noted from Table 3 and Figure 2(a) that although the topic-specific metrics (CHF count, BJH and AIM) are itself less predictive than other features, they provide complementary information that improves the overall predictive power significantly in combination with top-metrics such as SJR.

Figure 2.

(a) Correlation of the continuous variables with rating averages (b) Linear bivariate fit of rating average by weighted combination of best model

Table 3.

Top-10 accurate multiple linear regression combinations (numbers rounded to 3 decimal places).

IF	H-Index	SJR	Total Docs	Total Refs	3yr Docs	3yr Cites	3yr Cited	Ref/Doc	CHF count	BJH	AIM	Correla tion
		0.826	−0.003		0.002	0	−0.001	−0.018	0.002	0.628	−0.327	0.880
		1.065	−0.002	−0.000	0.001			0.019	0.003		−0.913	0.879
		0.699	−0.004		0.002		−0.002		0.003			0.879
		1.005	−0.003	−0.000	0.002				0.002	0.406		0.878
		0.661	−0.003		0.002		−0.002	−0.015	0.003			0.877
0.293			−0.002		0.001		−0.001	−0.009	0.003			0.876
		0.677	−0.002		0.001	0.000	−0.002	−0.007	0.003			0.875
0.332			−0.002		0.001		−0.001		0.003	0.328	−0.108	0.874
		1.055		−0.000	0.000	0.000			0.003	−0.210		0.874
		0.729	−0.004		0.003		−0.002		0.002	0.445	−0.529	0.874

B. Metric Analysis

The 60 rated journals by 142 cardiologists and twelve variables were used to create a Multiple Linear Regression model for rating average. Table 3 lists the top-10 combinations (among the 4095 combinations tested) of metrics and their coefficients. The best model has a correlation of 0.880. The correlation was calculated from the 5-fold cross validation and the final coefficients were obtained from the sixth run combining all the 60 journals. The only variable that is present in all the top-10 models is CHF count (Number of CHF-indexed Medline abstracts). In fact, the best 1,388 models employ the CHF count variable. Without employing this variable, the best correlation obtained is .806 (8.4% lesser). The other variables derived from NLM’s sources – BJH (Broad Journal Heading) and AIM (core clinical journals) respectively take into account whether the journal is related to cardiology or is a top medical journal. Using none of these three metrics, the best correlation is only 0.792 (10.0% lesser).

We obtained the below formula to automatically rate the journals.

Formula 1: Journal rating metrics and coefficients

Journal priority score =

• (0.82640 * SJR) −

• (0.00377 * Total Docs) +

• (0.00258 * 3yr Docs) −

• (0.00190 * 3yr Cited) −

• (0.01846 * Ref/Doc) +

• (0.00295 * CHF count) +

• (0.62864 * BJH) −

• (0.32753 * AIM))

The two groups – core clinical journals (AIM) and the rest have statistically significant difference in means of rating averages (p=0.0003<0.05, Rank sums test). The respective means are, however, close – 2.9 vs. 2.1. The groups – cardiology journals and non-cardiology journals (based on BJH) do not have statistically significant difference in means of rating averages (p=0.1888>0.05, Rank sums test). Surprisingly, the non-cardiology journals have a slightly higher rating average mean (2.5) than cardiology journals (2.3). This is because the non-cardiology journals with higher impact factor, h-index or SJR such as NEJM and JAMA have a high acceptance even among cardiologists.

In addition, the participants (159 participants answered this question) rated the factors mentioned in literature and suggested by local cardiologists as shown below:

1. Clinical relevance: 108 (76.6%)

2. Impact factor: 93 (66.0%)

3. Type of articles – Human Vs. Animal Research: 66 (46.8%)

4. Electronic Access: 61 (43.3%)

5. Study design of the articles: 61 (43.3%)

6. Quantity of topic-related articles: 53 (37.6%)

7. Aims and scope: 50 (35.5%)

8. Proportion of topic-related articles: 49 (34.8%)

9. Open-access: 28 (19.9%)

10. Readership composition: 26 (18.4%)

11. Editorial board composition: 26 (18.4%)

Discussion

A. Selection Bias

Such a survey conducted at a single institution might be influenced by selection bias (sample being not representative of population). In our case, we invited participants from across the United States. This study is part of a 4-year project funded by the NLM (USA) intending to help cardiologists in the USA in areas such as CHF. The cardiologists invited are affiliated with organizations from all the 50 states of USA and Puerto Rico and over 500 different cities of USA.

B. Number of Participants vs. Number of Journals (Nonparticipation Bias)

More participants will minimize a journal getting a significantly higher or lower rank than its true mean (central limit theorem). A higher number of journals allows for more statistical power in the calculations of association between the metrics and rating averages. However, we observed a tradeoff between the number of participants and number of journals they rate while testing the survey at Mayo Clinic. Several cardiologists recommended reducing the journals significantly. Therefore, we chose 60 journals instead of 100 and imposed a constraint in the survey so that the participants rated at least 20 (33%) journals. Despite these, we observed statistically significant differences for all metrics except Ref/Doc (Number of references per article) and BJH (Broad Journal Heading). On average, each participant rated 45.0 journals (out of 60 maximum) and each journal received 120.75 ratings (out of 161 maximum).

To further verify this, a replication survey was conducted at Mayo clinic in which 18 cardiology fellows participated. This step is also useful for comparing results of the previous survey where the participants are established clinicians or researchers and with this survey with a different population. The demographic characteristics of the participants are presented in Table 5. In Table 6, rating averages with the number of responses for the 10 journals with highest rating averages are presented. The highest rating average is 4.71 (The New England Journal of Medicine) and the lowest rating average is 1.50. The ranking of the journals show a large overlap.

Table 5.

Characteristics of survey participants of 18 respondents

Participant characteristic (N=18)	No. (%) of respondents
Experience (years)
0–5	7 (38.9%)
6–10	8(44.4%)
>11	3(16.7%)
Cardiology publications
0–5	13(72.2%)
6–20	5(27.8%)
>21	0(0.0%)
CHF related journals following
0–5	16(88.9%)
6–10	2(11.1%)
>11	0(0.0%)
All current roles
Clinician (7 serve also as researcher, 1 as researcher & instructor)	14 (77.8%)
Researcher	4(22.8%)
Instructor	0(0.0%)

Table 6.

Top journal rating averages for 18 participants (rank in the original survey in brackets)

Journal names	Rating Average	Response Count
The New England journal of medicine (1)	4.71	14
Circulation (2)	4.57	14
Journal of the American College of Cardiology (JACC) (3)	4.46	13
JAMA : the journal of the American Medical Association (4)	3.93	14
European heart journal (8)	3.75	16
Lancet (6)	3.75	16
Circulation. Heart failure (5)	3.58	12
Heart (British Cardiac Society) (21)	3.50	14
The American journal of cardiology (11)	3.38	13
JACC. Cardiovascular imaging (19)	3.36	11

C. Generalizability

The best Multiple Linear Regression model has a correlation as high as 0.880. Figure 2(b) shows the regression fit with the weighted combination in the X-axis. We will also be able to update the list with new journals related to CHF or automatically update the scores for existing journals with time. The focus of our project was to help cardiologists with the topics – CHF and atrial fibrillation (afib). The survey we conducted locally solicited cardiologists to rate the journals for both CHF and afib. The correlation between the ratings is 89.16%. Therefore, it is likely that the list might be repurposed for other cardiology topics with expert editing.

To check the generalizability of Formula 1 for non-cardiology topics, we used the same two articles from UpToDate to assess how the rank of a journal obtained using the formula is related to the number of references from the journal. Table 4 shows all the journal names in the respective articles sorted by the journal priority score calculated using Formula 1.

Table 4.

Distribution of citations to journal names

Heart Failure related article
Journal names	Rank based on Formula 1	Rank based on survey	AIM journal?	# of references in article
The New England journal of medicine	1	1	Yes	23
Circulation	3	1	Yes	31
Journal of the American College of Cardiology	2	3	Yes	20
Lancet	4	6	Yes	8
JAMA : the journal of the American	5	4	Yes	3
Medical Association
European heart journal	6	7	No	9
The American journal of cardiology	7	10	Yes	2
American heart journal	9	11	Yes	3
Annals of internal medicine	10	8	Yes	2
Journal of cardiac failure	14	9	No	6
BMJ (Clinical research ed.)	17	19	Yes	1
Cochrane database of systematic reviews (Online)	44		No	1
American journal of hypertension	45		No	1
The Canadian journal of cardiology	48	52	No	2
Blood pressure	60		No	1
Multiple Sclerosis related article
The New England journal of medicine	3		Yes	4
Neurology	4		Yes	22
Lancet	5		Yes	11
Multiple sclerosis (Houndmills, Basingstoke, England)	7		No	3
Lancet neurology	8		No	9
Annals of neurology	11		No	6
Archives of neurology	14		Yes	6
Brain : a journal of neurology	15		Yes	4
Cochrane database of systematic reviews (Online)	38		No	2
Archives of ophthalmology	57		Yes	2

It should be noted that the ranks of the journals obtained using the formula are similar to the rankings obtained from survey ratings. The top 15 journals as per the journal priority score calculated by Formula 1 for Heart Failure domain cover 94.69% (107 out of 113) of references. Using the same formula for the MS related article (with mesh counts calculated using “multiple sclerosis”), almost the same coverage is obtained (94.20% (65 out of 69)). In comparison, if the 119 AIM journals are used, 82.30% (93 out of 113) and 71.01% (49 out of 69) of references are covered in Heart Failure and Multiple Sclerosis topics respectively – a 14% less coverage on average. For building a topic-specific clinical decision support system using knowledge in literature, we might achieve a higher coverage even using lesser number of journals (about 15) than using all the AIM journals. When building a generic decision support system, one might expand the AIM journals for the key topics using the journal priority score formula.

Although this simple analysis suggests the formula might be generalizable on non-CHF related articles (94.20% coverage on MS topic), increasing the size of the corpus by conducting multiple surveys and training the system with more than one topic will increase the accuracy (measured by correlation) of the formula.

D. Choice of Algorithm

This was determined by the fact that this is not a classification problem (where algorithms such as Naïve Bayes are used) or a clustering problem (where algorithms such as K-means are used), but a regression problem. Since the individual metrics are linearly correlated, Linear Regression is more appropriate than other regression algorithms such as Logistic Regression. Here we chose Multiple Linear Regression algorithm since it accounts for all possible combinations of the metrics.

Conclusion

With the help of cardiologists across the United States, we rated and ranked journals related to CHF. A replication survey with cardiology fellows was done to verify that the ratings are generalizable to other demographics. While these ratings will help us move forward in CHF, we have also obtained a formula to automate the process for other topics such as Multiple Sclerosis. The Multiple Linear Regression model achieves a correlation of 0.880 (five-fold cross-validation). We demonstrated that general journal metrics such as impact factor, h-index and number of articles per year provide better results when used in combination with topic-specific metrics such as number of abstracts indexed with a MeSH term. Our participants ranked these and other factors in this order: clinical relevance (77%), impact factor (66%), human vs. animal research (47%), etc. All these factors need to be taken in consideration for further improving and generalizing this work.