Academic Ranking Score: A Publication-Based Reproducible Metric of Thought Leadership in Urology

Abstract

Background

Hospital rankings have become integral to the marketing strategies of many health care systems. Methodology used in compiling these lists appears highly flawed.

Objective

Improve on current hospital ranking systems and to develop a more meaningful measure of a urology department’s contribution to the field, we developed an academic ranking score (ARS) based on publicly available data.

Design, setting, and participants

An active faculty list was assembled for each department. A list of all publications from each department from 2005 to 2010 was then compiled. Only publications with faculty members as first or last author were considered. The ARS was then derived by identifying the number of publications within an institution, normalized by the impact factor of the peer-reviewed journal in which the publication appeared.

Measurements

The 2010 U.S. News and World Report (USNWR) urology list was reranked based on ARS and compared with the USNWR rank list. ARS was also calculated for several leading European urologic centers.

Results and limitations

A total of 6437 urologic publications were indexed to calculate the ARS. Two of the top three programs in the USNWR rankings dropped out of the top 10. The top 10 academically ranked programs increased or decreased an average of >5 positions (range: 0–17). No correlation was seen between programs ranked in the top 10 by USNWR and our objective ARS method (Spearman ρ: −0.1; p = 0.75). Because ARS only includes first- or last-author publications for faculty with clinical duties, ARS likely excludes basic science contributions and contributions from nonclinical faculty.

Conclusions

Ranking of urology departments through quantification of each program’s recent academic contribution, as captured by the ARS, differs substantially from rankings developed by USNWR. Integration of such objective measures into an overall urology program ranking system would replace current subjective opinions marred by historical biases with up-to-date merit-based assessments.

1. Introduction

The U.S. News and World Report (USNWR) rankings of US hospitals have become integral to the marketing strategies of many health care systems [1]. Patients, physicians, and administrators survey, quote, and legitimize this annual list. Nevertheless, the methodology used in compiling this list is overly dependent on subjective criteria. As such, it is flawed [2].

The methodology used by USNWR in assessing the quality of health care is based on principles described by Avedis Donabedian in 1966. Donabedian’s theory states that the quality of medical care can be quantitated through three domains: structure, process, and outcomes [3]. As such, USNWR generates its “best hospitals” rankings based on weighing metrics that fall into Donabedian’s three domains of quality [4]. For the structure domain, the magazine analyzes such variables as nurse staffing, availability of desired technologies, nurse magnet status, and patient volume. The outcomes domain is composed of variables such as adjusted mortality and adverse patient events. Finally, the process domain is represented by the hospital’s reputation score [4]. Whereas the structure and outcomes domains are defined by a composite of multiple objective measures, the process domain is based on a single subjective reputation score. In fact, it is this reputation metric, which USNWR has considered a proxy for process, that appears to have a dominating effect on the final ranking [5]. A recent study reveals that the process domain has the highest degree of variation among all of the USNWR score components, thus contributing disproportionately to the USNWR hospital ranking. In fact, nearly perfect correlation exists between the reputation score and the final hospital ranking for all 12 specialties examined [5]. Within the specialty of urology, the USNWR rankings seem to reflect reality particularly poorly [5]. In fact, the rankings every year generate competitive discussions within the specialty because some strong programs appear low on the list (or do not appear at all), whereas larger and “reputable,” but less accomplished, departments are placed high in the rankings. This is not surprising because the reputation score is generated from surveying approximately 250 urologists, asking them to list the top five hospitals. Furthermore, only 40–50% of specialists respond to this entirely subjective survey [5].

Given the increasing availability of data on academic contributions, clinical outcomes, and process metrics by hospitals, opportunities exist to improve ranking methodology. Instead of relying largely on highly subjective and bias-prone survey responses, as USNWR does, we propose moving toward more objective and thus more meaningful metrics. We developed an objective ranking system we call the academic ranking score (ARS) that quantitates thought leadership and scientific contribution from a given urology department and compares this contribution with contributions from other departments in the specialty.

2. Methods

Using hospital Web site registries, we compiled a list of all affiliated urologists. These included full-time and part-time clinicians on staff in urology departments/divisions across the United States. Physicians who were listed at one institution (at the time of publication) but have held positions and published at another institution in the past 5 yr were included as a faculty member at the most recent institution. A few urologists were listed as faculty members at multiple hospitals and were included in each hospital’s faculty list. We obtained faculty lists from either their Web site or by e-mailing the chairs of several prominent European urology departments. In the spirit of USNWR rankings, only faculty members with clinical responsibilities were including in calculating the ARS.

2.1. Medline search

A PubMed Medline search was performed for each hospital’s urology department. Using Boolean operators, each author’s name was searched in the first and last author position (ie, Author 1 (FIRST AUTHOR) OR Author 1 (LAST AUTHOR) OR Author 2 (FIRST AUTHOR) OR Author 2 (LAST AUTHOR), etc). This method avoided double-counting publications from the same institution. The search was limited to the years 2005–2010. A list of all publications generated by each search was alphabetized by journal title. The lists were checked for fidelity by removing publications by authors who did not belong to a given institution but had similar names. For instance, individuals with the same last name but with different first/middle initials were captured and removed from each institution’s list. Other unrelated entries were excluded by manually screening each manuscript for appropriate subject matter. If the manuscript was not published in a urology journal, did not appear to be related to anything that would be written by a urology physician, and did not stem from the appropriate institution, the author was assumed to be the namesake of the author of interest and the manuscript was excluded. Thus only manuscripts with each hospital’s faculty member as first or last author and with appropriate urology-related subject matter were included in the analysis. If both first and last authors were faculty members, the manuscript was only counted once.

2.2. Academic score

A count for all publications indexed in PubMed from each institution was determined and normalized by each publication’s impact factor (IF) to obtain a standardized score sum. We used the 5-yr IF for most journals, the standard IF if a 5-yr IF was absent in the Thompson Routers database, and a value of “0” for journals without an IF. As such, publications in journals without an IF did not contribute to the overall academic score. The 2010 USNWR urology rankings were then reranked based on the ARS (Table 1) and compared with the USNWR rank list.

Table 1.

Top 50 US hospitals as determined by the 2010 US News and World Report rankings reranked using the academic ranking score (2005–2010)

“Academic ranking”	“Academic ranking” normalized by FTE	USNWR ranking 2010	Change in ranking	Institution	Academic ranking score (No. of publications adjusted by impact factor)
1	1	8	7	Memorial Sloan- Kettering Cancer Center	2055.2
2	14	2	0	Cleveland Clinic	1214.6
3	2	15	12	University of Texas Southwestern Medical Center	1204.9
4	11	11	7	University of Michigan Hospitals and Health Centers	1183.9
5	13	4	−1	Ronald Reagan UCLA Medical Center	1167.7
6	3	6*	NA	New York Presbyterian University Hospital: Cornell	1128.5
7	4	5	−2	University of California, San Francisco	1034.9
8	5	7	−1	Duke University Medical Center	959.7
9	12	26	17	Northwestern Memorial Hospital	877.2
10	6	10	0	University of Texas M.D. Anderson Cancer Center	827.7
11	7	1	−10	Johns Hopkins Hospital	795.6
12	10	9	−3	Vanderbilt University Medical Center	654.0
13	15	3	−10	Mayo Clinic, Rochester	622.7
14	8	16	2	USC University Hospital	617.3
15	17	17	2	Barnes-Jewish Hospital/Washington University	513.6
16	18	6*	NA	New York-Presbyterian University Hospital: Columbia	433.9
17	26	12	−5	Massachusetts General Hospital	408.2
18	34	23	5	NYU Langone Medical Center	389.8
19	9	28	9	University of Chicago Medical Center	376.0
20	27	30	10	University of Washington Medical Center	365.8
21	20	25	4	Shands at the University of Florida	333.0
22	23	34	12	Henry Ford Hospital	286.1
23	28	32	9	University of California, Irvine Medical Center Irvine Medical Center	274.6
24	19	40	16	University of Wisconsin Hospital and Clinics	267.2
25	42	39	14	St. Luke’s Episcopal Hospital	262.2
26	25	18	−8	Methodist Hospital	222.9
27	33	46	19	Wake Forest University Baptist Medical Center	216.0
28	21	21	−7	Brigham and Women’s Hospital	212.6
29	35	48	19	Memorial Hermann - Texas Medical Center	204.3
30	16	51†	21	Fox Chase Cancer Center	202.1
30	24	22	−8	Stanford Hospital and Clinics	199.7
31	29	13	−18	Clarian Health	194.6
32	30	19	−13	University of Iowa Hospitals and Clinics	192.6
33	38	20	−13	UPMC - University of Pittsburgh Medical Center	187.6
34	22	31	−3	University of Virginia Medical Center	176.6
35	31	14	−21	Hospital of the University of Pennsylvania	163.0
36	47	45	9	Mount Sinai Medical Center	144.9
37	36	43	6	University of Kansas Hospital	127.1
38	32	37	−1	University of Maryland Medical Center	120.0
39	39	29	−10	Ohio State University Hospital	118.0
40	37	41	1	Rush University Medical Center	113.2
41	45	27	−14	Lahey Clinic	99.5
42	40	33	−9	Emory University Hospital	98.2
43	43	44	1	Loyola University Medical Center	92.1
44	41	24	−20	University of Alabama Hospital at Birmingham	75.8
45	48	36	−9	Beaumont Hospital	57.3
46	44	42	−4	Yale - New Haven Hospital	42.2
47	46	35	−12	Tampa General Hospital	24.7
48	49	38	−10	City of Hope (Duarte, CA)	24.2
49	50	49	0	St. Cloud Hospital (Saint Cloud, MN)	18.3
50	51	47	−3	Baylor University Medical Center	13.0
51	52	50	−1	Christiana Care (Newark, DE)	11.3

FTE = full-time equivalent; USNWR = U.S. News and World Report; NA = not applicable.

^*Columbia and Cornell Departments of Urology at New York–Presbyterian University Hospital were considered as a single entity in the USNWR analysis.

^†Fox Chase Cancer Center was not ranked in USNWR and was assigned a rank of 51 for current analysis.

3. Results

All urology programs listed in the USNWR ranking were evaluated based on the previously described methodology to generate an ARS. These results are summarized in Table 1 with the new academic ranking, academic ranking adjusted by full-time equivalent (FTE), USNWR rank, the change in rank from USNWR, and an ARS. The average change in rank was 8.24. A minority of the programs (7.8%, or 4 of 51) were unchanged in ranking as evidenced in Figure 1. The range of the ARS was from 11 to 2055 with a mean of 428 and a median of 219. The top 10 academically ranked programs increased or decreased an average of >5 positions (range: 0–17). No statistical correlation was seen between the programs ranked in the top 10 by USNWR and our objective ARS method (Spearman ρ: −0.1; p = 0.75) (Fig. 2). Even when adjusting ARS on a per FTE basis to eliminate any benefits of size, no statistically significant correlation between rank lists existed (Spearman ρ: 0.33; p = 0.23). Table 2 demonstrates ARS calculated for some representative prominent European urologic centers.

Fig. 1.

Change between U.S. News and World Report rank and academic ranking.

Fig. 2.

U.S. News and World Report score versus academic ranking amid the 10 top-ranked urology departments/divisions. The dashed line corresponds to perfect agreement between the two ranking methods; the solid line is the line of best fit to the data.

Table 2.

Select European urologic centers with corresponding academic ranking scores

European center	Academic ranking score
San Raffaele Hospital (Milan, Italy)	1105.9
Ludwig Maximilians Universitat (Munich, Germany)	612.3
Radboud University Nijmegen Medical Center (Nijmegen, the Netherlands)	309.4
The Royal Hallamshire Hospital (Sheffield, UK)	267.2
Erasmus Medical Centel (Rotterdam, the Netherlands)	231.0
Henri Mondor Hospital (Creteil, France)	220.4
Oxford Radcliffe Hospitals (Oxford, UK)	193.9
The University Hospital Carl Gustav Carus (Dresden, Germany)	182.8
Johannes Gutenberg University (Mainz, Germany)	131.0
Tampere University Hospital (Tampere, Finland)	80.7

4. Discussion

Ranking of departments through quantification of each department’s recent academic contribution, as captured by the ARS, differs substantially from rankings developed by USNWR (Table 1 and Fig. 1). Two of the top three programs in the USNWR rankings dropped out of the top 10. Meanwhile, the top 10 academically ranked programs dropped or rose an average of >5 positions (range: 0–17) compared with their position in the USNWR rankings (Fig. 2). The USNWR has been criticized for using reputation as a proxy for the process domain of patient care [5]. The physicians who contribute their survey responses need not have had any documented exposure to the hospitals they are “nominating” [6] and may have considerable biases for overvaluing or undervaluing the hospitals they consider ranking. Given the near total dependency of the final ranking on reputation [5], it is not hard to argue the degree of imperfection in using reputation as a proxy for quality, outcomes, and patient satisfaction. This correlation of USNWR ranking with reputation may be in part due to a focus on ensuring “face validity.” That is, if those programs that were expected to be at the top by decision makers were not at the top after developing a ranking methodology, then the methodology would be revised [7,8].

We believe quantification of each department’s recent academic contributions better reflects the thought leadership of existing faculty and prominence of a department than a historical reputation score. We submit that thought leadership is more relevant to providing high-quality state-of-the-art patient care than the subjectivity of reputation ranking as defined by 125 individuals. The ARS affords an objective metric of leadership in a given field that until now remained almost entirely subjective. Integration of such objective measures into an overall ranking system should replace the subjective opinions marred by historical biases and “face validity” with up-to-date merit-based assessments. Furthermore, the ARS may empower each department to improve its academic productivity and thus may contribute to the enhancement of both the quantity and quality of the urologic literature.

The academic ranking system is not without its own limitations. The process of obtaining the data for this ranking system is currently extremely laborious but in the future could be automated, particularly given the recent mandate to report published materials through the National Institutes of Health Public Access Policy [9]. Unfortunately, the current state of PubMed indexing does not afford generation of high-fidelity publication lists simply based on institution or physician names. The ARS can only quantitate a hospital’s or department’s published academic contribution. Although the relationship of this metric to a hospital’s clinical quality is unknown, it remains more objective than reputation. Also noteworthy is that the ARS presented in this paper only includes first- or last-author publications for faculty with clinical duties and thus may exclude a department’s basic science contributions or those from nonclinical faculty.

In summary, as we move toward increasingly accountable, evidence-based care, we invite the medical community to develop, test, and publish other objective performance assessments such as the academic ranking to better inform the public regarding our strengths and to identify our weaknesses.

5. Conclusions

Ranking of urology departments through quantification of each program’s recent academic contribution, as captured by the ARS, differs substantially from rankings developed by USNWR. The ARS, a simple and reproducible measurement of thought leadership, is calculated by identifying the number of publications originating from a specific department/division within an institution, normalized by the IF of the peer-reviewed journal. This novel metric affords an objective assessment of thought leadership that until now remained almost entirely subjective.

Take-home message.

As we move toward increasingly accountable, evidence-based care, the urology community must test and publish objective performance assessments. The academic ranking system presented in this paper affords quantification of a department’s thought leadership and academic contributions.