A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods

Abstract

Objective

Propensity score analyses attempt to control for confounding in non-experimental studies by adjusting for the likelihood that a given patient is exposed. Such analyses have been proposed to address confounding by indication, but there is little empirical evidence that they achieve better control than conventional multivariate outcome modeling.

Study design and methods

Using PubMed and Science Citation Index, we assessed the use of propensity scores over time and critically evaluated studies published through 2003.

Results

Use of propensity scores increased from a total of 8 papers before 1998 to 71 in 2003. Most of the 177 published studies abstracted assessed medications (N=60) or surgical interventions (N=51), mainly in cardiology and cardiac surgery (N=90). Whether PS methods or conventional outcome models were used to control for confounding had little effect on results in those studies in which such comparison was possible. Only 9 out of 69 studies (13%) had an effect estimate that differed by more than 20% from that obtained with a conventional outcome model in all PS analyses presented.

Conclusions

Publication of results based on propensity score methods has increased dramatically, but there is little evidence that these methods yield substantially different estimates compared with conventional multivariable methods.

Introduction

Randomized controlled trials are considered the gold standard for assessing the efficacy of medications, medical procedures, or clinical strategies. Nevertheless, particularly for research on the prevention of chronic disease, randomized trials are often infeasible because of their size, time, and budget requirements, questionable generalizability, and ethical constraints .[1]

On the other hand, non-experimental studies of interventions have frequently been criticized because of their potential for selection bias. This concern reached a crescendo with the disparity in estimated effects of hormone replacement therapy from randomized trials and non-experimental studies.[2] This imbroglio highlighted the need to develop and apply improved methods to reduce bias in non-experimental studies in which selection bias or confounding is likely to occur.[3]

The use of multivariate confounder scores to combine many covariates into a single variable can be traced back to Miettinen in 1976.[4] In 1983, Rosenbaum and Rubin developed the concept of propensity scores (PS) estimated at baseline to control for selection bias in cohort studies.[5] This technique has become popular to control confounding bias in epidemiologic studies that assess the outcomes of drugs and medical procedures. Propensity scores estimate the predicted probability (propensity) of use of a given drug or procedure in a particular subject, based on his or her characteristics when the treatment is chosen. In principle, the effect of the treatment can then be measured among patients who have the same predicted propensity of treatment, thus controlling for confounding.[5] Use of PS to reduce bias is especially appealing since, under the assumption that all relevant predictors of treatment have been adequately captured, subjects with the same PS should have the same chance of receiving treatment. Therefore, PS are often conceptualized as mimicking randomized trials, although they do so only with respect to factors that have been adequately measured. Randomization, in contrast, removes bias from both measured and unmeasured factors. PS allow simultaneous control for confounding by several variables in situations where conventional multivariable models might not be appropriate, owing to the small number of outcomes. PS, however, are frequently used in settings where the outcome is common; their value in this situation is not yet clear. We sought to review the application of PS in the medical literature and to assess its practical value.

Propensity scores

Background

A propensity score (PS) can be defined as the probability of exposure to e.g. a treatment given observed covariates.[5] The score is usually estimated using a multivariable logistic regression model, but can be estimated with a variety of multivariable scoring functions. In a logistic model, the scores range from 0 to 1 and reflect the estimated probability, based on the subject’s characteristics, that the subject will receive the treatment of interest, such that individuals with the same estimated PS have the same chance of receiving treatment. Any two subjects with the same PS can have different values for specific covariates, but overall, covariates entered in the PS model will tend to be balanced for treated and untreated subjects with similar PS. This balance of covariates can easily be checked and the performance of PS to achieve this goal can be clearly communicated, e.g. by presenting the distribution of covariates in exposed and unexposed separately, stratified by quintiles of the PS.

By estimating the PS and analyzing the data within homogeneous levels of PS, in theory one can achieve a ‘virtual randomization’, in which comparable patients are separated into the exposed and unexposed groups. Since PS are estimated using measured data, however, they cannot control for unmeasured or imperfectly measured variables. Therefore, residual systematic bias cannot be excluded.

Once PS are estimated they can be used in various ways to control for selection bias or confounding in non-experimental cohort studies. Possible implementations include matching on the PS, stratified analysis using PS as the stratification variable, and combinations of these two approaches with conventional multivariable outcome modeling. In theory, within each PS stratum, some patients will have received the treatment of interest while others will not. In practice, however, this is not always the case (see figure 1) and how one uses PS in analyses can make a difference.

Figure 1.

Non-overlap of the propensity score distributions among exposed and unexposed subjects*

* In this example subjects with low propensity scores are never exposed while subjects with high propensity scores are always exposed

Alternative ways to apply propensity scores

Matching

One strategy is to match each exposed subject to one or more unexposed subjects with a similar PS, thus avoiding the complexity of matching within multiple strata.[6] A variety of matching methods are available to identify unexposed subjects with PS similar to those of exposed subjects.[7,8] Effective balancing is achieved by any matching procedure that produces good agreement between the mean PS in exposed and unexposed subjects.[9] Selecting an equal number of exposed and unexposed subjects within categories of the PS (frequency matching) instead of individual matching enables the inclusion of exposed subjects for whom no exact unexposed match can be found, but may introduce bias stemming from non-overlapping ranges for exposed and unexposed subjects at the extremes of the distribution of the PS.[6] This bias can be avoided by restricting analyses to the range of PS common to both exposed and unexposed patients, i.e. excluding unexposed patients with a PS lower than the lowest PS observed in exposed patients and excluding exposed patients with a PS higher than the highest PS observed in unexposed patients. Plotting the PS distribution for exposed and unexposed subjects is an easy diagnostic for non-overlap (see figure 1).

Sub-classification on propensity score

As an alternative to matching, one can include all available subjects in an analysis and control for the PS. This can be achieved by simple stratification or modeling of the PS-disease association, e.g. as a continuous covariate. Often, a model with indicator variables for quintiles of the observed PS is used,[10] but control for confounding may be better when PS are modeled as continuous variables.[11] Again, inclusion of all subjects might introduce bias due to inclusion of subjects with a PS from outside the mutual range of scores among exposed and unexposed. Similar to matching, this bias may be reduced by excluding unexposed patients with a PS lower than the lowest PS observed in exposed patients and vice versa.

Combinations with ‘traditional’ multivariate outcome modeling

An issue of some controversy in use of PS is whether better control of confounding can be achieved, and hence better estimates of the effect of the treatment on the outcome can be obtained, by including the PS along with other important predictors of the outcome.[12,13] In theory, confounding can be controlled and a treatment effect can be estimated validly if only one of the two models - the treatment model (PS) or the outcome model (‘traditional’ multivariate modeling) - is specified correctly. Strategies that include both approaches, and thus involve possibly redundant control of confounding, have therefore been called ‘doubly robust’.[14] The theory behind these methods is complex, however, and software tools with adequate documentation are not yet available.

Literature search and abstraction

We identified studies in which the propensity score was used through PubMed and Science Citation Index. Initially, a keyword search was performed through PubMed, identifying studies including the term “propensity”. This broad search yielded 5311 unduplicated references published through December 31, 2003. After review of the abstracts, we identified 167 articles that used propensity score methods in the study of medical interventions and health outcomes (excluding articles focusing solely on methodological or statistical aspects, editorials, review articles or letters, and foreign language articles). To increase the sensitivity of our search, we also searched for articles that cited one of the important propensity score methods articles.[5, 6, 13, 15–17] This search yielded another 73 articles. All these papers were obtained and read by one of the authors. We excluded 48 articles that did not include analysis of data (28), randomized clinical trials (9), case-control studies (2), and articles primarily analyzing cost-effectiveness (6) or practice patterns (3).

Results

Our search revealed 58 substantive medical research studies that used PS in 2003,[18–75] 38 in 2002,[76–113] 28 in 2001, [114–141] 6 in 2000, [142–147] 5 in 1999, [148–152] 5 in 1998, [153–157] and a total of 5 before 1998 [158–162]. Additional articles found through a citation search of the significant methods articles written about PS, using Science Citation Index, yielded 13 medical research studies that used PS in 2003,[163–175]13 in 2002,[176–188] 11 in 2001,[189–199] 3 in 2000,[200–202]1 in 1999, [ 203], 3 in 1998, [204–206] and a total of 3 before 1998 [207–209]. We present the number of studies with results based on PS methods published in each of these years in figure 2.

Figure 2.

Number of medical research studies listed in PubMed and Science Citation Index using propensity score methods to control for confounding according to year of publication

After further review of articles, fifteen articles were excluded from further analysis as the outcomes were continuous and it was not possible to calculate an odds ratio or risk ratio [59, 86, 88, 111, 120, 124, 137, 150, 152, 160, 162, 193, 195, 196, 206]. The final selection of studies abstracted comprised 70 articles from 2003, 48 articles from 2002, 33 articles from 2001, 9 from 2000, 4 from 1999, 7 from 1998 and 6 articles before 1998.

For all selected papers published through 2003 we abstracted the following items: the number of variables used to predict treatment and outcome, respectively; the unadjusted (crude) estimate for the treatment-outcome association; the estimates for the treatment-outcome association adjusted by use of PS matching, PS adjustment, and/or multivariable outcome models, including models without PS and with PS as well as covariates; the predictive value of the PS as assessed by the area under the receiver operating characteristic (ROC) curve (equivalent to the c-statistic in logistic regression); and the percent of exposed participants that could be matched to unexposed participants (where applicable). We extracted or calculated odds ratios or relative risks whenever adequate data were presented.

In table 1, studies published in 2003 are presented. Corresponding tables for the years 2002 and prior years are included in the e-Extra online version. The following results are based on all 177 substantive studies reporting on dichotomous exposures and outcomes published through 2003. Among the medical specialties covered in these papers were cardiology (including cardiac and vascular surgery) (N=90), general internal medicine (N=34), oncology (N=20), nephrology (N=9), psychiatry (N=4), and rheumatology (N=2). The treatments studied included medications (N=60), surgical interventions (N=51), catheterization (N=13),other medical procedures (including care after myocardial infarction and in end-stage renal failure), lifestyle factors, or a wide variety of other comparisons. The main outcome assessed was mortality (N=118). Other outcomes included myocardial infarction (N=6), stroke (N=3), and a wide variety of other outcomes including complications of infection, gastrointestinal events, and emergency hospitalizations.

Table 1.

Substantive studies in 2003 using propensity score methods (PS)^*

	Size	Exposure			Outcome						Multivariable outcome modeling†
Author	N		N	# of variables in model		N	# of variables in model	Crude	PS matching	PS adjustment	without PS	PS and covariates	AUC‡	% matched§
Allen-Ramey, Duong, Goodman, et al.21	960	Inhaled steroids	153	?	Hospitalizations	?	Na	?	-	-	-	1.9 (0.7–4.6)	?	Na
Bhatt, Chew, Lincoff, et al. 26	10,480	Revascularization in acute CAD	4175	?	Mortality	~640	15	?	-	-	0.8 (0.6–1.0)	?	0.84	Na
Horwitz, Berlin, Sauer, et al.165	18,821	Glycoprotein inhibitors	2525	23	Bleeding	215	Na	1.9 (1.4–2.6)	-	1.4 (1.0–2.0)	-	-	?	Na
Rubins, Nelson, Noorbaloochi, et al. 64	16,470	Lipid lowering drugs	7012	31	Mortality	6066	Na	0.5 (0.5–0.5)	-	0.8 (0.7–0.9)	-	0.8 (0.7–0.8)	0.83	Na
Seeger, Walker, Williams, et al.68	8288	Statins	4144	58	MI	325	Na	2.1 (1.5–3.0)	0.7 (0.5–0.9)	-	-	-	0.92	70%
Vakili, Kaplan, Slater, et al.72	10,847	Glycoprotein inhibitors in PTCA	2183	17	Mortality, CABG, MI	507	Na	2.1 (1.7–2.6)	-	1.5 (1.2–1.9)	-	1.6 (1.2–2.0)	?	Na
Schwarz, Smith, Keny, et al.67	61	Intra-OP radiation in pancreatic-CA	30	?	Mortality	~46	Na	“sign. diff.”	-	1.5 (0.8–2.9)	-	1.7 (p=0.3)	?	Na
Ishani, Ibrahim, Gilbertson,et al.49	4046	Residual renal function	424	17	Graft failure	443	13	1.0 (0.8–1.4)	-	-	1.0 (0.7–1.3)	1.0 (0.7–1.3)	?	Na
Cho, Bhatt, Marso, et al.34	7897	Catheterizati on in acute MI	3361	?	Mortality	590	4	0.5	-	0.7 (0.6–0.8)	0.5 (0.1–0.8)	-	0.74	Na
Harbarth, Garbino, Pugin, et al.46	904	Inappropriate AB therapy	211	?	Mortality	250	20	2.0 (1.4–2.8)	-	-	-	1.8 (1.2–2.6)	?	Na
Leon, Solomon, Mueller, et al.56	285	Antidepressa nt therapy	?	11	Recovery	?	Na	?	-	1.9 (1.3–2.7)	-	-	?	Na
Urbach, Bell, Swanstrom,et al.174	1919	Bile bypass in pancreatic CA	945	23	Mortality	1845	24	1.2 (1.1–1.4)	-	1.2 (1.1–1.3)	1.2 (1.1–1.3)	-	0.62	Na
Ascione, Narayan, Rogers, et al.23	250	Off vs on pump CABG	74	38	Mortality	50	Na	2.5 (1.1–5.0)	-	2.0 (0.8–5.0)	-	-	?	Na
Boening, Friedrich, Hedderich, et al.27	650	Off vs on pump CABG	133	14	Mortality	3	Na	?	2.7 (0.2–31)	-	-	-	?	54%
Calafiore, Di Mauro, Canosa, et al.30	1266	Off vs on pump CABG	510	24	Mortality	46	Na	?	0.5 (0.3–1.0)	-	-	0.3 (0.2–0.5)	?	?
Hamamoto, Bando, Kobayashi, et al.45	379	Heart valve surgery	80	?	Mortality	53	6	?	-	-	-	1.0 (0.5–2.0)	?	Na
Kurlansky, Williams, Traad, et al.54	987	Arterial vs venous graft CABG	413	26	Mortality	403	25	?	?	-	0.8 (0.6–1.0)	-	?	?
Srinivasan, Grayson, Pullan,et al.69	?	Preop aspirin in CABG	170	10	Mortality	5	Na	?	0.7	-	-	-	0.68	?
Stamou, Kapetanakis ,Lowery, et al.70	511	Minimally invasive valve surgery	56	17	Blood transfusion	144	Na	0.4 (0.2–0.8)	0.8 (0.4–1.8)	-	-	-	?	100%
Schumacher, Schneider, Reich, et al.66	689	Mistletoe in BRCA	219	?	BRCA relapse	30	?	0.5 (0.2–1.3)	-	-	-	0.3 (0.1–0.8)	?	Na
Aronow, Novaro, Lauer, et al.22	2126	Lipid lowering drugs after coronary intervention	175	7	Persistent use after 6mo	628	7	3.1 (2.8–3.3)	2.3 (2.0–2.5)	2.8 (2.5–3.1)	3.2 (2.9–3.4)	3.2 (2.9–3.4)	0.73	91%
Stenestrand , Wallentin L.173	6891	Fibrinolytic therapy in acute MI	2994	22	Mortality & stroke	2476	29	0.9	-	0.8 (0.8–0.9)	0.9 (0.8–0.9)	0.9 (0.8–0.9)	0.66	Na
Murray, Singer, Dawson, et al.170	11,150	Rehabilitatio n vs no rehabilitation group	4738	112	Discharge	4404	36	0.3 (0.3–0.4)	1.38 (1.29–1.47)	-	-	1.5 (1.4–1.6)	0.82	59%
Bukhari , Wiles, Lunt, et al.29	335	DMARD in RA	82	10	RA progression	Na	Na	2.0 (1.4–2.8)	-	1.5 (1.0–2.2)	-	-	?	Na
Abramov, Tamariz, Fremes, et al.19	2214	Pre OP renal function	507	?	Mortality	?	Na	?	-	2.0 (1.1–3.8)	1.9 (1.3–2.8)	-	?	Na
Abramov, Tamariz, Serrick, et al.20	1820	Pulsatile perfusion in CABG	905	?	Cardiovascular accident	?	4	?	-	-	1.9 (1.1–3.3)	2.2 (1.1–4.4)	?	Na
Martelli, Miceli, De Palo, et al.168	671	Axillary dissection in BRCA	172	6	Mortality (BRCA)	60	7	“no sign. diff.”	-	-	-	1.0 (0.7–1.4)	0.72	Na
Chan, Bhatt, Chew, et al.32	1552	Statins before PTCA	615	22	Mortality & MI	?	3	0.5 (p=0.003)	-	-	-	0.4 (p=0.04)	0.71	Na
Hachamovitch, Hayes, Friedman, et al.43	10,627	Revasc in CAD	671	12	Mortality	146	12	2.3 (1.4–3.7)	-	-	?	?	0.94	Na
Landesberg, Mosseri, Wolf, et al.55	407	Revasc in CAD	74	?	Mortality	?	8	0.8 (0.5–1.3)	-	0.4 (0.2–0.7)	0.5 (0.3–0.9)	-	?	Na
Magee, Coombs, Peterson, et al.58	204, 602	Off vs on pump CABG	17,969	19	Mortality	~ 5860	27	0.8 (0.7–0.9)	0.8 (0.7–1.0)	-	0.8 (0.7–0.8)	-	?	94%
McGuire, Anstrom, Peterson.169	9619	Clinical alert	2644	?	% PTCA after vs before	?	12	0.9 (p=0.06)	-	-	-	1.2 (1.0–1.3)	?	Na
Moss, Humphries, Gao, et al.60	1981	Mitral valve repair vs replacement	338	43	Mortality	?	Na	?	0.5 (0.3–0.8)	-	-	-	?	95%
Trichon, Glower, Shaw, et al.71	2757	Surgery in mitral regurg	278	?	Mortality	~ 1400	?	0.5	?	-	0.6 (0.5–0.7)	-	?	?
Yu, Platt, Lanken, et al.175	1010	Pulmonary artery catheter	275	18	Mortality	404	3	1.3 (1.0–1.7)	0.8 (0.5–1.3)	1.0 (0.6–1.7)	-	-	0.85	51%
Brener, Ellis, Schneider, et al.28	10,471	Abciximab in PTCA	4816	21	Mortality	1504	25	0.9 (0.8–1.0)	-	-	1.0 (0.9–1.1)	?	0.83	Na
Calafiore, Di Mauro, Canosa,et al.30	4381	Off vs on pump CABG	~960	10	Mortality	44	Na	?	0.4 (0.2–0.8)	-	-	-	?	?
Karthik, Musleh, Grayson, et al.50	828	Off vs on pump CABG	411	20	Mortality	31	Na	0.9 (0.5–1.9)	-	0.8 (0.4–1.9)	-	-	0.8	Na
MacDonald, Morant, Goldstein , Burke, et al.167	~1,380, 000	Coxib vs NSAID	~ 1600	18	GI bleeding	8526	18	0.6 (0.2–1.5)	-	0.4 (0.2–1.1)	0.4 (0.1–1.0)	0.4 (0.1–1.0)	?	Na
Dendukuri, Normand, McNeil.36	37,788	Angiography in acute MI	9849	16	Mortality	?	Na	“no sign. diff.”	“no sign. diff.”	-	-	-	?	61%
O'Neill, Iturria, Link, et al.171	97	Surgery for brain Meta	23	25(sic!)	Mortality	80	3	“no sign. diff.”	-	-	1.2 (0.7–2.0)	1.0 (0.5–1.8)	0.89	Na
Chan, Moliterno, Berger,et al.33	4809	Clopidogrel in PTCA	332	8	Mortality	88	>10	0.5 (0.3–0.9)	-	-	-	0.5 (0.3–1.0)	0.64	Na
Johnson,Jin, Quan,et al.166	11,854	Treatment in CHF	546	?	Mortality	2681	18	?	-	“analogous”	0.5 (0.4–0.7)	-	?	Na
Peterson, Pollack, Roe ,et al.62	60,770	Glycoprotein inhib in acute MI	15,379	23	Mortality	~ 4860	?	0.3 (0.3–0.4)	0.9 (0.8–1.0)	-	0.6 (0.6–0.7)	0.9 (0.8–1.0)	0.78	92%
Young-Xu, Chan, Liao, et al.75	371	Satin use	140	24	Well being (depression)	?	25	0.7 (0.5–0.9)	-	-	0.6 (0.4–0.9)	0.6 (0.4–0.9)	0.86	Na
Beddhu, Samore, Roberts,et al. 25	2920	Timing of dialysis	1444	11	Mortality	1070	21	?	-	1.5 (1.3–1.6)	1.4 (1.3–1.4)	1.1 (1.1–1.2)	?	Na
Reddan, Szczech, Tuttle,et al.172	4584	Revasc in CAD	1380	?	Mortality	?	14	?	-	-	-	0.4 (0.3–0.5)	?	Na
Winkelmayer,Ow en, Levin, et al.74	3014	Late referral in ESRF	1039	48	Mortality	1429	?	?	1.4 (1.2–1.6)	-	1.4 (1.2–1.5)	1.3 (1.2–1.5)	0.68	100%
Ezri, Weisenberg, Khazin, et al.37	644	CABG	200	?	Difficult laryngoscopy	43	?	2.0 (p=0.02)	-	-	-	“no sign. assoc.”	?	Na
Krzyzanowska, Weeks, Earle.53	1696	Treatment in pancreatic CA	746	5(?)	Mortality	>> 50%	3	?	-	0.5	0.5 (0.5–0.6)	-	?	Na
Hall, Summers, Obenchain.44	11,443	Insulin (Lispro vs regular)	3341	25	Health care utilization (hosp)	678	Na	?	0.7 (0.6–0.8)	-	-	-	?	55%
Abbott, Trespalacios, Agodoa.18	993	AV-fistula in dialysis	257	12	CHF	462	?	?	-	-	“almost same”	0.9 (0.6–1.1)	0.70	Na
Banbury, White, Blackstone, et al.24	357	Vacuum assisted venous return	150	6	Blood usage	94	4	0.4 (0.2–0.6)	-	-	0.3 (0.2–0.6)	0.2 (0.1–0.5)	0.79	Na
Flameng, Herijgers, Dewilde, et al.38	1098	Cardioplegy in cardiac surgery	504	36	Mortality	74	6	0.6 (0.4–1.0)	-	-	0.4 (0.2–0.8)	0.4 (0.2–0.9)	?	Na
Gillinov, Faber, Houghtaling, et al.40	679	Mitral valve repair	232	8	Mortality	30	Na	0.6 (0.3–1.2)	-	-	-	1.0 (0.4–2.4)	0.86	Na
Gillinov, Blackstone, Cosgrove,et al.163	813	Mitral valve repair vs double replacement	295	13	Mortality	52	11	0.8 (.4–1.4)	-	-	-	-	.81	Na
Koch, Khandwala, Nussmeier, et al.52	15,597	Female gender in CABG	3596	68	Mortality	257	9	1.7 (1.3–2.2)	0.9 (0.5–1.7)	-	1.3 (1.0–1.8)	“not sign.”	?	26%
Murthy, Law, Whooley,et al.61	921	Atrial fib after esophagecto my	198	4	Mortality	42	Na	-	4.5 (2.0–9.7)	-	-	-	0.7	73%
Rice, Adelstein, Chidel, et al.63	83	Chemo in esophageal CA	31	15	Mortality	~54	Na	0.7	0 (no treated cases)	-	-	-	?	65%
Schmitz, Weinreich, White, et al.65	582	Intra-aortic filtration in CABG	278	9	Stroke	46	6	0.4	“no diff.”	-	“no diff.”	0.4 (0.2–0.8)	0.64	81%
Girou, Brun-Buisson, Taille, et al.41	479	Non-invasive ventilation	166	6	Mortality	53	4	0.5 (0.3–0.9)	-	-	0.4 (0.2–0.8)	0.4 (0.2–0.8)	?	Na
Gregg, Cauley, Stone,et al.164	3009	Start of physical activity	811	?	Mortality	~400	11	?	-	“similar”	0.5 (0.4–0.7)	-	?	Na
Grzybowski, Clements, Parsons, et al.42	19,917	Revasc in acute MI	4705	23	Mortality	5173	Na	0.3 (0.3–0.3)	0.5 (0.4–0.6)	-	-	-	0.80	83%
Vikram, Buenconsejo, Hasbun, et al.73	513	Valve surgery	230	12	Mortality	131	7	0.4 (0.3–0.6)	0.5 (0.2–0.9)	0.5 (0.2–0.9)	0.4 (0.2–0.5)	-	0.86	47%
Macaubas, de Klerk , Holt.57	407	Antenatal cytokines	110	?	Asthma	138	Na	0.7 (0.5–1.2)	-	0.6 (0.4–1.0)	-	-	?	Na
Frolkis, Pothier, Blackstone, et al.39	29,244	Ventricular ectopy	1080	30	Mortality	1862	30	2.4 (2.0–2.9)	1.5 (1.1–1.9)	-	1.6 (1.3–1.9)	-	0.80	99%
Katzan, Cebul, Husak, et al.51	11,286	Pneumonia in stroke	635	38	Mortality	644	20	6.1	-	-	4.4 (3.7–5.2)	3.0 (2.4–3.7)	0.83	Na
Christakis, Iwashyna. 35	195,553	Hospice use in widowers	30, 916	28	Mortality	4191	Na	?	0.9 (0.8–1.0)	-	-	-	?	100%
Heuschmann, Berger, Misselwitz, et al.47	13,440	Thrombolysis in stroke	384	14	Mortality	486	Na	2.8 (2.0–3.9)	1.7 (1.0–2.8)	-	-	-	0.94	98%
Hughes, Evans, Lightfoot, et al.48	1228	Transfusion in infection	587	7	Mortality	218	12	3.6 (2.6–5.0)	-	-	-	0.9 (0.5–1.5)	?	Na

^*From PubMed search and Science Citation Index.

^†Numbers presented are odds ratios, relative risks, or incidence rate ratios with their corresponding 95% confidence intervals (in brackets)

^‡Area under the receiver operating characteristics (ROC) curve (c-statistic) from logistic regression model used to estimate the PS

^§Percent of all exposed subjects for whom an unexposed subject could be matched on the PS; exposed subjects without match are discarded from analysis; Na = not applicable (no matching)

^¶qualitative statement; data not presented

The number of exposed subjects (or unexposed subjects, if this number was smaller) ranged from 61 to over 1,380,000, and the number of outcomes ranged from 23 to 285,965. In 109 studies, the number of exposed subjects was larger than the number of subjects who experienced the outcome; in 13 studies it was smaller. To estimate the PS, 2 to 112 variables were used (in those papers in which this information was presented), compared with 1 to 45 used in multivariable outcome models. Direct comparison of the number of variables was possible in 90 studies, of which only 51 used more variables to estimate the PS than to estimate the corresponding outcome model; 27 used fewer variables to do so. Sixty-five studies had fewer than 8 outcomes for each variable entered into the PS model, i.e. a setting where the use of PS methods was shown to be advantageous compared with conventional outcome modeling.[210] In 60% of studies (96 out of 161) the number of outcomes would have been sufficient to enter all variables used in the propensity score model in the corresponding outcome model.

The area under the receiver operating characteristic (ROC) curve or c-statistic was presented for 73 studies. It ranged from 0.56 to 0.93, indicating poor to good predictive power. The lowest predictive value (c=0.56) was achieved predicting the annual volume of patients treated by admitting physicians (in a study assessing its association with mortality in acute myocardial infarction);[199] the highest (c=0.94) was achieved when predicting revascularization in coronary artery disease [43] and thrombolysis in patients with stroke [47]. Very high values (c > 0.90) were reported in six additional studies for treatments including statins [68], amiodarone after acute myocardial infarction [87], chemotherapy in colon cancer [85], heart valve repair vs. replacement [122], bilateral thoracic artery bypass [148], and a hospital comparison [90].

Fifty-one studies used matching on the PS as either the main analytic strategy or as one of several analytic strategies presented. The percentage of exposed participants that could be matched to an unexposed participant was presented for 49 studies, and ranged from 26% to 100% (median=90%).

Most studies showed clear evidence of confounding, with substantial changes in the point estimate after adjustment. Whether PS methods or conventional outcome models were used to control for confounding, however, seemed to matter little in most of those 69 studies in which such a comparison was possible. These included 10 studies in which the authors made a qualitative statement that (mostly PS) analyses showed “similar” results.

In 20% of studies (14 out of 69), however, there was a more than 20% difference in the point estimate obtained from the conventional outcome model compared with any propensity score method presented.[22, 24, 34, 51, 52, 62, 73, 100, 102, 105, 107, 121, 123, 192] We used this arbitrary cut-point as a marker of a substantial difference in results. Of these, 5 [22, 100, 107, 121, 123] showed results not meeting our 20% criterion for at least one of the analytic strategies using PS. In four of these studies, the PS strategy not meeting criterion was when the PS was added to the conventional multivariable outcome model [22, 100, 107, 123]. In the study by Foody et al. [121], the result matched on the PS did not meet our criterion. This left the remaining 13% of studies (9 out of 69) in which all PS analyses presented showed a substantial difference compared with conventional outcome models.

Discussion

The number of studies using PS methods, though not yet large, is climbing rapidly.[211] According to the authors of many of these studies, the main reason to use PS methods was better control for confounding compared with conventional multivariable outcome modeling. We found no empirical evidence, however, that PS analyses controlled confounding more effectively than conventional outcome modeling in the majority of the studies where results from both methods were presented. Potentially meaningful differences in the control of confounding were observed in less than 15% of studies. Since the true underlying association is unknown, it remains unclear whether these differences are due to better control for confounding using the PS or whether adjusting for an inaccurate PS distorted results in some studies.[212,213] The use of PS as the only analytic technique applied comes at the price of losing potentially useful information about predictors of outcome. It therefore seems desirable to use PS only if a reduction in bias or an improvement in efficiency can be achieved.

Cook and Goldman compared the performance of tests of significance under the null hypothesis (i.e. assuming no difference between treatments) for PS and for ‘traditional’ multivariable outcome models using simulations.[214] PS appeared to produce valid results in most circumstances, but were biased in situations with very strong treatment-confounder associations.

In some practical situations the choice of analytic method will be limited. Because 10 events per covariate is usually considered to be a minimum requirement for stable estimates in multivariable models,[215,216] PS analyses combining multiple covariates into a single score are especially desirable if the treatment is common and the outcome is rare.[217,218] A recent simulation study comparing PS with multivariable outcome models concluded that PS performed better in situations with less than 8 outcomes per covariate.[210] Apart from this specific condition (relevant in 65 of the 161 studies presenting the necessary information), there is little if any practical guidance for researchers regarding when the use of PS will produce different, and in particular, better estimates compared with conventional multivariable outcome models.

PS are used to reduce bias. Drake observed that the magnitude and direction of bias resulting from omitting an important confounder from analysis was similar in multivariable outcome modeling and in estimating the treatment-outcome relation controlling for PS.[219] This observation implies that PS may not be superior to conventional multivariable outcome models in controlling bias from unobserved confounders.

Several strategies for using PS are currently being applied in medical research, and often the results from more than one of these strategies are reported in a single paper. Individual matching on a PS has intuitive appeal and in those studies that used matching, the proportion of exposed subjects that could be matched ranged from 26% to 100%. Excluding a large proportion of exposed subjects because of a lack of unexposed matches, however, may severely alter the composition of the study population. Because comparisons may be valid within that altered population, we would therefore not call this issue a bias. Nevertheless, it is essential to appreciate and to describe clearly the differences between this altered population and the original study population. On the other hand, including subjects with a PS outside the overlapping range, such as using conventional outcome modeling or PS methods including non-overlapping ranges, can lead to bias due to model extrapolation or smoothing. Such subjects might include, for example, patients with absolute indications or contraindications to treatment, who should not be included in any treatment comparison,[220] but are usually not recognized using conventional multivariable outcome modeling. Since this is a clear advantage of PS, a graphical exploration similar to figure 1 could be used as a routine procedure before doing any multivariable outcome modeling in treatment comparisons. Unfortunately, systematic comparisons of the different strategies to apply PS with respect to validity and efficiency with specific attention to exclusion of participants and non-linear associations between the PS and the outcome are sparse.[221]

Variable selection in constructing PS is at present an ad hoc process that lacks guidelines and well-understood model diagnostics. The area under the ROC curve or c-statistic (from logistic regression) to quantify the predictive power of a model is a well established concept in clinical epidemiology.[222] Its value when assessing the performance of PS to control confounding is unclear, however. Indeed, a very high c-statistic can indicate considerable non-overlap in PS distributions between exposed and unexposed as shown in figure 1.

Some authors argue that variables that only predict treatment choice but are not associated with the study outcome should not be included in the PS.[223] By definition, these are not confounders, but they may increase the area under the ROC curve and thereby erroneously imply a high validity of the PS analysis.

A practical way of assessing the value of the PS model in controlling for confounding is to check the balance of important risk factors for the outcome between exposed and unexposed within levels of the estimated PS. This method has the advantage of being driven by substantive knowledge rather than statistics, and the results can easily be communicated to the reader in a table. It allows direct assessment of comparability of exposed and unexposed by the reader, a clear advantage of using PS methods compared with the ‘black box’ of the conventional outcome model.

This review of the application of propensity score methods in the medical literature has several limitations. We may well have missed some studies by using a specific search strategy, but this problem should not affect the comparison over time. Important information in understanding similarities and differences between the analytic approaches, including description of the types of variables, variable selection procedures, and measures of model adequacy, could not be abstracted systematically, since these are rarely presented with sufficient detail in published papers.

In conclusion, methods using propensity scores may be good candidates for improving inference in non-experimental studies, but a better understanding of the benefits and limitations of these methods in practical circumstances is needed. Meanwhile, propensity scores, like any other method, should not be automatically regarded as a preferable and sole method to control for confounding in non-experimental research, but rather as a promising addition.