Abstract
Purpose
Angiotensin-converting enzyme (ACE) inhibitors are recommended for patients with chronic kidney disease (CKD) because they slow disease progression. But physicians’ concerns about the risk of hyperkalemia (elevated serum potassium level), a potentially fatal adverse effect, may limit optimal management with ACE-inhibitors. We synthesized known predictors of hyperkalemia into a prognostic risk score to predict the risk of hyperkalemia.
Methods
We assembled a retrospective cohort of adult patients with possible CKD (at least one estimated glomerular filtration rate (eGFR) value less than 60 mL/min/1.73m2) who started an ACE-inhibitor (i.e., incident users) between 1998 and 2006 at a health maintenance organization. We followed patients for hyperkalemia: (1) potassium value > 5.5 mmol/L; or, (2) diagnosis code for hyperkalemia. Cox regression synthesized a priori predictors recorded in the electronic medical record into a risk score.
Results
We followed 5,171 patients and 145 experienced hyperkalemia, a 90-day risk of 2.8%. Predictors included: age, eGFR, diabetes, heart failure, potassium supplements, potassium-sparing diuretics, and a high dose for the ACE-inhibitor (lisinopril). The risk score separated high-risk patients (top quintile, observed risk of 6.9%) from low-risk patients (bottom quintile, observed risk of 0.7%). Predicted and observed risks agreed within 1% for each quintile. The risk increased gradually in relation to declining eGFR with no apparent threshold for contraindicating ACE-inhibitors.
Conclusions
The risk score separated high-risk patients (who may need more intensive laboratory monitoring) from low-risk patients. The risk score should be validated in other populations before it is ready for use in clinical practice.
Keywords: hyperkalemia, chronic kidney disease, ACE-inhibitors, risk score, cohort study, adverse effects
Introduction
Angiotensin-converting enzyme (ACE) inhibitors are recommended for patients with chronic kidney disease (CKD) because they slow disease progression.1,2 But physicians’ concerns about the risk of hyperkalemia (elevated serum potassium level), a potentially fatal adverse effect, may limit optimal management with ACE-inhibitors.3–6
Although possible predictors of hyperkalemia have been reported, physicians lack a tool that can integrate a patient’s medical history, including current drug therapies and comorbid conditions, to help them identify patients at higher risk of hyperkalemia. Prognostic risk scores can identify patients at higher risk of suffering an adverse effect; for example, investigators have developed a risk score to predict the risk of cough among patients starting ACE-inhibitors.7,8
We integrated known predictors of hyperkalemia into a risk score to predict the risk of hyperkalemia within 90 days after starting an ACE-inhibitor in a population with possible CKD. To our knowledge, this is the first risk score that predicts hyperkalemia in this patient population.
Methods
Objective
We conducted a retrospective cohort study of patients with possible CKD who started an ACE-inhibitor. Our objective was to develop a prognostic risk score to predict the risk of hyperkalemia using patient characteristics documented during routine practice before patients started ACE-inhibitor therapy.
Setting
We conducted the study in a group-model health maintenance organization, Kaiser Permanente Northwest (KPNW), which serves the Portland, Oregon and Vancouver, Washington metropolitan area. KPNW has a membership of approximately 450,000 people on any given day. The study was reviewed and approved by KPNW’s Human Subjects Committee.
Identification of Patients and Follow-up
We identified a cohort of patients with an indication of possible CKD who started an ACE-inhibitor—incident users. We sought to design a pragmatic risk score that would fit the practice patterns observed in routine care.9 As we were selecting the cohort, we discovered that many providers at KPNW started ACE-inhibitors after testing serum creatinine only once; they appeared to treat possible kidney disease empirically without waiting to confirm that the patient had chronic kidney disease. Consequently, our estimate of kidney function was based on one outpatient laboratory test for serum creatinine (used to estimate glomerular filtration rates, GFRs, with the four-variable Modification of Diet in Renal Disease (MDRD) Study equation, but omitting race) and we chose the most recent, baseline serum creatinine value.10 Serum creatinine tests, the basis for calculating eGFR, were collected before patients started their ACE-inhibitor. The sole serum creatinine used to define eGFR could have been tested up to one year before the start of ACE-inhibitor therapy. We excluded patients with any use of ACE-inhibitors or angiotensin II-receptor antagonist therapies during the one-year baseline period to ensure that the cohort represented incident users. Patients with an eGFR<60mL/min/1.73 m2 between January 1998 and December 2006 were enrolled if they filled a new ACE-inhibitor prescription within one year, which defined time zero and the beginning of follow-up.
Because lisinopril accounted for 99% of all new ACE-inhibitor fills, we restricted the cohort to lisinopril users to simplify the calculation of dose, one of the predictors. Potassium values as well as hyperkalemia diagnoses (ICD-9-CM 276.7) observed between one day and 90 days after the start of lisinopril were used to determine the risk of hyperkalemia. We considered patients censored when they died, progressed to renal replacement therapy, or discontinued insurance coverage with KPNW.
Data Collection and Other Eligibility Criteria
We measured characteristics that have predicted hyperkalemia in other studies by using the coded information in the electronic medical record, the laboratory values, and the drug prescription fills at KPNW. Characteristics were measured during the one-year baseline period (i.e., before time zero). All patients met the following eligibility criteria:
30 to 89 years at the time the patient started lisinopril (i.e., time zero)
Contributed at least one year of continuous membership in KPNW before time zero
Maintained prescription drug coverage through KPNW for the year before time zero
Had no history of renal replacement therapy preceding time zero (looked back to 1997)
Had no history of prior ACE-inhibitor or angiotensin II-receptor antagonist use (looked back to 1997)
Lisinopril was not part of combination therapy (e.g., hydrochlorothiazide)
Outcome
We analyzed the time to first occurrence of hyperkalemia, defined as a composite outcome:
outpatient serum potassium level greater than 5.5 mmol/L;
outpatient encounter, emergency department visit, or hospitalization with a diagnosis code of hyperkalemia in any position, not necessarily the primary code.
Selection of Candidate Predictor Characteristics
We identified predictors of hyperkalemia from a review article on the occurrence of hyperkalemia in patients treated with ACE-inhibitors,3 and asked our nephrologists to rank the strength of those predictors, a proxy for their capacity to discriminate which patients would experience hyperkalemia. We also considered how easily and reliably we could measure the characteristics using retrospective databases and the prevalence of the characteristics in our population. For example, a history of hyperkalemia may be a strong predictor, but only 1% of the patients in our cohort had a history of hyperkalemia (using the same algorithm that we used to define the outcome), so we did not consider that characteristic as a candidate in the regression. To avoid over-fitting our risk score to the development data, we limited the number of candidate characteristics and their degrees of freedom; specifically, we required at least 10 events of hyperkalemia per degree of freedom.11 Consequently, we had an insufficient number of hyperkalemia events to evaluate all possible predictors that could be reliably measured (e.g., current use of diuretics, such as hydrochlorothiazide, which reduce the risk of hyperkalemia). Our final list of a priori candidate characteristics included: age, eGFR, diabetes, heart failure, potassium supplements, potassium-sparing diuretics, and a high dose for the ACE-inhibitor (lisinopril).
Statistical Analysis
Using Cox regression, we evaluated patient characteristics that might predict hyperkalemia. Categorical variables were modeled using indicator variables. Continuous variables were modeled using restricted cubic splines. The number and location of the knots was selected using methods proposed by Harrell.12 The number of knots was assigned based on sample size, the number of events, and the available degrees of freedom relative to the expected explanatory power of each variable. Since there was no a priori assumption about the location of change points in associations, the location of the knots was chosen by default to be equally spaced, with the first knot at the 5th percentile and the last at the 95th percentile. We observed too few hyperkalemia events to evaluate interactions between characteristics. We retained characteristics in the model if they improved the accuracy of the predictions and were statistically independent predictors (P<0.05 according to the likelihood ratio test).
We translated coefficients from the Cox regression into a points-based system or risk score where a higher number of points means a higher risk of hyperkalemia.13 The coefficients from the Cox model were translated into a point-based score as follows. First, the linear predictor in the Cox model was mapped to the corresponding 90-day risk of hyperkalemia. Then, the components of the linear predictor were rescaled to an arbitrary axis where the lowest-risk category for each variable was scored as zero points with increasing points counted for proportionate increases in the linear predictor. More points mean a higher risk of hyperkalemia. Each unit increase in the linear predictor, corresponding to a hazard ratio of 2.7183, equated to 48.90 points. For example, patients with a history of heart failure were 2.92 times as likely to experience hyperkalemia, which corresponded to 52 points. Patients had to score at least 84 points to place in the highest risk quintile (i.e., 80th percentile or higher). The Framingham Heart Study published a tutorial on how to convert regression coefficients into risk score points.13
We also calculated the observed risk of hyperkalemia for each quintile of patients’ predicted risks of hyperkalemia to measure discrimination and calibration; we tested calibration using the Hosmer-Lemeshow statistic.14 Observed and predicted risks were plotted using risk predictiveness curves that show the cohort’s distribution of predicted risks. 15
Results
Of the 5,171 patients, 145 experienced hyperkalemia within 90 days of starting lisinopril, a risk of 2.8% (95% CI, 2.4% to 3.3%). An identical number of patients (n=145) were lost to follow-up. We classed patients without a potassium test or diagnosis code for hyperkalemia in the first 90 days (27%) as not having experienced hyperkalemia.
Table 1 shows patients’ baseline characteristics for the entire cohort and the subgroups that experienced hyperkalemia or were lost to follow-up. On average, patients were in their early seventies with stage 4 CKD (i.e., an eGFR between 59 and 30 mL/min/1.73m2, and many patients had another indication for an ACE-inhibitor that increased their risk of hyperkalemia: 26% had diabetes and 18% had heart failure. Most patients (89%) started on a low dose (10 mg) of lisinopril. Few patients (1%) had a history of hyperkalemia in the year before they started lisinopril.
Table 1.
Baseline characteristics for all patients in the cohort, the subgroup who experienced hyperkalemia, and the subgroup lost to follow-up during the first 90 days after starting lisinopril
| Characteristics | All patients n=5,171 |
Patients with hyperkalemia n=145 |
Patients lost to follow-up n=145 |
|---|---|---|---|
| Age in years, mean (SD) | 71.1 (11.6) | 74.1 (11.6) | 72.9 (14.4) |
| Age in years, n (%) | |||
| 30 to 39 | 68 (1.3) | 0 (0) | 6 (4.1) |
| 40 to 49 | 207 (4.0) | 9 (6.2) | 6 (4.1) |
| 50 to 59 | 525 (10.2) | 9 (6.2) | 13 (9.0) |
| 60 to 69 | 1,255 (24.3) | 25 (17.2) | 22 (15.2) |
| 70 to 79 | 1,725 (33.4) | 42 (29.0) | 33 (22.8) |
| 80 to 89 | 1,391 (26.9) | 60 (41.4) | 65 (44.8) |
| Men, n (%) | 1,922 (37.2) | 62 (42.8) | 51 (35.2) |
| Kidney function, mL/min/1.73m2, eGFR, mean (SD) | 50.2 (8.5) | 45.7 (10.1) | 48.2 (10.0) |
| Kidney function, mL/min/1.73m2, eGFR, n (%) | |||
| 59 to 45 | 4,025 (77.8) | 90 (62.1) | 105 (72.4) |
| 44 to 30 | 995 (19.2) | 44 (30.3) | 31 (21.4) |
| 29 to 15 | 144 (2.8) | 11 (7.6) | 9 (6.2) |
| ≤15 | 8 (0.2) | 0 (0) | 0 (0) |
| History of hyperkalemia, n (%) | 51 (1.0) | 3 (2.1) | 6 (4.1) |
| Diabetes, n (%) | 1,354 (26.2) | 52 (35.9) | 36 (24.8) |
| Heart failure, n (%) | 932 (18.0) | 63 (43.5) | 56 (38.6) |
| Starting dose for the ACE-inhibitor, lisinopril (n, %) | |||
| Low 10 mg per day | 4,609 (89.1) | 127 (87.6) | 123 (84.8) |
| High >10 mg per day | 562 (10.9) | 18 (12.4) | 22 (15.2) |
| Current use of potassium supplements, n (%) | 1,336 (25.8) | 57 (39.3) | 53 (36.6) |
| Current use of aldosterone-receptor blockers or potassium-sparing diuretics: spironolactone,* eplerenone, amiloride, triamterene (n, %) | 1,292 (25.0) | 41 (28.3) | 29 (20.0) |
| Current use of thiazide or loop-diuretics: hydrochlorothiazide, chlorthalidone, furosemide, bumetanide, metolazone, n (%) | 1850 (35.8) | 75 (51.7) | 71 (49.0) |
Spironolactone use was rare (3.1% of the total cohort) and no one used eplerenone.
Table 2 shows the seven characteristics that were retained in the Cox regression model and that contributed to the risk score. Because of the limited number of hyperkalemia events and the need for 12 degrees of freedom to represent the seven characteristics, we did not evaluate additional characteristics. The first column of the Table shows the prevalence of the characteristic and the second column shows the number of risk score points assigned to each level of a characteristic. The risk increased gradually in relation to declining eGFR with no apparent threshold for contraindicating ACE-inhibitors. However, patients with an eGFR < 30 mL/min/1.73m2 accounted for only 3% of the cohort and only 11 hyperkalemia events. Consequently, we have limited data to assess the shape the relation beyond stage 3 CKD.
Table 2.
Patient characteristics and the number of risk score points assigned by the Cox regression model to predict hyperkalemia during the first 90 days after starting lisinopril
| Characteristics | Prevalence n=5,171 |
Risk points |
|---|---|---|
| Age in years, n (%) | ||
| 30 to 34 | 21 (0.4) | * |
| 35 to 39 | 47 (0.9) | * |
| 40 to 44 | 87 (1.7) | 11 |
| 45 to 49 | 120 (2.3) | 8 |
| 50 to 54 | 236 (4.6) | 6 |
| 55 to 59 | 289 (5.6) | 3 |
| 60 to 64 | 573 (11.1) | 1 |
| 65 to 69 | 682 (13.2) | 0 |
| 70 to 74 | 803 (15.5) | 1 |
| 75 to 79 | 922 (17.8) | 6 |
| 80 to 84 | 875 (16.9) | 14 |
| 85 to 89 | 516 (10.0) | 23 |
| Kidney function in mL/min/1.73m2, eGFR, n (%) | ||
| 59 to 55 | 2,027 (39.2) | 0 |
| 54 to 50 | 1,195 (23.1) | 15 |
| 49 to 45 | 802 (15.5) | 28 |
| 44 to 40 | 489 (9.5) | 38 |
| 39 to 35 | 332 (6.4) | 46 |
| 34 to 30 | 174 (3.4) | 54 |
| 29 to 25 | 85 (1.6) | 62 |
| 24 to 20 | 42 (0.8) | 69 |
| 19 to 15 | 17 (0.3) | 77 |
| ≤15 | 5 (0.1) | * |
| Diabetes, n (%) | ||
| No diabetes | 3,817 (73.8) | 0 |
| Diabetes | 1,354 (26.2) | 29 |
| Heart failure, n (%) | ||
| No heart failure | 4,239 (82.0) | 0 |
| Heart failure | 932 (18.0) | 52 |
| Starting dose for the ACE-inhibitor, lisinopril, n (%) | ||
| Low 10 mg per day | 4,609 (89.1) | 0 |
| High >10 mg per day | 562 (10.9) | 11 |
| Current use of potassium supplements, n (%) | ||
| No use | 3,835 (74.2) | 0 |
| Any use | 1,336 (25.8) | 6 |
| Current use of aldosterone-receptor blockers or potassium-sparing diuretics: spironolactone, eplerenone, amiloride, triamterene, n (%) | ||
| No use | 3,879 (75.0) | 0 |
| Any use | 1,292 (25.0) | 14 |
Cannot assign risk score points because no hyperkalemia events were observed
The Figure shows the risk predictiveness curve based on the risk score along with the observed risks of hyperkalemia for each quintile of predicted risk. The close agreement between the predictions (curve) and the observed risk (open circles) reflects successful calibration, which we confirmed with the Hosmer-Lemeshow test (P=0.99). Patients in the bottom quintile of predicted risk have an observed risk of less than 1%. The risk increases gradually for patients in the middle three quintiles and then increases markedly for patients in the top quintile: Their observed risk is ten times higher than patients in the bottom quintile, which demonstrates effective discrimination.
Figure.
The risk predictiveness curve (solid line) shows patients’ (n=5,171) predicted risk of hyperkalemia according to the risk score based on the following characteristics: age, eGFR, diabetes, heart failure, starting dose of lisinopril, current use of potassium supplements, and current use of potassium-sparing diuretics. Within each quintile of predicted risk, noted on the x-axis, the open circles show the observed risks of hyperkalemia. The risk score’s accuracy (calibration) is measured by the agreement between the predicted and observed risks.
Physicians could choose any cut-point from the risk score to target patients for more intensive monitoring of potassium values. The statistical consequences of that choice depend on the population and setting. If physicians at Kaiser Permanente Northwest chose the 80th percentile of the risk score (i.e., the top quintile of patients), they would focus their attention on patients who experienced nearly half of the hyperkalemia events (sensitivity=49.0%; 95% CI, 40.6% to 57.4%). The specificity for that cut-point was 80.8% (95% CI, 79.7% to 81.9%). Seven percent of those patients in the top quintile developed hyperkalemia (positive predictive value = 6.9%; 95% CI, 5.4% to 8.6%). The negative predictive value was 98.2% (95% CI, 97.8% to 98.6%). To score in the top quintile, a patient would require at least 84 risk points (on an arbitrary scale). A typical patient in our cohort (age 71 years) who started on a low dose of lisinopril would require kidney function near 30 mL/min/1.73m2, and a condition such as diabetes to score 84 points (age 71 (1 point) + low dose lisinopril (0 points) + eGFR=30 (54 points) + diabetes (29 points) = 84 points).
Discussion
We developed a risk score to predict the 90-day risk of hyperkalemia in patients with possible CKD who started lisinopril. Seven patient characteristics identified the higher-risk patients accurately. Physicians could use the risk score to target higher-risk patients for more intensive monitoring of potassium values. The risk score may offer a more accurate risk management strategy than asking physicians to consider a patient’s individual predictors of hyperkalemia without the benefit of a statistical algorithm that accounts for their correlation, a hypothesis that could be tested in a randomized controlled trial.16,17
We designed the risk score to work with the data that physicians collect in routine practice to improve the relevance of its predictions. For example, we measured eGFR using only one serum creatinine value instead of the two values recommended by the National Kidney Foundation guidelines.7 At our HMO, physicians often started lisinopril after a single serum creatinine test instead of waiting to confirm that the kidney disease was chronic. Although we improved the relevance to routine practice, eGFR in the risk score may not predict as strongly as it would if it were measured more reliably with two tests. Similarly, drugs known to increase the risk of hyperkalemia were measured in the 90 days before the start of lisinopril; their use may not have persisted after the start of lisinopril, which would weaken their ability to predict hyperkalemia. The accuracy of the risk score’s predictions and the low incidence of hyperkalemia (2.8%) mean that most patients designated as “high-risk” will be false positives who do not develop hyperkalemia in the first 90 days of therapy (i.e., the positive predictive value is low).
We were surprised that the risk of hyperkalemia appeared to increase modestly at younger ages. Younger age may be a marker for other characteristics that increase the risk of hyperkalemia, which were omitted from the risk score because we had too few hyperkalemia events. The number of points awarded to patients at younger ages is small and unlikely to reclassify patients into different risk quintiles (e.g., only 8 points for patients in their late forties).
We lacked race data for most patients, so we estimated GFR without considering race, which means we under-estimated GFR in black patients; consequently, some black patients would not have been eligible for our cohort according the criterion that required an eGFR< 60 mL/min/1.73m2. Based on unpublished surveys of the KPNW members, we suspect that fewer than 5% of the patients in the cohort were black and that any bias is modest.
One limitation is that we had insufficient data to predict diagnoses of hyperkalemia separately from elevated serum potassium levels (without a diagnosis). Of the 145 patients with hyperkalemia, 97 had the ICD-9-CM code for hyperkalemia (and 20 of those patients also had an outpatient serum potassium value greater than 5.5 mmol/L).
Others have noted that randomized controlled trials of patients with chronic kidney disease assigned to ACE-inhibitors rarely report the absolute risk of hyperkalemia,18 but a recent report by investigators from the African American Study of Kidney Disease and Hypertension (AASK) trial is an exception.6 Patients assigned to the ACE-inhibitor arm (ramipril) experienced 2.5 hyperkalemia events per 100 person-years (95% CI, 1.7 to 3.5) during an average of three years of follow-up. The AASK rate may be lower than the rate we observed at KPNW because AASK excluded patients with diabetes and heart failure. We are aware of only one other community-based cohort study that reported the absolute risk of hyperkalemia in patients using ACE-inhibitors, a study of US Veterans.19 The study designs differ too markedly to allow for meaningful comparisons of the absolute incidence of hyperkalemia.
Before the risk score can be used in other populations and settings—beyond KPNW—its findings should be validated clinically; those findings might indicate a need to recalibrate the points in relation to a lower or higher absolute risk of hyperkalemia in the new population.20 It’s too early to recommend that our prognostic risk score be integrated into clinical practice because it requires validation in other populations and settings.21 Pending successful validation, the impact of the risk score on patient management and outcomes should be compared with usual care in a randomized controlled trial.17
Acknowledgments
Our study and manuscript were sponsored by the Oregon Clinical and Translational Research Institute (OCTRI), grant number UL1 RR024140 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), and NIH Roadmap for Medical Research.
Footnotes
The authors have no conflicts of interest to report.
References
- 1.Crowe E, Halpin D, Stevens P. Early identification and management of chronic kidney disease: summary of the NICE guidance. BMJ. 2008;337:a1530. doi: 10.1136/bmj.a1530. [DOI] [PubMed] [Google Scholar]
- 2.Kent DM, Jafar TH, Hayward RA, et al. Progression risk, urinary protein excretion, and treatment effects of angiotensin-converting enzyme inhibitors in nondiabetic kidney disease. J Am Soc Nephrol. 2007;18:1959–1965. doi: 10.1681/ASN.2006101081. [DOI] [PubMed] [Google Scholar]
- 3.Palmer BF. Managing hyperkalemia caused by inhibitors of the renin-angiotensin-aldosterone system. N Engl J Med. 2004;351:585–592. doi: 10.1056/NEJMra035279. [DOI] [PubMed] [Google Scholar]
- 4.Hebert LA. Optimizing ACE-inhibitor therapy for chronic kidney disease. N Engl J Med. 2006;354:189–191. doi: 10.1056/NEJMe058295. [DOI] [PubMed] [Google Scholar]
- 5.Parfrey PS. Inhibitors of the renin-angiotensin system: proven benefits, unproven safety. Ann Intern Med. 2008;148:76–77. doi: 10.7326/0003-4819-148-1-200801010-00191. [DOI] [PubMed] [Google Scholar]
- 6.Weinberg JM, Appel LJ, Bakris G, et al. Risk of hyperkalemia in nondiabetic patients with chronic kidney disease receiving antihypertensive therapy. Arch Intern Med. 2009;169:1587–1594. doi: 10.1001/archinternmed.2009.284. [DOI] [PubMed] [Google Scholar]
- 7.Moons KGM, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? BMJ. 2009;338:b375. doi: 10.1136/bmj.b375. [DOI] [PubMed] [Google Scholar]
- 8.Morimoto T, Gandhi TK, Fiskio JM, et al. Development and validation of a clinical prediction rule for angiotensin-converting enzyme inhibitor-induced cough. J Gen Intern Med. 2004;19:684–691. doi: 10.1111/j.1525-1497.2004.30016.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Thorpe KE, Zwarenstein M, Oxman AD, et al. A pragmatic—explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62:464–475. doi: 10.1016/j.jclinepi.2008.12.011. [DOI] [PubMed] [Google Scholar]
- 10.Levey AS, Coresh J, Balk E, et al. National Kidney Foundation practice guidelines for chronic kidney disease: evaluation, classification, and stratification. Ann Intern Med. 2003;139:137–147. doi: 10.7326/0003-4819-139-2-200307150-00013. [DOI] [PubMed] [Google Scholar]
- 11.Smith LR, Harrell FE, Muhlbaier LH. Problems and potentials in modeling survival. In: Grady ML, Schwartz HA, editors. Medical Effectiveness Research Data Methods (Summary Report). AHCPR Pub No 92-0056. Rockville, MD: US Department of Health and Human Services, Agency for Health Care Policy and Research; 1992. pp. 151–159. [Google Scholar]
- 12.Harrell FE. Regression Modeling Strategies. New York: Springer; 2001. General aspects of fitting regression models; pp. 11–37. [Google Scholar]
- 13.Sullivan LM, Massaro JM, D’Agostino RB., Sr Presentation of multivariable data for clinical use: The Framingham Study risk score functions. Statist Med. 2004;23:1631–1660. doi: 10.1002/sim.1742. [DOI] [PubMed] [Google Scholar]
- 14.Royston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model. BMJ. 2009;338:b604. doi: 10.1136/bmj.b604. [DOI] [PubMed] [Google Scholar]
- 15.Pepe MS, Feng Z, Huang Y, Longton G, et al. Integrating the predictiveness of a marker with its performance as a classifier. Am J Epidemiol. 2008;167:362–368. doi: 10.1093/aje/kwm305. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Maddirala S, Khan A, Vincent A, Lau K. Effect of angiotensin converting enzyme inhibitors and angiotensin receptor blockers on serum potassium levels and renal function in ambulatory outpatients: risk factor analysis. Am J Med Sci. 2008;336:330–335. doi: 10.1097/MAJ.0b013e3181836ac7. [DOI] [PubMed] [Google Scholar]
- 17.Moons KG, Altman DG, Vergouwe Y, Royston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009;338:b606. doi: 10.1136/bmj.b606. [DOI] [PubMed] [Google Scholar]
- 18.Kunz R, Friedrich C, Wolbers M, Mann JFE. Meta-analysis: Effect of monotherapy and combination therapy with inhibitors of the renin-angiotensin system on proteinuria in renal disease. Ann Intern Med. 2008;148:30–48. doi: 10.7326/0003-4819-148-1-200801010-00190. [DOI] [PubMed] [Google Scholar]
- 19.Einhorn LM, Zhan M, Hsu VD, et al. The frequency of hyperkalemia and its significance in chronic kidney disease. Arch Intern Med. 2009;169:1156–1162. doi: 10.1001/archinternmed.2009.132. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Altman DG, Royston P. What do we mean by validating a prognostic model? Statist Med. 2000;19:453–473. doi: 10.1002/(sici)1097-0258(20000229)19:4<453::aid-sim350>3.0.co;2-5. [DOI] [PubMed] [Google Scholar]
- 21.Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338:b605. doi: 10.1136/bmj.b605. [DOI] [PubMed] [Google Scholar]