Skip to main content
The Permanente Journal logoLink to The Permanente Journal
. 2016 Dec 23;21:16-143. doi: 10.7812/TPP/16-143

End-Stage Renal Disease Outcomes among the Kaiser Permanente Southern California Creatinine Safety Program (Creatinine SureNet): Opportunities to Reflect and Improve

John J Sim 1, Michael Batech 2, Kim N Danforth 3, Mark P Rutkowski 4, Steven J Jacobsen 5, Michael H Kanter 6
PMCID: PMC5283793  PMID: 28241912

Abstract

Objectives:

The Kaiser Permanente Southern California (KPSC) creatinine safety program (Creatinine SureNet) identifies and outreaches to thousands of people annually who may have had a missed diagnosis for chronic kidney disease (CKD). We sought to determine the value of this outpatient program and evaluate opportunities for improvement.

Methods:

Longitudinal cohort study (February 2010 through December 2015) of KPSC members captured into the creatinine safety program who were characterized using demographics, laboratory results, and different estimations of glomerular filtration rate. Age- and sex-adjusted rates of end-stage renal disease (ESRD) were compared with those in the overall KPSC population.

Results:

Among 12,394 individuals, 83 (0.7%) reached ESRD. The age- and sex-adjusted relative risk of ESRD was 2.7 times higher compared with the KPSC general population during the same period (94.7 vs 35.4 per 100,000 person-years; p < 0.001). Screening with the Chronic Kidney Disease Epidemiology Collaboration (vs Modification Diet in Renal Diseases) equation would capture 44% fewer individuals and have a higher predictive value for CKD. Of those who had repeated creatinine measurements, only 13% had a urine study performed (32% among patients with confirmed CKD).

Conclusion:

Our study found a higher incidence of ESRD among individuals captured into the KPSC creatinine safety program. If the Chronic Kidney Disease Epidemiology Collaboration equation were used, fewer people would have been captured while improving the accuracy for diagnosing CKD. Urine testing was low even among patients with confirmed CKD. Our findings demonstrate the importance of a creatinine safety net program in an integrated health system but also suggest opportunities to improve CKD care and screening.

INTRODUCTION

Screening for chronic kidney disease (CKD) remains controversial. Among high-risk populations, screening and surveillance for CKD is recommended.13 However, organizations such as the US Preventive Services Task Force and the American College of Physicians do not recommend routine screening for asymptomatic individuals in the general population.4,5 The method of assessing kidney function with estimated glomerular filtration rate (eGFR) derived from blood creatinine measurements has also been an area of ambiguity. The eGFR calculations all have inherent accuracy and reliability concerns.6 Among them, the Chronic Kidney Disease Epidemiology Collaboration equation (CKD-EPI) has been shown to have superior accuracy and prognostic value compared with others such as the Cockcroft-Gault and the Modification Diet in Renal Diseases (MDRD) equations.7,8 Another important consideration is the role of urine studies because they help to define and prognosticate CKD.912

CKD is highly prevalent and associated with adverse outcomes, including end-stage renal disease (ESRD), cardiovascular events, and all-cause mortality.8,1315 Among adults in the US, the estimated prevalence is around 14%.13 Unfortunately, CKD is not always identified and managed in an optimal and timely manner.16,17

We previously described the Kaiser Permanente Southern California (KPSC) creatinine safety program (Creatinine SureNet), which was designed to identify and to reach potential patients with CKD who otherwise would have been missed.18 This program leveraged the health system and central laboratory data to create a surveillance system to identify and to outreach to more than 12,000 individuals who did not have a follow-up creatinine test after an initial abnormal creatinine measurement.

Our current study sought to determine the importance of the KPSC creatinine safety program and to identify areas for improvement. We sought to determine the overall rate of ESRD among this presumed high-risk creatinine safety program cohort compared with the rest of the KPSC population. We also evaluated how often patients with confirmed CKD received important complementary studies such as urine analyses and urine protein quantitation. Finally, we sought to compare the rate of capture into the KPSC creatinine safety program if the CKD-EPI had been used to calculate eGFR instead of the existing MDRD equation.

METHODS

A longitudinal cohort study was performed from February 1, 2010, through December 31, 2015, among KPSC members with blood creatinine laboratory studies whose results indicated a reduced eGFR using the MDRD formula. KPSC is an integrated health system composed of 14 Medical Centers, more than 200 satellite clinics, and more than 6000 physicians who care for greater than 4 million members in Southern California. Information on demographics, laboratory results, comorbidities, and clinical events that were captured as part of routine clinical care were extracted from the electronic health records. All laboratory measurements were performed and reported from an American College of Pathology/Clinical Laboratory Improvement Act (CLIA) certified laboratory. Kidney function is reported in the electronic health record as eGFR, calculated using the modified 4-variable MDRD equation.19 For the purpose of our current study, we also calculated eGFR in the same individuals using the CKD-EPI equation.7 End-stage renal disease was defined as any individual receiving dialysis or who received a renal transplant. The study population was followed-up until they reached ESRD, died, or lost KPSC membership, or until the end of study observation (December 31, 2015). This study was approved by the local institutional review board and exempted from the need for informed consent (no. 10572).

The details of the KPSC creatinine safety program have been previously described.18 Born from the KPSC Complete Care program established in 2009, the creatinine safety program (called Creatinine SureNet) was one of many safety nets implemented to capture clinical care gaps using electronic health surveillance and multidisciplinary outreach.20 The Creatinine SureNet uses the concept of electronic clinical surveillance to identify an abnormal creatinine measurement that was not followed-up with a repeated measurement.21 Individuals with a single creatinine measurement that computed to an eGFR less than 60 mL/min (MDRD equation) and had no repeated eGFR measurement 90 days or more later were included. Individuals who fit the following formula—eGFR + ½ age > 85 years—were excluded.22 From February 1, 2010, to March 1, 2014, more than 12,000 members were identified by the Creatinine SureNet. A coordinated effort between a centralized regional nurse and clinicians was used to communicate with patients to obtain a second measurement.

Outcomes and Analyses

All individuals who were included in the Creatinine SureNet cohort were characterized by eGFR and whether they had a follow-up creatinine measurement or other urine tests. The primary outcome was incident ESRD. Age- and sex-adjusted ESRD incidence rates by year were determined for the SureNet cohort and for the KPSC general population for comparison. Age and sex adjustments were standardized to the US Census 2010 population.23 The relative risk (RR) of the SureNet cohort relative to the KPSC general population was estimated, for each year and for all years combined, as the ratio of the 2 incidence rates. A 2-sided test of the null hypothesis that RR = 1 was performed. All-cause mortality information and rates were also determined using data through December 31, 2014.

We also evaluated the proportion of individuals who had urine studies performed, particularly among those who had confirmed CKD by repeated creatinine measurement. Urine studies included urine dipstick, urine microscopy analysis, 24-hour urine protein quantitation, spot urine protein-to-creatinine ratio, and/or spot urine albumin-to-creatinine ratio. Last, we used the CKD-EPI equation to calculate eGFR and then stratified the study population using the new results. We compared the proportion of individuals who would have met the inclusion criteria for the creatinine safety program by the CKD-EPI formula vs the MDRD equation (which was originally used). The ESRD incidence and receiver operating characteristics curve were computed by the different equations for eGFR calculation. Also evaluated were multivariable Cox proportional hazards models examining ESRD or all-cause mortality outcomes adjusted for the confounding effects of baseline eGFR, age at first eGFR measurement, sex, race/ethnicity, Charlson Comorbidity Index, presence of proteinuria, and a history of stroke.

For descriptive statistics, continuous variables were reported as mean with standard deviation, median and interquar-tile range, and categorical variables were reported as the number and proportion at each level. Differences between groups for continuous variables or tests of association for categorical variables were made using t-test or nonparametric Wilcoxon rank sum tests and χ2 or Fisher exact test, respectively, and as appropriate. Shapiro-Wilks test was used to determine normality for parametricity. All hypothesis tests conducted were 2-sided and considered significant at the 5% Type I error rate. All analyses and data management were conducted using SAS Enterprise Guide Version 5.1 (SAS Institute Inc, Cary, NC).

RESULTS

A total of 12,394 individuals were captured into the creatinine safety program in the period February 1, 2010, through March 1, 2014 (Table 1). On the basis of the initial abnormal creatinine measurement, 86.7% of the study population had an eGFR between 45 and 59 mL/min/m2. Among the 6980 individuals who eventually had a repeated creatinine measurement, 53.3% were found to have CKD (eGFR < 60 mL/min/m2).

Table 1.

Study population characteristics by KPSC Creatinine SureNet follow-up statusa

Characteristic No follow-up Follow-up Total p value
Population 5414 (43.7) 6980 (56.3) 12,394 (100.0) < 0.001
Female sex 2289 (42.3) 3878 (55.6) 6167 (49.8)
First eGFR (mL/min/m2)
Mean (SD) 52.1 (7.6) 52.1 (7.2) 52.1 (7.3) 0.231
< 15 14 (0.3) 9 (0.1) 23 (0.2)
15 ≤ 30 93 (1.7) 85 (1.2) 178 (1.4)
30 ≤ 45 626 (11.6) 817 (11.7) 1443 (11.6)
45 ≤ 60 4681 (86.5) 6069 (86.9) 10,750 (86.7)
> 60
Age at index date (years)
Mean (SD) 47.5 (12.38) 50.9 (12.54) 49.4 (12.58) < 0.001
18–39 1299 (24) 1152 (16.5) 2451 (19.8)
40–64 3631 (67.1) 4825 (69.1) 8456 (68.2)
65–85 454 (8.4) 939 (13.5) 1393 (11.2)
> 85 30 (0.6) 64 (0.9) 94 (0.8)
Race/ethnicity
Hispanic 1046 (19.3) 1272 (18.2) 2318 (18.7) < 0.001
White, non-Hispanic 2423 (44.8) 3897 (55.8) 6320 (51)
Black, non-Hispanic 645 (11.9) 753 (10.8) 1398 (11.3)
Asian, non-Hispanic 373 (6.9) 509 (7.3) 882 (7.1)
Other, non-Hispanic 927 (17.1) 549 (7.9) 1476 (11.9)
Charlson Comorbidity Index
0 3300 (61) 3366 (48.2) 6666 (53.8) < 0.001
1–2 1723 (31.9) 2819 (40.4) 4542 (36.7)
≥ 3 384 (7.1) 794 (11.4) 1178 (9.5)
Comorbidities
Hypertension 1491 (27.5) 2562 (36.7) 4053 (32.7) < 0.001
Diabetes mellitus 418 (7.7) 679 (9.7) 1097 (8.9) < 0.001
History of systemic lupus 1 (0.0) 4 (0.1) 5 (0.0) 0.286
Stroke 114 (2.1) 201 (2.9) 315 (2.5) 0.007
Congestive heart failure 76 (1.4) 66 (0.9) 142 (1.1) 0.017
Outcomes
All-cause mortality 142 (2.6) 159 (2.3) 301 (2.4) 0.216
End-stage renal disease 27 (0.5) 56 (0.8) 83 (0.7) 0.04
Length of follow-up (years)b
Mean (SD) 4.2 (2.99) 4.1 (2.22) 4.2 (2.59) < 0.001
Median (IQR) 3.5 (2.1–5.4) 3.8 (2.64.9) 3.7 (2.4–5.1)
Range 0–18.9 0.3–18.7 0–18.9
a

Data are presented as no. (%) unless indicated otherwise.

b

Follow-up was estimated from the date of first creatinine measurement through the earliest of 1) death, 2) end of KPSC membership, 3) December 31, 2015, or 4) date of diagnosis of end-stage renal disease.

eGFR = estimated glomerular filtration rate; IQR = interquartile range; KPSC = Kaiser Permanente Southern California; SD = standard deviation.

End-stage Renal Disease and Mortality Outcomes

In the follow-up period up to December 31, 2015, a total of 83 individuals reached ESRD (56 among those with repeated creatinine measurements and 27 among those who did not follow-up with a repeated measurement; see Table 1). The mean follow-up was 4.2 years. The rate of ESRD by different categories of initial eGFR shows that most patients who reached ESRD came from the lower eGFR groups (< 45 mL/min/m2). The age- and sex-adjusted incidence of ESRD for the creatinine safety cohort during the study period was 94.7 per 100,000 person-years. Among the KPSC general population during the same period, the age- and sex-adjusted ESRD incidence was 35.4 per 100,000 person-years. Overall, the ageand sex-adjusted relative risk of ESRD was 2.68 times higher for the safety program cohort compared with the KPSC population (p < 0.001, Table 2, Figure 1). Among the study cohort, 301 patients died (159 who repeated and 142 patients who did not repeat their creatinine measurement).

Table 2.

Age- and sex-adjusted incidence of end-stage renal disease in KPSC population and Creatinine SureNet population, by year and all years combined, 2008–2015

Population 2010 2011 2012 2013 2014 2015 All years combined
KPSC population, excluding cohort
New cases 1279 1212 1178 1215 1362 1383 7629
Population at risk 3,244,757 3,383,365 3,479,530 3,544,815 3,662,032 3,902,995 21,217,493
Ratea 40.0 36.4 33.9 33.4 35.7 33.9 35.4
Standard error 1.1 1.1 1.0 1.0 1.0 0.9 0.4
SureNet population
New cases 8 5 12 18 18 22 83
Population at risk 6041 7835 9093 9826 9771 9092 61,228
Ratea 116.4 44.8 75.6 117.3 112.0 229.6 94.7
Standard error 50.1 23.9 23.2 30.7 27.6 82.0 12.8
SureNet population relative to the overall KPSC populationb
Relative risk 2.91 1.23 2.23 3.51 3.14 6.77 2.68
Standard error 0.43 0.53 0.31 0.26 0.25 0.36 0.14
p value 0.013 0.698 0.009 < 0.001 < 0.001 < 0.001 < 0.001
a

All rates are reported as per 100,000 KPSC members.

b

Relative risk (RR) is the ratio of the SureNet population relative to the overall KPSC population, and the reported p value is for the 2-sided test of the null hypothesis that RR = 1. KPSC = Kaiser Permanente Southern California.

Figure 1.

Figure 1.

Incidence rates of end-stage renal disease by year and population.

KPSC = Kaiser Permanente Southern California.

Among the safety program cohort, multivariable adjusted Cox proportional hazards models for ESRD demonstrated that higher eGFR (hazard ratio [HR] = 0.57, 95% confidence interval [CI] = 0.53–0.62 for every 5-unit increase), male sex (HR = 0.58, 95% CI = 0.36–0.93), and older age at first eGFR (HR = 0.80, 95% CI = 0.74–0.87 for every 5-year increment) were associated with a reduced hazard for ESRD. Proteinuria was a substantial risk factor for ESRD in these models (HR = 5.37, 95% CI = 3.35–8.62) in addition to higher Charlson Comorbidity Index scores. Similar associations were seen for all-cause mortality, but higher age at first eGFR and male sex increased the hazard for death, demonstrating a competing risk between death and ESRD (Table 3).

Table 3.

Cox proportional hazards ratios (HR) with 95% confidence intervals (CI) for end-stage renal disease (ESRD) or deatha

Parameter ESRD, HR (CI) Death, HR (CI)
Baseline eGFR (5-unit increase) 0.57 (0.53-0.62) 0.74 (0.69–0.80)
Age at first eGFR (5-year increase) 0.80 (0.74–0.87) 1.28 (1.21–1.36)
Male vs female sex 0.58 (0.36–0.93) 1.33 (1.03–1.72)
Race/ethnicity
White, non-Hispanic Reference Reference
Black, non-Hispanic 3.69 (1.83–7.44) 1.12 (0.76–1.65)
Hispanic 3.38 (1.78–6.40) 0.84 (0.59–1.19)
Asian, non-Hispanic 2.90 (1.39–6.05) 0.80 (0.49–1.29)
Other, non-Hispanic 0.48 (0.06–3.72) 1.25 (0.80–1.93)
Charlson Comorbidity Index
0 Reference Reference
1–2 7.25 (1.66–31.61) 1.30 (0.87–1.95)
≥ 3 22.56 (4.95–102.73) 2.97 (1.93–4.58)
Presence of proteinuria (yes vs no) 5.37 (3.35–8.62) 1.28 (0.90–1.81)
Presence of stroke (yes vs no) 1.43 (0.72–2.84) 1.43 (1.00–2.04)
a

Models are adjusted for all other variables reported in this table.

eGFR = estimated glomerular filtration rate.

Urine Studies Performed

Among the study cohort, 1602 individuals (12.9%) had any urine study performed (Table 4) within 180 days of their creatinine measurement. Among the individuals who went for a repeated creatinine measurement, 23.0% had a urine study performed. For those with confirmed CKD (second eGFR < 60 mL/min/m2), the proportion with a urine test of any kind within 180 days of the second eGFR was 31.8%. Among those with confirmed CKD and a repeated eGFR below 45 mL/ min/m2, only 44.1% had a urine study of any kind performed. In total, 1619 of the 6980 with a repeated measurement had any urine test of any kind within 180 days of that measurement (Table 5).

Table 4.

Study population urine study characteristics by category of baseline estimated glomerular filtration rate (mL/min/1.73 m2)a

Characteristic < 15 15 ≤ 30 30 ≤ 45 45 ≤ 60 Total No. p value
Population, no. (%) 23 (0.2) 178 (1.4) 1443 (11.6) 10,750 (86.7) 12,394 (100.0)
Proteinuria,b no. (%) 3 (13) 31 (17.4) 141 (9.8) 207 (1.9) 382 (3.1) < 0.001
Urine protein quantification, no. (%) 4 (17.4) 42 (23.6) 308 (21.3) 1248 (11.6) 1602 (12.9) < 0.001
24-hour urinary total protein (g)
No. of patients 0 3 6 14 23 0.061
Mean (SD) 495.0 (314.42) 1064.5 (1836.40) 525.7 (1369.56) 662.3 (1394.13)
Median (interquartile range) 594.0 (143.0–748.0) 354.0 (192.0–564.0) 104.0 (77.0–192.0) 150.0 (81.0–564.0)
Range 143.0–748.0 123.0–4800.0 5.0–5238.0 5.0–5238.0
Urine protein-to-creatinine ratio (mg/mg creatinine)
No. of patients 2 18 49 72 141 < 0.001
Mean (SD) 2.1 (0.0) 2.2 (2.56) 1.0 (1.81) 0.5 (0.97) 0.9 (1.64)
Median (interquartile range) 2.1 (2.1–2.1) 1.5 (0.5–2.5) 0.4 (0.1–0.8) 0.1 (0.1–0.5) 0.3 (0.1–1.1)
Range 2.1–2.1 0.3–10.2 0.0–8.6 0.0–3.8 0.0–10.2
Albumin-to-creatinine ratio (mg/mg albumin)
No. of patients 3 31 234 613 881 < 0.001
Mean (SD) 1.7 (1.56) 0.9 (1.03) 0.4 (0.95) 0.2 (0.58) 0.2 (0.74)
Median (interquartile range) 2.0 (0.0–3.1) 0.4 (0.1–1.5) 0.0 (0.0–0.2) 0.0 (0.0–0.0) 0.0 (0.0–0.1)
Range 0.0–3.1 (0.0–4.7) 0.0–8.3 0.0–7.7 0.0–8.3
Hematuriac 0/3 (0.0) 4/22 (18.2) 40/135 (29.6) 144/458 (31.4) 188/618 (30.4) 0.371
Any urine study,d no. (%) 4 (17.4) 42 (23.6) 313 (21.7) 1260 (11.7) 1619 (13.1) < 0.001
a

For those with confirmed chronic kidney disease (second estimated glomerular filtration rate [eGFR] < 60 mL/min/1.73 m2), the proportion with a urine test of any kind within 180 days of the second eGFR is 32.0%. In total, 1619 out of the 6980 patients with a repeated measurement had any urine test of any kind within 180 days of that measurement.

b

Proteinuria is defined as meeting any of the following criteria: urine microalbumin-to-creatinine ratio > 30 mg/g, urine protein-to-creatinine ratio > 0.2, 24-hour urine protein concentration > 200 mg, 24-hour urine albumin level > 30 mg, or urinanlysis with 2+ or higher protein.

c

Hematuria is defined as any urine dipstick result that was 1+ or higher for blood or urine microscopy reporting 5 or greater red blood cells per high-power field. Numbers are reported as number of patients with hematuria over total number with available laboratory measurement (and percentage).

d

Within 6 months of the follow-up creatinine measurement. SD = standard deviation.

Table 5.

Study population characteristics by KPSC Creatinine SureNet follow-up groups for estimated glomerular filtration rate (mL/min/m2)a

Characteristic < 15 15 ≤ 30 30 ≤ 45 45 ≤ 60 ≥ 60 Total p value
Population 21 (0.3) 123 (1.8) 783 (11.2) 2730 (39.1) 3323 (47.6) 6980 (100.0)
Female sex 9 (42.9) 58 (47.2) 409 (52.2) 1537 (56.3) 1865 (56.1) 3878 (55.6) 0.051
Age at index date (years)
Mean (SD) 51.5 (13.14) 62.0 (16.65) 63.8 (13.27) 53.5 (10.07) 45.3 (10.66) 50.9 (12.54) < 0.001
18–39 5 (23.8) 11 (8.9) 33 (4.2) 209 (7.7) 894 (26.9) 1152 (16.5)
40–64 12 (57.1) 53 (43.1) 358 (45.7) 2086 (76.4) 2316 (69.7) 4825 (69.1)
65–85 4 (19) 49 (39.8) 345 (44.1) 428 (15.7) 113 (3.4) 939 (13.5)
> 85 0 (0) 10 (8.1) 47 (6) 7 (0.3) 0 (0) 64 (0.9)
Race/ethnicity
White, non-Hispanic 6 (28.6) 53 (43.1) 426 (54.4) 1706 (62.5) 1706 (51.3) 3897 (55.8) < 0.001
Black, non-Hispanic 4 (19) 16 (13) 80 (10.2) 251 (9.2) 402 (12.1) 753 (10.8)
Hispanic 8 (38.1) 35 (28.5) 141 (18) 367 (13.4) 721 (21.7) 1272 (18.2)
Asian, non-Hispanic 3 (14.3) 13 (10.6) 72 (9.2) 186 (6.8) 235 (7.1) 509 (7.3)
Other, non-Hispanic 0 (0) 6 (4.9) 64 (8.2) 220 (8.1) 259 (7.8) 549 (7.9)
Charlson Comorbidity Index
0 0 (0) 2 (1.6) 77 (9.8) 1043 (38.2) 2244 (67.5) 3366 (48.2) < 0.001
1–2 9 (42.9) 59 (48) 415 (53) 1369 (50.1) 967 (29.1) 2819 (40.4)
≥ 3 12 (57.1) 62 (50.4) 291 (37.2) 318 (11.6) 111 (3.3) 794 (11.4)
Any urinalysis laboratory test or procedure 13 (61.9) 57 (46.3) 343 (43.8) 752 (27.5) 454 (13.7) 1619 (23.2) < 0.001
Previous comorbidities
Hypertension 19 (90.5) 103 (83.7) 592 (75.6) 1121 (41.1) 727 (21.9) 2562 (36.7) < 0.001
Diabetes mellitus 6 (28.6) 42 (34.1) 208 (26.6) 267 (9.8) 156 (4.7) 679 (9.7) < 0.001
History of systemic lupus 0 (0) 0 (0) 2 (0.3) 2 (0.1) 0 (0) 4 (0.1) 0.113
Stroke 4 (19) 9 (7.3) 71 (9.1) 57 (2.1) 60 (1.8) 201 (2.9) < 0.001
Congestive heart failure 1 (4.8) 5 (4.1) 24 (3.1) 21 (0.8) 15 (0.5) 66 (0.9) < 0.001
Outcomes
All-cause mortality 7 (33.3) 19 (15.4) 59 (7.5) 51 (1.9) 23 (0.7) 159 (2.3) < 0.001
End-stage renal disease 15 (71.4) 19 (15.4) 16 (2) 5 (0.2) 1 (0) 56 (0.8)
Length of follow-up (years)
Mean (SD) 2.1 (1.33) 3.4 (2.18) 3.6 (1.87) 4.0 (2.01) 4.4 (2.41) 4.1 (2.22) < 0.001
Median (IQR) 1.4 (1.0–3.0) 3.0 (2.0–4.1) 3.3 (2.2–4.6) 3.8 (2.5–4.8) 4.1 (2.8–5.3) 3.8 (2.6–4.9)
Range 0.6–4.6 0.8–18.6 0.4–18.4 0.5–16.5 0.3–18.7 0.3–18.7
a

Data are presented as no. (%) unless indicated otherwise.

IQR = interquartile range; KPSC = Kaiser Permanente Southern California; SD = standard deviation.

Comparison of Equations

The redistribution of the study cohort based on the initial eGFR calculated using the CKD-EPI equation demonstrated that 4732 individuals (43.5%) would have had a calculated eGFR of 60 mL/min/m2 and higher. Thus, these individuals would not have been captured into the safety program if the CKD-EPI instead of MDRD equation had been used (Table 6). Only 5 of these individuals reached ESRD during our observation period. Thus, the CKD-EPI equation captured 56.5% of the original MDRD-based cohort but still included almost all patients who reached ESRD. With confirmed CKD based on the second eGFR as the predictive outcome, 67.8% of individuals captured by CKD-EPI would have had confirmed CKD compared with 52.3% using MDRD. Furthermore, using a logistic regression model to estimate the area under the receiver operating characteristics curve demonstrated that using the CKD-EPI equation for eGFR has higher validity compared with the MDRD equation (area under the curve = 0.943 vs 0.931, p = 0.025).

Table 6.

Study population characteristics classified by whether patients were captured by the Modification Diet in Renal Diseases (MDRD) equation alone or by both MDRD and the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equationsa

Characteristic MDRD only MDRD and CKD-EPI Total p value
Population 4732 (43.5) 6154 (56.5) 10,886 (100.0)
Female sex 2464 (52.1) 3007 (48.9) 5471 (50.3) 0.001
Follow-up MDRD eGFR (mL/min/m2)
Mean (SD) 66.7 (11.49) 55.2 (14.80) 60.0 (14.66) < 0.001
< 15 0 (0.0) 21 (0.6) 21 (0.3)
15 ≤ 30 7 (0.3) 96 (2.6) 103 (1.6)
30 ≤ 45 31 (1.2) 661 (18.1) 692 (11)
45 ≤ 60 788 (29.6) 1695 (46.5) 2483 (39.4)
> 60 1832 (68.9) 1174 (32.2) 3006 (47.7)
Demographics
Age at index date (years)
Mean (SD) 43.2 (8.54) 54.7 (12.55) 49.7 (12.40) < 0.001
18–39 1354 (28.6) 692 (11.2) 2046 (18.8)
40–64 3366 (71.1) 4164 (67.7) 7530 (69.2)
65–85 12 (0.3) 1214 (19.7) 1226 (11.3)
> 85 0 (0.0) 84 (1.4) 84 (0.8)
Race/ethnicity
White, non-Hispanic 2287 (48.3) 3360 (54.6) 5647 (51.9) < 0.001
Black, non-Hispanic 757 (16) 442 (7.2) 1199 (11)
Hispanic 870 (18.4) 1151 (18.7) 2021 (18.6)
Asian, non-Hispanic 321 (6.8) 470 (7.6) 791 (7.3)
Other, non-Hispanic 497 (10.5) 731 (11.9) 1228 (11.3)
Charlson Comorbidity Index
0 3261 (68.9) 2512 (40.8) 5773 (53) < 0.001
1–2 1369 (28.9) 2689 (43.7) 4058 (37.3)
≥ 3 102 (2.2) 951 (15.5) 1053 (9.7)
Any urinalysis laboratory test or procedure 427 (9) 1146 (18.6) 1573 (14.4) < 0.001
Previous comorbidities
Hypertension 844 (17.8) 2778 (45.1) 3622 (33.3) < 0.001
Diabetes mellitus 150 (3.2) 801 (13) 951 (8.7) < 0.001
History of systemic lupus 0 (0.0) 4 (0.1) 4 (0.0) 0.079
Stroke 53 (1.1) 213 (3.5) 266 (2.4) < 0.001
Congestive heart failure 22 (0.5) 99 (1.6) 121 (1.1) < 0.001
Outcomes
All-cause mortality 15 (0.3) 242 (3.9) 257 (2.4) < 0.001
End-stage renal disease 5 (0.1) 72 (1.2) 77 (0.7)
Length of follow-up
Mean (SD) 3.8 (1.76) 3.8 (1.85) 3.8 (1.81) 0.862
Median (interquartile range) 3.7 (2.5–4.7) 3.6 (2.5–4.9) 3.7 (2.5–4.8)
Range 0.1–9.0 0.0–9.0 0.0–9.0
a

On the basis of the patient’s initial serum creatinine measurement. Data are presented as no. (%) unless indicated otherwise.

eGFR = estimated glomerular filtration rate; SD = standard deviation.

DISCUSSION

We sought to evaluate the impact of a creatinine safety program within the clinical care environment and infrastructure of an integrated health system (Kaiser Permanente). Our study evaluated ESRD outcomes on more than 12,000 individuals with single abnormal creatinine measurements who were captured into the KPSC creatinine safety program. We were able to demonstrate higher ESRD incidence rates among those who were captured into the safety program and still higher among those who had a repeated creatinine measurement. Black race, female sex, and proteinuria were associated with higher risk of ESRD in the safety program cohort. These findings support our assertion that the safety net is useful in identifying and outreaching to a high-risk population. We also found a potential care gap in that most of those who had confirmation of their CKD did not receive a urine study within 6 months. Last, our study findings demonstrated that using the CKD-EPI instead of the MDRD equation would have resulted in substantially fewer people being captured into the safety program while capturing almost all of those who progressed to ESRD.

Determining the added value of this creatinine safety program would validate the current efforts and help pave the way to improve the program moving forward. Most of our safety program population had early-stage CKD; 87% had eGFR in the range of 45 to 59 mL/min/m2. This population is not considered at high risk of ESRD or mortality as are later stages of CKD.14,24 Our findings demonstrate that a high-risk population for CKD was indeed identified while we were able to track pertinent practice patterns and CKD-related outcomes. The main assumption is that the program will lead to interventions that result in better overall CKD and CKD-related care. Early CKD education and management have been shown to result in improved pre-ESRD and post-ESRD outcomes,25 and this KPSC safety program was implemented to prevent delays in the diagnosis and treatment of CKD.

The method of screening and diagnosing CKD is an area of uncertainty. Measurement of eGFR is one way of assessing kidney function, but it is usually estimated with equations. KPSC laboratories report the eGFR using the MDRD calculation. However, the MDRD equation has systemic biases in that it was derived from a population already with a diagnosis of CKD.6,19 It has been described that up to 29% of healthy people had their kidney function underestimated on the basis of the MDRD equation.6 The CKD-EPI equation derived from people with and without CKD has been demonstrated to be superior to the MDRD equation, especially in the higher ranges of eGFR.7 This is particularly relevant to our current study cohort in which most patients were initially identified with marginally low eGFR. Furthermore, the CKD-EPI is a better prognosticator of ESRD and mortality outcomes.8,26 Similar to the findings of our study, the global estimate of CKD was lowered by 24% when CKD-EPI was used instead of the MDRD equation.8 Although it would identify some patients as having CKD that MDRD would not, the net persons identified with CKD would be less using the CKD-EPI equation.27 Given our findings, there may be an opportunity to refine the creatinine safety program by using a different eGFR equation. Using the MDRD equation may put the population at risk of overdiagnosis, potentially unnecessarily alarming patients. In addition, it may unnecessarily divert cost and resources from other clinical priorities. Within KPSC, had the CKD-EPI equation been used, 44% fewer individuals would have been contacted by the safety program.

The overall low rate of urine studies among patients with confirmed CKD in our study cohort appears alarming. Among patients with confirmed CKD in our study, only one-third had a urine study performed within six months. However, individuals with proteinuria had five times greater risk of progressing to ESRD compared with those without proteinuria. Markers of kidney damage such as proteinuria, hematuria, and anatomic abnormalities help define and prognosticate outcomes in CKD.12,28 Specifically, urine protein studies are recommended to use as a marker to manage CKD.29 Proteinuria has been associated with worsened cardiovascular outcomes, ESRD, and mortality.911 Although screening for CKD may be controversial, urine testing among patients with CKD is advocated by organizations such as the Kidney Disease Improving Global Outcomes, National Kidney Foundation, and American Diabetes Association.1,12

Chronic kidney disease is not always identified in an optimal and timely manner. Overall, there is low patient awareness and low clinician identification and documentation of CKD.16,17,30 Establishing a diagnosis of CKD is difficult because of a multitude of systemic, physician, and patient-oriented barriers.31,32 The chronicity needed for diagnosis, the lack of symptoms for many of the patients, and the large volume of laboratory results that flow through the routine clinical practice environment all contribute to a potential care gap leading to missed CKD diagnoses.

Although it appears that the creatinine safety program captures a high-risk who have higher Charlson Comorbidity Index scores are less likely to be captured into the creatinine safety net (unpublished internal data 2016). Another example is that when the KPSC SureNet program was started in February 2010, we dated our search to laboratory results from January 1997. However, only 13.5% of the safety program cohort was captured in the 13-year period from 1997 through 2009 compared with the remainder that were identified from 2010 through 2014.18 Already, KPSC has certain infrastructures in place to care for patients with a diagnosis of chronic conditions.33 Patients who have more obvious manifestations of CKD will more likely utilize the access to care that is readily available for all members. The Complete Care model at KPSC is inclusive of many of the tools and personnel needed to adequately ensure the success of safety nets such as the Creatinine SureNet.20 It takes advantage of the electronic health records and the integrated health system. It also has a model of being proactive toward patient care at all encounters and levels of care. One example is that the electronic charts have best-practice alerts that populate the screen for clinicians during patient encounters. Clinicians also have a proactive care screen section accessible during visits. The patients can use tools, such as the Internet portal (www.kp.org), to help awareness, communication, and follow-up. These online personal action tools have enabled faster closure of care gaps in KPSC.34

Future Direction of Creatinine Safety Research

Our research is currently funded (Agency for Healthcare Research and Quality, R01HS024437, principal investigator: KND), and we are studying more detailed aspects of care related to the creatinine safety program. Specifically, what are the contributors and barriers from a physician perspective that lead to care gaps and subsequent capture into the safety program? We are in the early stages of studying and identifying systemic, patient, and provider-related factors. Using our current and future findings, we hope to develop and implement preventive strategies to minimize the care gap. We need to continuously population, patients who have severe or more urgent CKD were likely already captured and managed by the KPSC health system. One example is that members reflect on ways to “best cast the safety net” for the betterment of patient care.

CONCLUSION

Our study found a higher incidence of ESRD among individuals captured into the KPSC creatinine safety program. We found that the CKD-EPI instead of the MDRD equation would have identified 44% fewer individuals for the safety net while capturing almost all patients whose CKD progressed to ESRD. Among patients with confirmed CKD, 68% of patients did not receive urine testing in a timely manner. Although the creatinine safety program has an important role and place in an integrated health system such as Kaiser Permanente, our findings also suggest opportunities to improve CKD care and screening by refining this program.

Acknowledgments

This study was funded and supported by the Kaiser Permanente Southern California (KPSC) Clinician Investigator Award (JJS). The authors would like to thank David Selevan and Royann Timmins for their important work in the KPSC Creatinine SureNet program. The authors would also like to thank Mandy Cheung and Noel Pascual of the Renal Business Group for their assistance in information gathering and analysis pertaining to the chronic kidney disease and end-stage renal disease populations at KPSC. Finally, the authors wish to thank Stephanie D Goldman, MPH, Senior Consultant, Evidence-Based Medicine Services, Southern California Permanente Medical Group, for her assistance and evidence review comparing Chronic Kidney Disease Epidemiology Collaborarion and Modification Diet in Renal Diseases equations.

Kathleen Louden, ELS, of Louden Health Communications provided editorial assistance.

Footnotes

Disclosure Statement

The author(s) have no conflicts of interest to disclose.

References

  • 1.American Diabetes Association Executive summary: Standards of medical care in diabetes—2011. Diabetes Care. 2011 Jan;34(Suppl 1):S4–10. doi: 10.2337/dc11-S004. DOI: https://doi.org/10.2337/dc11-s004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Brosius FC, 3rd, Hostetter TH, Kelepouris E, et al. American Heart Association Kidney and Cardiovascular Disease Council; Council on High Blood Pressure Research; Council on Cardiovascular Disease in the Young; Council on Epidemiology and Prevention; Quality of Care and Outcomes Research Interdisciplinary Working Group Detection of chronic kidney disease in patients with or at increased risk of cardiovascular disease: A science advisory from the American Heart Association Kidney and Cardiovascular Disease Council; the Councils on High Blood Pressure Research, Cardiovascular Disease in the Young, and Epidemiology and Prevention; and the Quality of Care and Outcomes Research Interdisciplinary Working Group: Developed in collaboration with the National Kidney Foundation. Circulation. 2006 Sep 5;114(10):1083–7. doi: 10.1161/CIRCULATIONAHA.106.177321. DOI: https://doi.org/10.1161/CIRCULATIONAHA.106.177321. [DOI] [PubMed] [Google Scholar]
  • 3.Chobanian AV, Bakris GL, Black HR, et al. National Heart, Lung, and Blood Institute Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure; National High Blood Pressure Education Program Coordinating Committee The seventh report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure: The JNC 7 report. JAMA. 2003 May 21;289(19):2560–72. doi: 10.1001/jama.289.19.2560. DOI: https://doi.org/10.1001/jama.289.19.2560. [DOI] [PubMed] [Google Scholar]
  • 4.Fink HA, Ishani A, Taylor BC, et al. Screening for, monitoring, and treatment of chronic kidney disease stages 1 to 3: A systematic review for the U.S. Preventive Services Task Force and for an American College of Physicians Clinical Practice Guideline. Ann Intern Med. 2012 Apr 17;156(8):570–81. doi: 10.7326/0003-4819-156-8-201204170-00004. DOI: https://doi.org/10.7326/0003-4819-156-8-201204170-00008. [DOI] [PubMed] [Google Scholar]
  • 5.Saunders MR, Cifu A, Vela M. Screening for chronic kidney disease. JAMA. 2015 Aug 11;314(6):615–6. doi: 10.1001/jama.2015.9425. DOI: https://doi.org/10.1001/jama.2015.9425. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Rule AD, Larson TS, Bergstralh EJ, Slezak JM, Jacobsen SJ, Cosio FG. Using serum creatinine to estimate glomerular filtration rate: Accuracy in good health and in chronic kidney disease. Ann Intern Med. 2004 Dec 21;141(12):929–37. doi: 10.7326/0003-4819-141-12-200412210-00009. DOI: https://doi.org/10.7326/0003-4819-141-12-200412210-00009. [DOI] [PubMed] [Google Scholar]
  • 7.Levey AS, Stevens LA, Schmid CH, et al. CKDEPI (Chronic Kidney Disease Epidemiology Collaboration) A new equation to estimate glomerular filtration rate. Ann Intern Med. 2009 May 5;150(9):604–12. doi: 10.7326/0003-4819-150-9-200905050-00006. DOI: https://doi.org/10.7326/0003-4819-150-9-200905050-00006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Matsushita K, Mahmoodi BK, Woodward M, et al. Chronic Kidney Disease Prognosis Consortium Comparison of risk prediction using the CKDEPI equation and the MDRD study equation for estimated glomerular filtration rate. JAMA. 2012 May 9;307(18):1941–51. doi: 10.1001/jama.2012.3954. DOI: https://doi.org/10.1001/jama.2012.3954. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Astor BC, Matsushita K, Gansevoort RT, et al. Lower estimated glomerular filtration rate and higher albuminuria are associated with mortality and end-stage renal disease. A collaborative meta-analysis of kidney disease population cohorts. Kidney Int. 2011 Jun;79(12):1331–40. doi: 10.1038/ki.2010.550. DOI: https://doi.org/10.1038/ki.2010.550. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Chronic Kidney Disease Prognosis Consortium. Matsushita K, van der Velde M, Astor BC, et al. Association of estimated glomerular filtration rate and albuminuria with all-cause and cardiovascular mortality in general population cohorts: A collaborative meta-analysis. Lancet. 2010 Jun 12;375(9731):2073–81. doi: 10.1016/S0140-6736(10)60674-5. DOI: https://doi.org/10.1016/S0140-6736(10)60674-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Gansevoort RT, Matsushita K, van der Velde M, et al. Chronic Kidney Disease Prognosis Consortium Lower estimated GFR and higher albuminuria are associated with adverse kidney outcomes. A collaborative meta-analysis of general and high-risk population cohorts. Kidney Int. 2011 Jul;80(1):93–104. doi: 10.1038/ki.2010.531. DOI: https://doi.org/10.1038/ki.2010.531. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Kidney Disease: Improving Global Outcomes (KDIGO) CKD Work Group KDIGO 2012 clinical practice guideline for the evaluation and management of chronic kidney disease. Kidney Int Suppl. 2013 Jan;3(1):1–150. [Google Scholar]
  • 13.Saran R, Li Y, Robinson B, et al. US renal data system 2015 annual data report: Epidemiology of kidney disease in the United States. Am J Kidney Dis. 2016 Mar;67(3 Suppl 1):Svii, S1–305. doi: 10.1053/j.ajkd.2015.12.014. DOI: https://doi.org/10.1053/j.ajkd.2015.12.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Go AS, Chertow GM, Fan D, McCulloch CE, Hsu CY. Chronic kidney disease and the risks of death, cardiovascular events, and hospitalization. N Engl J Med. 2004 Sep;351(13):1296–305. doi: 10.1056/NEJMoa041031. DOI: https://doi.org/10.1056/nejmoa041031 Erratum in: N Engl J Med 2008; 18(4): 4. [DOI] [PubMed] [Google Scholar]
  • 15.van der Velde M, Matsushita K, Coresh J, et al. Lower estimated glomerular filtration rate and higher albuminuria are associated with all-cause and cardiovascular mortality. A collaborative meta-analysis of high-risk population cohorts. Kidney Int. 2011 Jun;79(12):1341–52. doi: 10.1038/ki.2010.536. DOI: https://doi.org/10.1038/ki.2010.536. [DOI] [PubMed] [Google Scholar]
  • 16.de Lusignan S, Chan T, Stevens P, et al. Identifying patients with chronic kidney disease from general practice computer records. Fam Pract. 2005 Jun;22(3):234–41. doi: 10.1093/fampra/cmi026. DOI: https://doi.org/10.1093/fampra/cmi026. [DOI] [PubMed] [Google Scholar]
  • 17.Plantinga LC, Boulware LE, Coresh J, et al. Patient awareness of chronic kidney disease: Trends and predictors. Arch Intern Med. 2008 Nov 10;168(20):2268–75. doi: 10.1001/archinte.168.20.2268. DOI: https://doi.org/10.1001/archinte.168.20.2268. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Sim JJ, Rutkowski MP, Selevan DC, et al. Kaiser Permanente creatinine safety program: A mechanism to ensure widespread detection and care for chronic kidney disease. Am J Med. 2015 Nov;128(11):1204–11.e1. doi: 10.1016/j.amjmed.2015.05.037. DOI: https://doi.org/10.1016/j.amjmed.2015.05.037. [DOI] [PubMed] [Google Scholar]
  • 19.Levey AS, Bosch JP, Lewis JB, Greene T, Rogers N, Roth D. A more accurate method to estimate glomerular filtration rate from serum creatinine: A new prediction equation. Modification of Diet in Renal Disease Study Group. Ann Intern Med. 1999 Mar 16;130(6):461–70. doi: 10.7326/0003-4819-130-6-199903160-00002. DOI: https://doi.org/10.7326/0003-4819-130-6-199903160-00002. [DOI] [PubMed] [Google Scholar]
  • 20.Kanter MH, Lindsay G, Bellows J, Chase A. Complete care at Kaiser Permanente: Transforming chronic and preventive care. Jt Comm J Qual Patient Saf. 2013 Nov;39(11):484–94. doi: 10.1016/s1553-7250(13)39064-3. DOI: https://doi.org/10.1016/s1553-7250(13)39064-3. [DOI] [PubMed] [Google Scholar]
  • 21.Danforth KN, Smith AE, Loo RK, Jacobsen SJ, Mittman BS, Kanter MH. Electronic clinical surveillance to improve outpatient care: Diverse applications within an integrated delivery system. EGEMS (Wash DC) 2014 Jun 24;2(1):1056. doi: 10.13063/2327-9214.1056. DOI: https://doi.org/10.13063/2327-9214.1056. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Rutkowski M, Mann W, Derose S, et al. Implementing KDOQI CKD definition and staging guidelines in Southern California Kaiser Permanente. Am J Kidney Dis. 2009 Mar;53(3 Suppl 3):S86–99. doi: 10.1053/j.ajkd.2008.07.052. DOI: https://doi.org/10.1053/j.ajkd.2008.07.052. [DOI] [PubMed] [Google Scholar]
  • 23.United States Census Bureau [Internet] Washington, DC: US Census Bureau; c2016. [cited 2016 Aug 10]. Available from: www.census.gov. [Google Scholar]
  • 24.Kovesdy CP, Coresh J, Ballew SH, et al. CKD Prognosis Consortium Past decline versus current eGFR and subsequent ESRD risk. J Am Soc Nephrol. 2016 Aug;27(8):2447–55. doi: 10.1681/ASN.2015060687. DOI: https://doi.org/10.1681/asn.2015060687. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Kurella Tamura M, Li S, Chen SC, et al. Educational programs improve the preparation for dialysis and survival of patients with chronic kidney disease. Kidney Int. 2014 Mar;85(3):686–92. doi: 10.1038/ki.2013.369. DOI: https://doi.org/10.1038/ki.2013.369. [DOI] [PubMed] [Google Scholar]
  • 26.Ku E, Xie D, Shlipak M, et al. CRIC Study Investigators Change in measured GFR versus eGFR and CKD outcomes. J Am Soc Nephrol. 2016 Jul;27(7):2196–204. doi: 10.1681/ASN.2015040341. DOI: https://doi.org/10.1681/asn.2015040341. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Rocco MV, Chapman A, Chertow GM, et al. SPRINT Research Group Chronic kidney disease classification in systolic blood pressure intervention trial: Comparison using modification of diet in renal disease and CKD-epidemiology collaboration definitions. Am J Nephrol. 2016;44(2):130–40. doi: 10.1159/000448722. DOI: https://doi.org/10.1159/000448722. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Sim JJ, Batech M, Hever A, et al. Distribution of biopsy-proven presumed primary glomerulonephropathies in 2000–2011 among a racially and ethnically diverse US population. Am J Kidney Dis. 2016 Oct;68(4):533–44. doi: 10.1053/j.ajkd.2016.03.416. DOI: https://doi.org/10.1053/j.ajkd.2016.03.416. [DOI] [PubMed] [Google Scholar]
  • 29.de Jong PE, Curhan GC. Screening, monitoring, and treatment of albuminuria: Public health perspectives. J Am Soc Nephrol. 2006 Aug;17(8):2120–6. doi: 10.1681/ASN.2006010097. DOI: https://doi.org/10.1681/asn.2006010097. [DOI] [PubMed] [Google Scholar]
  • 30.Jolly SE, Navaneethan SD, Schold JD, et al. Chronic kidney disease in an electronic health record problem list: Quality of care, ESRD, and mortality. Am J Nephrol. 39(4):288–96. doi: 10.1159/000360306. 201. DOI: https://doi.org/10.1159/000360306. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Gandhi TK, Kachalia A, Thomas EJ, et al. Missed and delayed diagnoses in the ambulatory setting: A study of closed malpractice claims. Ann Intern Med. 2006 Oct 3;145(7):488–96. doi: 10.7326/0003-4819-145-7-200610030-00006. DOI: https://doi.org/10.7326/0003-4819-145-7-200610030-00006. [DOI] [PubMed] [Google Scholar]
  • 32.Kanter M. Casting a wider net. Manag Care. 2011 Jun;20(6):27–31. [PubMed] [Google Scholar]
  • 33.Sim JJ, Handler J, Jacobsen SJ, Kanter MH. Systemic implementation strategies to improve hypertension: The Kaiser Permanente Southern California experience. Can J Cardiol. 2014 May;30(5):544–52. doi: 10.1016/j.cjca.2014.01.003. DOI: https://doi.org/10.1016/j.cjca.2014.01.003. [DOI] [PubMed] [Google Scholar]
  • 34.Henry SL, Shen E, Ahuja A, Gould MK, Kanter MH. The online personal action plan: A tool to transform patient-enabled preventive and chronic care. Am J Prev Med. 2016 Jul;51(1):71–7. doi: 10.1016/j.amepre.2015.11.014. DOI: https://doi.org/10.1016/j.amepre.2015.11.014. [DOI] [PubMed] [Google Scholar]

Articles from The Permanente Journal are provided here courtesy of Kaiser Permanente

RESOURCES