Pragmatic clinic randomized trial to improve chronic kidney disease care: Design and adaptation due to COVID disruptions

Abstract

Background

We describe a clinic-randomized trial to improve chronic kidney disease (CKD) care through a CKD-clinical decision support (CKD-CDS) intervention in primary care clinics and the challenges we encountered due to COVID-19 care disruption.

Methods/design

Primary care clinics (N = 32) were randomized to usual care (UC) or to CKD-CDS. Between April 17, 2019 and March 14, 2020, more than 7000 patients had accrued for analysis by meeting study-eligibility criteria at an index office visit: age 18–75, laboratory criteria for stage 3 or 4 CKD (eGFR 15–59 mL/min/1.73 m²), and one or more opportunities algorithmically identified to improve CKD care such as blood pressure (BP) or glucose control, angiotensin converting enzyme inhibitor (ACEI) or angiotensin receptor blocker (ARB) use, discontinuance of a nonsteroidal anti-inflammatory drug (NSAID), or nephrology referral. At CKD-CDS clinics, CDS provided individualized treatment suggestions that were printed for patients and clinicians at the start of office encounters and were viewable within the electronic health record. By initial design, the impact of the CKD-CDS intervention on care gaps was to be assessed 12 months after the index date, but COVID-19 caused major disruptions to care delivery during the intervention period. In response to disruptions, the intervention was temporarily suspended while we expanded CDS use for telehealth encounters and programmed new criteria for displaying the CKD-CDS to intervention patients due to clinic closures and scheduling changes.

Discussion

We describe a NIH-funded pragmatic trial of web-based EHR-integrated CKD-CDS and modifications necessary mid-study to complete the study as intended in the face of COVID-19 pandemic challenges.

1. Introduction

Chronic kidney disease (CKD) is a common problem in adults with diabetes, but only about 20% of diabetes patients with stage 3 and 4 CKD (eGFR 15–59 mL/min/1.73 m²) are aware of having it. Disturbingly, there has been no significant improvement in patient awareness of CKD between 1999 and 2016 based on National Health and Nutrition Examination Survey data. [1,2] Data from 2018 national VA CKD surveillance reveals that more than half of patients with CKD by eGFR values do not have an diagnostic code identified for it in the electronic record. [3] Furthermore, national CKD guidelines recommend blood pressure control, use of angiotensin converting enzyme inhibitors (ACEIs) or angiotensin receptor blockers (ARBs) in adults with albuminuria, optimal glucose control in patients with diabetes, and avoidance of nonsteroidal ant-inflammatory agents (NSAIDs) to slow progression of renal disease and reduce the occurrence of major cardiovascular events. [[4], [5], [6]] Even when laboratory data is present to substantiate a CKD diagnosis, primary care clinicians often do not recognize or diagnose it, and, as a result, these care goals are often not achieved. Patients with stage 3b (eGFR 30–44 mL/min/1.73 m²⁾ or stage 4 CKD (eGFR 15–29 mL/min/1.73 m²) often are not referred to nephrology early enough to have meaningful discussions about advanced CKD care and renal replacement options such as dialysis or renal transplantation. [7]

Preliminary data collected for this study as preparation for research (unpublished) from 17 primary care clinics in Minnesota in 2015–2016 showed that of 5766 patients aged 18–75 years with confirmed eGFR 15–59 mL/min/1.73 m², 51% did not have a CKD diagnosis, 63% had a blood pressure (BP) above 130/80 mm Hg, 50% of those with diabetes had glycated hemoglobin (A1C) ≥7%, and merely one in four had a test for albuminuria within the last year. Of those with hypertension or albuminuria (and without hyperkalemia), barely more than half (57%) had been prescribed an ACEI or ARB. Of patients with more advanced CKD (eGFR 15–29 mL/min/1.73 m², or eGFR 30–44 mL/min/1.73 m² with albuminuria), less than half (45%) had seen a nephrologist within the last 2 years.

Widespread use of electronic health records (EHRs) offer new opportunities to identify and address these care gaps, but few clinical decision support (CDS) applications are specifically designed to improve CKD management in primary care. [8] Several recent RCT's have shown some encouraging outcome trends but most were limited either by pragmatic design issues such as smaller than intended sample sizes, randomization group imbalances, or difficulty implementing the CDS. [[9], [10], [11]] In prior work we developed a successful EHR-linked Web-based CDS for diabetes and cardiovascular risk for diabetes and cardiovascular risk factors that followed the Five Rights Framework for CDS implementation. The framework suggests that to be maximally effective, CDS must present the right information, to the right person, in the right format, through the right channel, at the right time. [12] CDS is also more likely to be effective if provided to both the clinician and the patient compared to providing CDS to the clinician alone. [[13], [14], [15]] We addressed essential components for CDS success through integration of CDS into the point-of-care workflow at the beginning of the visit; engagement of both office staff, clinicians, and patients in the CDS process using simple and intuitive formats; and CDS that is more sophisticated than simple prompts or reminders. The CDS system achieved and sustained high use rates (over 75% of targeted primary care encounters) with clinician satisfaction over 90%. [[16], [17], [18]] In previous clinic-randomized trials with 12–18 months of patient follow up, this CDS system significantly improved glycemic and BP control for patients with diabetes, [16] significantly lowered 10-year CV risk in patients without diabetes or vascular disease, [17] improved hypertension in adolescents, [19] and been shown to be cost effective. [20] In this project we integrated patient-specific CKD treatment suggestions with the existing CDS system and are conducting a pragmatic clinic-randomized trial to assess CKD-CDS intervention impact on quality of CKD care.

Pragmatic trials are often designed to reduce selection effects and ensure that evidence-based care translates to real world settings. [21] However, design features of pragmatic trials commonly present unanticipated challenges related to regulatory issues, patient consent, and the need to adapt study designs and interventions to accommodate changes in clinical guidelines and “on the ground” challenges that happen frequently in busy clinical practice settings. [22,23]

The COVID-19 pandemic was a major unanticipated challenge requiring us to make substantial overhauls to the intervention delivery mechanism that required rapid regulatory and funder approval to implement. This manuscript describes the original study, the challenges that emerged, and some of the advantages and limitations of the adaptations that were implemented to respond to COVID related care delivery disruptions.

2. Study design

The overall consort flow of the study showing enrollment, randomization, allocation, follow up and analysis is shown in Figure 1 . Further details about the study design are described below.

Fig. 1.

CONSORT Flow diagram.

2.1. Study setting

The study was conducted at primary care clinics that were part of two health care delivery systems, HealthPartners (HP) and Park Nicollet (PN) located in Minnesota and Wisconsin. We identified 47 clinics, and enrolled and randomized 32 primary care clinics in the study that (a) had sufficient numbers of patients with stage 3 or 4 CKD (eGFR 15–59 mL/min/1.73 m²), (b) were proficient in the use of the EpiCare (Verona, Wisconsin) EHR software, and (c) had locations within the metro area and access to nephrology referrals. The clinics who participated in the project had also participated in our previous CDS interventional studies.

2.2. Clinic randomization

Clinics were randomized 1:1 to intervention or UC using covariate constrained randomization (CCR). [24] The SAS-based CCR macro [25] identified a pool of randomization schemes that would balance five selected clinic covariates across the intervention and UC arms of the study. CCR then randomly selects one randomization scheme for the study randomization. The clinic balancing covariates were calculated in February 2019: 1) care delivery system (PN or HP), 2) clinic eligibility for another cluster-randomized trial that started in the month prior to CKD-CDS randomization 3) number of CKD patients, 4) proportion of CKD patients with Medicaid health insurance coverage, and 5) proportion of hypertension patients meeting the care group's performance measure for achieving the blood pressure goal of <140/90 mm Hg. CKD-CDS software programming, software training and intervention delivery all required that the investigative team, clinic leaders and clinicians not be blinded to clinic allocation. Patients were assigned to the intervention or UC control group based on the assignment of the clinic at which their index visit took place.

2.3. Patient accrual

Patients were accrued into the study if they met the inclusion and exclusion criteria outlined below at an index visit. Automated calculations to assess these criteria were performed by the CDS tool at the beginning of each primary care encounter in randomized clinics over the course of the accrual period.

• 1.Aged 18–75 years inclusive
• 2.Laboratory evidence of Stage 3 or 4 CKD as defined by either one of the following:

• a.Stage 4 CKD: An estimated glomerular filtration rate (eGFR) value of 15–29 cc/min/1.73 m² on the most recent eGFR result in the last 5 years
• b.Stage 3 CKD: An eGFR 30–59 cc/min/1.73 m² on the most recent eGFR in the last 5 years and the next most recent eGFR 15–59 cc/min/1.73m²

• 3.One or more of the following care improvement opportunities (care gaps):

• a.BP over goal: Defined as the lowest recorded BP ≥ 130/80 mm Hg at the visit.
• b.A1C over goal: At the time of the visit, a glycated hemoglobin (A1C) was identified in the last 12 months and the most recent value is above an individualized threshold as determined by the CDS system:

• i.A1C greater or equal to 8% if the patient meets diagnostic criteria for ESRD, CHF, active cancer, hypoglycemia, or cognitive impairment in the last year, or is on a complex medication regimen defined as three or more non-insulin glycemia medications or insulin plus two or more other glycemia medication
• ii.A1C value greater or equal to 7% for all patients not meeting the diagnostic criteria listed above

• c.Not on an ACEI or ARB among patients with eGFR ≥30 and without a diagnostic code for hyperkalemia in the last year and meeting one of the following criteria:

• i.hypertension on the problem list or two or more ICD10 diagnosis codes for hypertension in the last 2 years
• ii.urine albumin creatinine ratio (ACR) ≥30 mg/g

• d.Nonsteroidal anti-inflammatory drug (NSAID) use, defined as one or more NSAID medications (other than aspirin) on the current medication list at the start of the index visit.
• e.No Nephrology visits found in the last 12 months with one of the following additional conditions:

• i.most recent eGFR 15–29 cc/min/1.73m² (Stage 4 CKD)
• ii.albumin creatinine ratio (ACR) ≥300 mg/g (proteinuria)
• iii.most recent GFR 30–44 AND albumin creatinine ratio (ACR) ≥30 mg/g (Stage 3b CKD with proteinuria)

• 4.They had none of the following exclusion criteria:

• a.Evidence of end stage renal disease (ESRD) as defined by

• i.two or more dialysis procedure codes, or
• ii.two or more eGFR values <15 cc/min/1.73m² in the last 5 years

• b.Evidence of pregnancy in the last year as follows:

• i.an identified positive serum of urine pregnancy test
• ii.diagnostic codes for Pregnancy (V22)

• c.Active cancer defined as 3 or more visit diagnostic codes for cancer in the last year excluding non-melanoma skin cancers
• d.Hospice care or palliative care identified in the prior two years.

The care gaps included above correlated with potential improvement of intermediate clinical outcomes of importance to the care system. Preparatory data also revealed that rates of albuminuria testing in the CKD population, especially those without diabetes, were suboptimal. Although the periodic measurement of albuminuria itself was not chosen as a primary care gap, it was considered very important for staging and management of CKD and was recommended by the CDS when indicated for study eligible patients. Based on preliminary data and power analysis, we aimed to accrue about 6100 study-eligible adults with stage 3 or 4 CKD (eGFR 15–59 mL/min/1.73 m²) over a 12-month accrual period.

3. Intervention

3.1. CKD-CDS development

Letters of support to conduct the study if funded had been obtained from health care leadership at the time the grant application was submitted. Leadership was supportive of the application because it attempted to address major care gaps in patients with CKD that were high priorities and aligned with publicly reported performance measures (such as BP, glycemic control, and ACE/ARB use). The nephrology specialists within the care systems were also supportive especially because of concerns about the costs and excess morbidity and mortality that occurs from delays in nephrology referrals for patients with late stage CKD. Because the CKD-CDS was incorporated into an established CDS tool, the burden to clinicians was perceived as minimal. During the planning phase of the study, regular meetings were conducted with healthcare systems leaders from the care systems and nephrology consultant experts to develop the intervention and reach consensus regarding the following:

• •Participant inclusion and exclusion criteria
• •Rules to display the CKD-CDS best practice advisory (BPA) prompting rooming staff to print CDS for patients who met intervention eligibility criteria
• •Workflow for rooming staff and clinicians
• •Content and wording of the CKD-CDS care recommendations, and design of the printed and online materials presented to patients and primary care clinicians
• •Communication to leadership and training procedures for intervention clinics
• •Agreement and support from the two nephrology specialist departments for the nephrology referral recommendation criteria, especially because of potential impact on appointment access if large numbers of patients were referred

3.2. CKD-CDS security

The exchange and storage of data utilizes a very carefully designed system architecture and infrastructure that has passed multiple layers of security approval. Risk is highly mitigated in several ways: (a) all the data servers reside within the HealthPartners firewall (b) Instead of exchanging the medical record number, an artificially generated Study ID was assigned at the initial visit and used as a token to link the patient in the CDS application with the one in the EMR (c) the patient identifiable information that was initially exchanged included name, date of visit, and dates of labs. Once the operational need to display the patient name on the CDS was fulfilled, the name was scrubbed from the system. Also, a scrubbed limited dataset needed for analysis was removed from the transactional system in order to mitigate risk of data exposure and limit data access to only programmers on the research team. The final system architecture passed security checks and penetration testing before deployment. Furthermore, the layers of the infrastructure are regularly tested for vulnerabilities by our internal security team and also scanned annually by external security firms.

3.3. CKD-CDS pilot

The CKD-CDS was first turned on for a nephrology department and specialty feedback was obtained on the CKD-CDS content for accuracy and clarity. Nephrologists were trained to use the CKD-CDS and how to provide feedback to the research team either electronically through email or a “suggestions tab” on the CDS display, or to write feedback directly on the patient's printed CDS forms and send it back to the research team via internal mail routing.

Prior and during pilot implementation of the CDS, we conducted clinician interviews to gain their perception of the usefulness of Priority Wizard and CKD content. The information learned from these processes were used to further modify and improve the tool.

3.4. CDS data flow

Fig. 2 describes how data related to the CKD-CDS intervention flowed. Hosting the CDS on a secure web service securely linked with the EHR allowed for maximum efficiency and versatility. Maintaining the CKD-CDS on a single web service allowed the study team to make necessary updates to risk equations and algorithms relatively easily. It also avoided potentially disruptive changes to the CDS with Epic upgrades and allowed the CDS to run without risk of slowing down the EHR production environment. The average web service run time was less than one second. Most importantly, the web service algorithms not only drove the intervention, but also assured complete collection of necessary data for analysis at both intervention and control clinics. This greatly reduced the data collection burden and increased the accuracy of collecting data needed for Data and Safety Monitoring of the study, and for assessment of intervention effectiveness at the end of the study.

Fig. 2.

Data flow between CDS web service, EHR, and data repository.

The CKD-CDS content was incorporated into a pre-existing CDS tool used for diabetes and CV risk. When the CKD content was added, the nonproprietary CDS tool was redesigned and rebranded from the name CV Wizard© to Priority Wizard©. The clinic workflow for CDS use was developed previously by our research team in conjunction with clinic leaders in previous successful studies [16,17,19,26] and subsequently adopted and maintained by the care system for diabetes and CV risk factor management. For patients who presented for an appointment at any randomized primary care clinic (intervention or control), when a blood pressure was entered into the electronic health record (EHR) and the vitals section closed, selected clinical data were sent from the EHR to the CDS website to assess eligibility for the Priority Wizard© intervention.

3.5. CKD-CDS description

The CKD-CDS components included four major features as shown in Fig. 3 : 1) Evaluation of EHR data, 2) Recognition of CKD, 3) provision of information to clinicians and patients to discuss treatment options, and 4) Facilitation of appropriate actions, orders and nephrology referrals.

Fig. 3.

CKD-CDS intervention conceptual model.

For the subset of study-eligible CKD patients with sufficient clinical data at the index visit the CDS also estimated and displayed the 5 year probability of kidney failure (dialysis or transplantation) using data on age, sex, eGFR, and albuminuria levels and a published equation (Tangri) validated in more than 700,000 individuals worldwide [27,28] and used in many online medical calculators.

For any eligible patient encounter, the CDS was viewable by clinicians within the EHR and also printable in two companion versions, one designed for patients that displayed “Kidney Health” as a priority (Fig. 4 ) and one designed for clinicians that displayed “Chronic Kidney Disease” as a priority (Fig. 5 ).

Fig. 4.

Patient version of priority wizard with CKD-CDS.

Fig. 5.

Clinician version of priority wizard with CKD-CDS.

3.6. CKD-CDS clinician and staff training

High rates of Priority Wizard CDS printing had already been achieved in the CKD-CDS study clinics and sustained through previous studies and the processes and workflow described below. Specifically, for the CKD intervention at HP and PN intervention clinics, training sessions for clinicians and rooming staff occurred via a live webinar accompanied by lunch served at each of the clinics. The training emphasized the importance of the CKD-CDS and how to incorporate it into their clinic workflows. All training was recorded and made available for listening or viewing after the live webinar. Clinicians could receive one continuing medical education credit for attending.

3.7. CKD-CDS workflow

CKD-CDS eligible patients that accrued at intervention clinics at index visits were eligible to have the intervention display at their index visits and subsequent primary care clinical encounters during the post-index period (including video encounters after August 2021) as long as they continued to meet the inclusion and exclusion criteria. The most important aspect of the workflow relied on rooming staff (the staff who typically prepare a patient for a primary care visit and obtain vital signs) to open and print the patient and clinician versions of Priority Wizard in preparation for the encounter using the following steps: 1) A BPA appears on the EHR screen within seconds of a BP entry for study-eligible patients. 2) A URL link within the BPA is clicked on to automatically open and print patient and clinician versions of the CDS. 3) The rooming staff hand the patient version to the patient to review while waiting to be seen, with encouragement to talk to their clinician about any of their personalized clinical priorities on the page. 4) Rooming staff typically place the clinician version of the CDS on the exam room door for the clinician to review. While there was some resistance to printing, we justified printing the tools by rooming staff for the following reasons: 1) In a clinician survey, 90% of respondents wanted the CDS printed, 70% said they reviewed the printed materials with patients frequently or always, and 90% said that their patients like the information on the handouts; 2) In a pilot that did not prompt rooming staff to print materials for targeted patients with high risk situations, the CDS was triggered for less than 20% of targeted patients, 3) in an online patient questionnaire called MY VOICE used to obtain patient feedback on the prototype CDS patient interface, the most highly preferred time to receive the CDS information was during the rooming process while waiting for the clinician (41%) compared to electronically through the EHR patient portal before the appointment (20%) or after the appointment (20%). With respect to major CV risk factors, 63% of patients said they were likely to take action on a risk factor because of the handout, 58% said it is very convincing to get them to stop smoking, and 55% to improve their cholesterol.

The CDS was also viewable by clinicians online within the EHR, and a smart dot phrase (.ckdrisk) was developed to facilitate clinician documentation of the CKD status within EHR notes and to summarize CKD-related information in the patient's printed after visit summary.

An advantage to this CDS system is that the CDS could be easily updated/refreshed by rooming staff or clinicians if new results became available (such as a repeat BP or a point of care A1C).

In control clinics, the same data flow and workflow was used to generate the Priority Wizard©, but it contained no CKD-CDS information and the BPA to rooming staff was not displayed for patients meeting the CKD intervention eligibility criteria alone.

4. Analysis

4.1. Data sources

The CDS web service collected data elements documented in the electronic health record during the five-year periods prior to each web service call in both intervention and control clinics. It also retained analytic data elements that originated in the EHR and outputs from web service algorithms on a secure server, including a randomly-generated patient identification number, selected demographics, vital signs, smoking status, current and prior medications, medication allergies, diagnostic codes, problem list codes, and lab values. These data elements are used to identify study-eligible patients, CKD care gaps and calculate risk of kidney failure. Additional EHR data are harvested for all study-eligible patients to calculate primary and secondary outcomes, and to identify potentially important safety events for the study's data safety and monitoring plan such as hyperkalemia, hypokalemia, hypoglycemia, hyperglycemia, hospitalizations and emergency department visits.

4.2. Fidelity to the intervention

Use of the CKD-CDS intervention was estimated by tracking CDS print rates at the system, clinic, and clinician levels. The clinics were given the goal to print the CDS at 75% of encounters targeted with a BPA. Monthly print rate reports were provided to clinic leadership comparing print rates across clinics listed in the order of highest to lowest print rate for the current month, and by displaying the print rates of each clinic for the past 12 months in a grid that highlighted print rate levels in green (meeting goal), yellow (close to meeting goal) and red (not close to meeting goal). Each clinic leader was also given a clinician-level use rates report so that they could troubleshoot low use rates at the clinician level. In addition to distribution of monthly print rate information, study staff periodically contacted the lead nurse at clinics with low print rates to inquire if assistance in improving rates was desired. Use of Priority Wizard© was not monetarily incentivized. Fig. 6 shows print rates achieved by treatment group from May 2019 through February 2021. The diabetes and cardiovascular CDS had already been adopted by the care system as part of usual care prior to the start of the CKD-CDS intervention. Because some of the CKD-CDS eligible patients in the CKD usual care arms were eligible for the CDS for reasons other than CKD (for example poorly controlled diabetes or hypertension), there was some printing of the Priority Wizard (without a CKD Priority) in control clinics.

Fig. 6.

CDS use rates.

4.3. Analysis eligibility

The index visit was the first visit for a patient at a randomized clinic during the accrual period that met all study and intervention eligibility criteria and was the point at which the patient was accrued into the study. All subsequent patient encounters of those patients were classified as post-index visit regardless of whether study or intervention eligibility were met. A small number of patients request to be excluded from research through privacy authorizations completed approximately yearly at encounters within the health care system. A list of these patients is maintained in a database at HealthPartners Institute. These patients received the CDS intervention but will be omitted from the analysis. This analysis approach assures complete and non-biased accrual and follow up of study participants in the randomized clinics.

4.4. Study outcomes

The extent to which the CKD-CDS intervention reduces deficits in key elements of CKD care are to be assessed after 12 months of active intervention follow up (with the intervention follow up period extended by 6 months to account for the intervention suspension period that occurred due to the COVID pandemic). Outcomes and outcome definitions are shown in Table 1 .

Table 1.

Study outcomes and definitions.

Study outcome	Analysis denominator	Outcome definition
Recognition of CKD	Study eligible and no documentation of CKD at index	CKD diagnosis code (ICD10) assigned at an inpatient or outpatient encounter, or the entry of CKD diagnosis on the problem list, from index through 18 months post-index
BP control	Study eligible and BP care gap identified at indexa	Most recent systolic BP (SBP) and diastolic BP (DBP) values in the 18 months post-index in the recommended range using office BP measurements
ACEI or ARB use	Study eligible and ACEI / ARB use care gap identified at indexa	Prescription for an ACEI or ARB medication in the 18 months post-index
Glucose control	Study eligible and glucose care gap identified at indexa	Last A1C value in the 18 months post-index below the A1C goal recommendation
Nephrology referral	Study eligible and nephrology visit care gap at indexa	Referral to nephrology (electronic) in the 18 months post-index period.

^aCare gaps for the analysis denominators are the same as those defined in the inclusion criteria.

4.5. Analysis plan

Study hypotheses posit that the CKD-CDS intervention will reduce deficits in key elements of CKD care relative to UC. The planned analysis for each outcome accommodates the covariate constrained cluster randomization and its clinic level intraclass correlation coefficient (ICC) estimated from pilot data. Data elements required for calculating the outcomes are extracted from EHR production tables. The absence of documentation of a care process, vital sign, or medication will not be interpreted as a missing value but rather as indicative of a care process or test not having been performed or medication not prescribed within the health system. Truly missing observations (e.g., systolic BP measured, value not available) will be extremely rare, undetectable and assumed to be missing at random (MAR).

The primary study hypotheses will be tested using random coefficients models in which each outcome will be predicted using distribution-appropriate link functions from clinic fixed effects for treatment group and the balancing factors; patient fixed effects for pre-randomization characteristics (e.g., outcome status); and random effects to accommodate clustering (e.g., clinics, patients within clinics), as needed. A priori power analyses estimated the minimum detectable difference in each of the 5 outcomes listed in Table 1 based on the patient sample size, event rate and clinic and patient level ICCs calculated from pilot data. Absolute between-groups differences of 10% (BP control, ACEI or ARB use, glucose control) and 20% (recognition of CKD, Nephrology referral) were determined to be clinically meaningful. The study is adequately powered to detect a clinically meaningful difference for all major study hypotheses at 80% with two-tailed alpha of 0.05.

5. Impact of the COVID-19 pandemic

The COVID-19 pandemic emerged as a major disruptive force in primary care clinics starting in March 2020 in the middle of this 3-year study, about one month prior to the end of the planned 12-month accrual period. The most immediate impact of the pandemic was a dramatic decrease in office-based clinical encounters, which, by design, were the point of delivery for the CKD-CDS intervention. At the clinics involved in this study, virtual clinical encounters by video or telephone suddenly became the recommended mode for all appointments other than those that required in-person evaluation for diagnosis or treatment. These changes required some primary care clinicians and many patients to relocate primary care services to new locations or seek care through virtual visits that were not necessarily scheduled through their home clinic. An additional challenge was that the CKD-CDS intervention was not originally programmed for use at video visits. This “perfect storm” of COVID-19-related disruptions had the potential to undermine the intervention and the integrity of this clinic randomized study design.

Based on preliminary data and power analysis, we aimed to accrue about 6100 study-eligible adults with stage 3 or 4 CKD (eGFR 15–59 mL/min/1.73 m²) over a 12-month accrual period. Near the start of the pandemic disruptions, we had already accrued more than 7000 patients. Due to the dramatic reduction in the number of possible index visits, potential selection effects of patients attending in-person office visits and inability at the time to deliver the intervention outside of in-person office visits, the research team decided to end accrual of new patients on March 13, 2021. From the start of the pandemic, video visits could be scheduled either at a specific clinic, or within a pool of clinicians drawn from all clinics. Visits with a clinician pool that was drawn from both intervention and UC clinics created potential contamination of care delivered to patients whose index visit had been either in an intervention or UC clinic. This problem was aggravated a few weeks later, when several randomized primary care clinics were closed on short notice, and several more randomized clinics were temporarily re-allocated to provide care only to patients with respiratory symptoms. Due to disruption of the nested patient-provider-clinic structure, the study team temporarily suspended the CKD-CDS intervention on March 23, 2021. During the suspension, which lasted until August 19, 2020, no CKD-CDS content was presented, automatically or on demand, at any type of clinical encounter.

During the suspension, several intervention modifications were made. The determination of whether to display the CDS at encounters was initially determined by the clinic location of the office visit (clinic intervention assignment). Due to concern about feasibility of maintaining the intended clinic level randomization process going forward due to video visit locations and clinic closures, the study team modified the criteria that determines whether CKD-CDS can be displayed at a CKD-eligible visit from using the clinic location of the current visit (i.e., intervention or control clinic location) to the clinic location of the index visit. The goal of this modification was to preserve the integrity of the intervention status assigned to each patient at their index visit regardless of the clinic location, clinician, or encounter modality (telehealth vs. in person clinical encounter) at their subsequent clinical encounters. Thus a patient randomized to usual care at index would never receive the CDS if they were to be seen by a clinician at an intervention randomized clinic and a patient randomized to intervention at index would continue to receive the CKD-CDS if seen at a clinic randomized to usual care. This modification was designed to adhere to intent to treat principles as much as possible and preserved the initial randomization structure to a greater extent than other options that were considered. During the suspension period, the web-service was also reprogrammed to add an Epic trigger for a web service call at the opening of a video encounter so that the CDS could be viewed in video visits. Eligible patients had a BPA display in the clinician's BPA section recommending use of the CDS. Because rooming staff were not involved in a rooming process for telehealth encounters and printing would not be helpful, there was no pop-up BPA programmed. Because the software our care system used for video visits did not include a feature for screen sharing, there was not a practical way to share the CKD-CDS with patients other than verbally. The rates of telehealth encounters and clinician rates of opening the CDS during telehealth encounters will be monitored closely as low use rates in telehealth encounters could impact study outcomes.

These study modifications required approval of the NIH, DSMB, IRB, and care delivery leadership. After a 5-month suspension, the intervention was restarted at eligible clinical encounters and available for use at office and telehealth encounters for intervention assigned patients. Because of the intervention suspension, the planned follow up time for analysis was extended from 12 to 18 months. Because the grant permitted a no cost extension, the study timeline and budget were adapted accordingly.

By early October 2020, all but one of the closed primary care clinics had reopened and all clinics had returned to blended operation seeing both routine office visits and patients with respiratory symptoms. One randomized intervention clinic was permanently closed. Telehealth encounters were still encouraged for all clinic patients when appropriate, and patients could schedule both in-person or telehealth encounters online.

In intervention clinics, print rates were high (approximately 70% of CKD eligible encounters post index) but dropped precipitously in March 2020 due to COVID-19 and our suspension of the CKD-CDS. The print reports sent to clinic leadership were paused at the beginning of the pandemic due to the additional stress and higher priorities the clinics were facing, and the print rates of Priority Wizard© at in-person office visits within the care delivery system declined by about 10–15% at targeted clinical encounters after the intervention was resumed in August 2020.

5.1. Ethical and regulatory approval

Study design and procedures were reviewed and approved by the Institutional Review Board for the care delivery system. The intervention suspension related to COVID and the resumption with design and intervention modifications were approved by the IRB and funder, National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK).

5.2. Informed consent

A waiver of patient and provider consent for this study was requested and granted by the HealthPartners Internal Review Board (IRB).

6. Discussion

This cluster-randomized pragmatic trial conducted at 32 primary care clinics is designed to improve CKD care management and outcomes for patients with stage 3 or 4 CKD (eGFR 15–59 mL/min/1.73 m²). The CKD-CDS identifies patients with stage 3 or 4 CKD and one or more selected care gaps immediately before the start of a primary care clinical encounter and creates an opportunity for timely intervention to recognize and improve CKD care. The high frequency of care gaps we identified through preliminary data underscores the urgent need to improve CKD care.

Effectiveness of the CDS system will be assessed using an intent-to-treat analysis, and outcomes will be assessed through a combination of EHR data and data harvested by the CDS tool itself. Thus, the CDS system acts as both a means of intervention delivery and a means of data collection. Data needed for analysis, including both typical EHR data and CKD-CDS algorithm output for all patients are securely stored as a limited data set on the firewall protected CDS web server and linked for each study patient using a unique random study ID number assigned electronically by the CDS at the index visit. This repository of data can also be used to monitor intervention fidelity such as the proportion of eligible encounters with CDS printing. Because the CDS system runs silently in the background in control clinics, patient identification and data collection occur identically for patients in intervention and control clinics and there is unbiased comparison of similar data across all clinics. These data collection procedures continued at all randomized clinics during the 5-month suspension of intervention deployment while adaptions were being made to respond to COVID-19 disruptions and facilitated monitoring of rates of office, phone, and video encounters during this challenging period.

There were several very important things we learned from this study and challenges posed by the COVID-19 pandemic that could help others conducting pragmatic trials:

• Lesson 1: Close relationships and collaboration between the research team and care delivery leadership are especially important in pragmatic trials to make intervention training and implementation go smoothly and to recognize and adapt quickly to unexpected real-life challenges.

• Lesson 2: The waiver of patient consent granted by the IRB was critical, as the office-based intervention would not have been feasible if formal consent was required for large numbers of study eligible patients in the primary care delivery setting, particularly at UC clinics.

• Lesson 3: A co-variate constrained randomization procedure proved to be a practical method to randomize clinics and created acceptable balance on patient characteristics, insurance status, and a co-existing research initiative related to medication adherence implemented during the CKD study intervention period.

• Lesson 4: Having dedicated web and Epic programmers supported by research funding was key to making timely programming changes required when intervention adaptations were necessary. In this study our research programmers were able to quickly program the intervention to work in telehealth encounters and to change criteria for displaying CDS from the clinic level to the patient level, even when most other organizational programming resources were assigned to a variety of critical pandemic-related tasks. Reliance on stressed information technology resources of the care delivery system could have resulted in delays that would likely have made study completion impossible within the study timeline and budget.

• Lesson 5: Pragmatic trials have practical limitations that often need to be recognized and accepted. One limitation of this study is that approximately 25% of CKD-CDS eligible patients in usual care clinics also received CDS for cardiovascular or diabetes risks due to system-wide CDS installations that pre-dated the CKD-CDS study. Identification of these patients within the data repository will allow us to perform secondary sensitivity analysis to evaluate the magnitude of this effect.

• Lesson 6: Study design and analysis adaptations that solve one problem may introduce other problems. While study eligible patients will not cross over from intervention to control or vice versa, the modifications to the intervention and analysis made to address pandemic-related disruptions could potentially lead to a potential learning contamination effect of UC clinicians (learning contamination could occur if a patient originally assigned to CKD-CDS intervention has a post-index encounter with a UC clinician). This limitation was discussed at length by the study team, project officer, and DSMB with acknowledgement that a beneficial intervention effect might be harder to detect as statistically significant, but a positive result of the study would still be very meaningful. We will be able to quantify the amount of patient crossover to UC clinician encounters and could evaluate the extent of a learning effect on UC clinicians by comparing changes in CKD outcomes in non-study clinics. If a learning effect could be identified, it could actually enhance the value of the CKD-CDS intervention.

• Lesson 7: The change in deployment of the intervention to accommodate telehealth visits and clinic closings could make it more challenging to assess CKD-CDS intervention fidelity. Extending the use of the CKD-CDS intervention to new types of clinical encounters and other adaptations to the study design will increase variation in how the CDS tools are actually used, especially for purposes of shared decision making. There was a steep learning curve when clinicians were required to rapidly pivot to new video technology. However, gathering qualitative and quantitative data on how clinicians use the tool and how much they value it will provide important new information relevant to future CDS design, and will have a bearing on how to interpret the main study findings. Although the study team was able to devise technical solutions to challenges posed by COVID-19, the impact of COVID-19 on patient behavior and receipt of health services through telehealth encounters could impact the potential for the intervention to improve outcomes in a time of greatly increased stress for both clinicians and patients.

7. Conclusion

Almost 30 years ago heath care leaders recognized the potential of EHR-linked CDS to improve chronic disease and preventive care. [29] Despite profound advances in health information technology and the wide use of sophisticated EHRs through most of the U.S. health care system, that potential to improve chronic disease care has yet to be fully realized. [30,31] In particular, there have been few successful efforts to systematically improve CKD care in primary care settings, despite persistent observed deficits in care. [1,2,[9], [10], [11]] Here we report our ongoing efforts to use EHR-linked web-based CDS interventions to improve CKD care, and provide examples of some of the challenges that need to be overcome to achieve this goal. In particular, we describe how disruptions to the delivery of primary care during the COVID-19 pandemic threatened the intervention delivery, thus giving rise to innovations and adaptations, including new ability to deliver CKD-CDS at virtual clinical encounters that occur outside primary care clinic walls. However, the development of some of these innovations, and the very positive response of primary care clinicians and patients to these innovations will inform future efforts to improve the design, delivery, and effectiveness of CDS interventions that aspire to improve chronic disease care in primary care settings. Importantly, the study modifications made in the face of COVID-19 pandemic adversities adhered to intent to treat principles by preserving the initial randomization structure to a greater extent than other solutions to the challenges that were considered. The most important lesson we learned from the challenges we experienced was the importance of having good relationships, respect, and collaboration between the research project staff, care delivery system leaders, and IT departments to implement timely solutions.

7.1. Trial status

The trial was funded through the National Institute of Diabetes and Digestive and Kidney Diseases (R18DK118463). The CKD-CDS intervention will become available throughout the system at the request of the care delivery systems at the conclusion of the study. Analyses is 5–6 months delayed due to the COVID-related temporary intervention suspension, and trial results are expected in late 2021.

Funding

Supported by the National Institute of Diabetes and Digestive and Kidney Diseases (R18DK118463)).

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.