Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Mar 1.
Published in final edited form as: Contemp Clin Trials. 2016 Jan 26;47:196–201. doi: 10.1016/j.cct.2016.01.012

Pre-Visit Prioritization for Complex Patients with Diabetes: Randomized Trial Design and Implementation within an Integrated Health Care System

Richard W Grant 1, Connie S Uratsu 1, Karen R Hansen 1, Andrea Altschuler 1, Eileen Kim 2, Bruce Fireman 1, Alyce S Adams 1, Julie A Schmittdiel 1, Michele Heisler 3,4
PMCID: PMC4818191  NIHMSID: NIHMS756297  PMID: 26820612

Abstract

Background/Aims

Despite robust evidence to guide clinical care, most patients with diabetes do not meet all goals of risk factor control. Improved patient-provider communication during time-limited primary care visits may represent one strategy for improving diabetes care.

Methods

We designed a controlled, cluster-randomized, multi-site intervention (Pre-Visit Prioritization for Complex Patients with Diabetes) that enables patients with poorly controlled type 2 diabetes to identify their top priorities prior to a scheduled visit and sends these priorities to the primary care physician progress note in the electronic medical record. In this paper, we describe strategies to address challenges to implementing our health IT-based intervention study within a large health care system.

Results

This study is being conducted in 30 primary care practices within a large integrated care delivery system in Northern California. Over a 12-week period (3/1/2015 – 6/6/2015), 146 primary care physicians consented to enroll in the study (90.1%) and approved contact with 2496 of their patients (97.6%). Implementation challenges included: (1) Navigating research vs. quality improvement requirements; (2) Addressing informed consent considerations; and (3) Introducing a new clinical tool into a highly time-constrained workflow. Strategies for successfully initiating this study included engagement with institutional leaders, Institutional Review Board members, and clinical stakeholders at multiple stages both before and after notice of Federal funding; flexibility by the research team in study design; and strong support from institutional leadership for “self-learning health system” research.

Conclusions

By paying careful attention to identifying and collaborating with a wide range of key clinical stakeholders, we have shown that researchers embedded within a learning care system can successfully apply rigorous clinical trial methods to test new care innovations.

Keywords: randomized clinical trial, doctor-patient communication, type 2 diabetes, primary care, health informatics interventions, translation into practice

Introduction

In 2014, the United States spent more than $2 trillion on health care, yet quality of care as measured by life expectancy or good control of common chronic conditions such as diabetes lags behind many other countries.1,2 Currently, less than 0.1% of US health expenditures is devoted to research designed to improve how we deliver health care.3 Health care organizations play a leading role in efforts to improve care delivery, with most health organizations (e.g. health insurance plans, accountable care organizations, hospital or outpatient practice networks, and integrated delivery systems) actively measuring and seeking to improve quality, either by implementing best practices identified from other organizations or engaging in often uncontrolled pilot testing and before/after evaluations of practice innovations. As a general rule, quality improvement personnel within these organizations lack the resources to conduct randomized clinical trials.

To address this need to more effectively translate evidence into practice, research funding agencies such as the National Institutes of Health, the Agency for Healthcare Research and Quality, and the Patient-Centered Outcomes Research Institute have increased funding to support research to reduce barriers to evidence-based care. However, Federal grant reviewers, Institutional Review Board members, and health system administrators often make a strong distinction between “research” and “quality improvement.” In this paradigm, research is the domain of externally-funded scientists using rigorous experimental methods, whereas quality improvement is an activity conducted by clinicians and administrators to rapidly improve clinical care processes.4 This distinction creates significant barriers to investigators who seek to apply rigorous research methods such as randomized trial designs to the scientific problem of improving health care delivery.

The care of patients with type 2 diabetes provides an excellent model for testing new approaches for how health systems manage patients with complex chronic conditions. In the US, over 29 million people (9.3% of the population) have diabetes, representing an annual health care cost of $245 billion dollars.5 A robust evidence base from clinical trials and large cohort studies informs the clinical management of diabetes and associated risk factors.6 Despite gains in the quality of diabetes care in the past decade, the majority of patients with diabetes do not reach all goals of evidence-based management.7 One barrier may be the difficulty in providing effective diabetes care during brief and complicated primary care visits. In this report, we describe the rationale, study design, process evaluation plans, and major strategies for successfully implementing a Federally-funded, cluster-randomized clinical trial to support more effective diabetes primary care within a large integrated care system.

Methods

Setting

This study was implemented within Kaiser Permanente Northern California, a non-profit integrated care delivery system providing care for over 3.6 million members throughout Northern California, including more than 260,000 members with diabetes. The distribution of member demographic and socioeconomic factors is diverse and similar to that of the area population.8 Kaiser Permanente members are actively encouraged to register for an on-line patient portal account linked to the system’s electronic health record. Features of this portal include secure e-mail messaging to their provider, on-line appointment booking and prescription refills, and review of most laboratory results. As of 2014, 65% of members were registered users of the portal, including nearly half of patients with diabetes. Use of the on-line portal was somewhat lower by older members and non-White race/ethnic groups.

The conception, planning, and implementation of this Pre-Visit Prioritization Study was undertaken by research scientists affiliated with the Division of Research, an independent research unit embedded within the Kaiser Permanente integrated health delivery system. First established in 1961, the Division includes 58 faculty research scientists and over 500 support staff engaged in epidemiologic and health services research. This Pre-Visit Prioritization study is externally funded (as is over three-quarters of research funding supporting Division investigators). The existence of an independent research division within an integrated care system provides a natural advantage for conducting research such as the project described in this manuscript.9 Funding for this study was provided by the National Institute of Diabetes, and Digestive and Kidney Diseases (DK R01-099108).

Conceptual Model for the Pre-Visit Prioritization Study

The rationale for the Pre-Visit Prioritization Study is based on the recognition that a large and growing number of patients with type 2 diabetes live with multiple concurrent conditions that require complicated medical regimens and daily self-management behaviors.5,10–13 This comorbidity burden, along with the depression, pain, and functional limitations associated with multimorbidity, contributes to medication non-adherence and presents a barrier to effective diabetes care.14–17 For these patients, diabetes management decisions and treatment goals must be considered within the larger context of competing health concerns and patient goals and preferences.18 In parallel with increasing patient complexity, there has been a substantial rise in evidence-based guidelines that primary care providers must address at each visit.19,20 Increasing patient and visit complexity present formidable challenges to effective diabetes management in primary care. For patients with diabetes, competing demands and lack of visit preparation may lead to less effective visit encounters.21,22 The combination of ineffective patient-provider communication during primary care visits and sub-optimal care planning represents a major barrier to the effective translation of evidence into practice.

We hypothesized that providing complex patients with a tool to help them identify their top one or two concerns prior to the visit – and sending these priorities to the primary care provider in a way that fits with provider workflow – would lead to more effective visit interactions by reducing communication barriers and thereby increasing the efficiency and productivity of the visit. The goal of this intervention model was to maximize the value of available time during the primary care encounter by facilitating communication between a prepared patient who has had time before the visit to identify his/her priorities and an informed provider who is aware of the patient’s care priorities at the beginning of the visit. This approach is supported by the Chronic Care Model, which provides a framework for pro-active approaches to improving care in health systems,23 by the Patient Centered Medical Home Model which places the patient at the center of the care team,24 and by the 4 Habits Model for patient-provider communication, which emphasizes the importance for providers to invest in the beginning of the visit to establish rapport, elicit patient goals, demonstrate empathy, and plan the visit.25

Study Design Considerations

We conceptualized the Pre-Visit Prioritization intervention as a tool to help primary care physicians during visits with their complex patients. Thus, we randomized at the provider level, with patients clustered within provider. This strategy avoids the information contamination that could occur if the same provider treated a control patient after treating an intervention patient (e.g. the provider could potentially change her behavior towards the second patient based on her experience with the prior patient).26 Because the intervention was restricted exclusively to visits by eligible patients with their own primary care provider, there is no patient crossover between providers. In addition, the intervention was designed to not involve other care team members or staff. For this reason, we did not randomize at the practice or facility level, which would have reduced our effective power to show differences between study arms. PCP eligibility criteria included having eligible patients and not planning to retire or transfer in the next year.

We used a functional definition of patient complexity based on sub-optimally controlled hemoglobin A1c levels. Patient eligibility criteria included being an adult with type 2 diabetes, last measured hemoglobin A1c > 8.0%, and registered as active on the Kaiser Permanente patient portal (called kp.org). Because the Pre-Visit Prioritization intervention was designed as a tool to support physicians in conducting visits with their complex patients, our plan was to recruit, consent, and randomize primary care providers. Enrolled physicians reviewed a paper list of their potentially eligible patients and gave approval for which patients to whom we could send the Pre-Visit Prioritization survey. Lists ranged from 5 – 52 patients presented on a single sheet of paper with name, gender, and age.

Choice of study outcomes

The primary outcome for this study is the change difference in hemoglobin A1c from baseline to 12-month follow-up comparing patients in the intervention and control arms. A patient’s hemoglobin A1c level is a result of multiple factors, most proximally dose and adherence to medications and lifestyle changes over the prior 8–12 weeks. These factors, in turn, reflect multiple influences such as patient understanding of the care plan, physician willingness to intensify therapy, patient motivation to improve diabetes control, and patient adherence to these plans. Our causal model for this intervention is that improved patient-provider communication during visits will, over time, result in more effective care plans for diabetes control. Secondary clinical outcomes include changes in diabetes-related risk factor control (i.e. blood pressure and cholesterol) and differences in medication adherence as measured by prescription refill gaps.

Sample size and power calculation

While there have been no directly comparable prior interventions similar to the Pre-Visit Prioritization tool, we used prior research involving non-pharmacologic system interventions for diabetes care27 to estimate our potential effect size. We designed the study to have 80% power to detect a change difference of 0.25% in hemoglobin A1c between study arms based on a cluster size of 10 patients per provider, an intra-cluster correlation co-efficient of 0.01, and < 5% missing outcome data. We chose 0.25% as the minimal clinically significant difference based on the fact that intensive behavioral interventions are often considered successful with a 0.5% relative improvement. Given that our intervention is population-wide and light touch, a modest reduction of 0.25% would represent a significant difference for a health system.

Process Evaluation Plans

Process evaluations are recommended to better understand the “black box” of how complex clinical trial interventions are actually implemented.28 In this report, we present several of the key process evaluation elements we measured during the initial implementation phase.

Results

Developing the Health IT Intervention

Informed by our conceptual model29 and by published studies in this area, we applied User-Centered Design principles to develop our tool.30 A key task of this approach was to elicit feedback and specific advice from the final users of the intervention. In our first development phase, we conducted 4 patient focus groups and interviewed 43 primary care providers. Patient groups included 29 participants. Fourteen were women, 15 were African-American, one was Latino, the mean age was 59.0 ± 11.0 years and the mean HbA1c was 9.1% (± 1.3%). Providers were either met individually (n=7) or during weekly practice meetings (n = 36 providers at 3 primary care practices). For both patients and providers, we described our study proposal and elicited feedback regarding the type and format of information they would find most useful to communicate prior to a primary care visit.

This preliminary User-Centered Design work helped us to define the parameters of the tool and to develop a list of priorities relevant to both patients and providers. Some of the key insights from this first phase that informed tool development included: 1) Neither patients nor physicians were willing to spend more than a few minutes with the tool; 2) Any information communicated to the provider needed to be actionable, and 3) The tool could not increase the work of the visit. During this development phase, we also presented our intervention plan to key stakeholders within the organization (e.g. quality improvement directors, electronic health record managers, and clinical chiefs of service) to establish support for the study.

Designing the Pre-Visit Prioritization Intervention

Based on stakeholder input, we designed a tool linked to the electronic health record to help primary care providers conduct visits with their patients with poorly controlled diabetes. In the Pre-Visit Prioritization study, an email message is sent via the secure patient portal on the primary care provider’s behalf to eligible patients whenever they schedule an appointment with an intervention physician. This secure email invites the patient to consider their top priorities for the upcoming visit from a list of 5 key domains (diabetes-specific symptoms, medication concerns, recent life changes, issues related to mood or motivation, and non-diabetes symptoms or concerns). These 5 domains were defined based on patient and provider interviews and informed by previously published literature.31–33 A link in the message sends them to a survey form within the patient portal where they choose their 1 or 2 top priorities to discuss at their upcoming visit. Patients can also add free text explanation after making their choices. When the survey is submitted by the patient, their information becomes available within the scheduled visit encounter section of the electronic health record for the provider to view prior to or at the beginning of the visit.

Approval for Informed Consent Strategy

Permission to waive informed consent from patients for the overall study implementation was based on two key factors required by the “Common Rule” set out by US Federal Policy for the Protection of Human Subject34: The study was deemed to be very low risk to patients, and requiring informed consent from each patient would have substantially reduced the generalizability and feasibility of the study. Our study met criteria for minimal/low risk because: 1) we were adapting two already existing tools (secure electronic messages to patients and electronic patient surveys submitted to the medical record); 2) we were using standard clinical practices (secure messages from providers before visits); 3) all clinical decision making remained directly under the control and responsibility of the patient’s primary care physician; 4) the Pre-Visit Prioritization tool was designed to support rather than change best practices; and, 5) only patients preapproved by their primary care physicians received the Pre-Visit Prioritization secure message (if provider randomized to intervention arm). The secondary criterion of generalizability/feasibility was also of major importance in light of the self-selection process that occurs when patients are asked to actively consent for research. The scientific value of our study to the health care system would have been significantly diminished if the patients least likely to agree to participate in research (often racial/ethnic minorities and patients with less motivation or engagement with health care)35 were not included. Narrowing the participants to the most motivated subset of patients willing to participate in a clinical research trial would hamper the generalization of results and therefore raise the potential that the study would ultimately be uninformative.

Recruitment and Enrollment

Primary care physicians and their patients were recruited over a 12-week period (3/2015 – 6/2015) from 30 primary care practices at 13 different facilities across 4 Northern California counties. The study was presented during routine physician practice meetings attended by 162 primary care physicians, of whom 146 (90%) consented to enroll. Enrolled providers excluded very few potentially eligible patients (60 of 2556, 2.4%). The clinical trial patient study cohort was 58.4% male, 39.7% non-White race/ethnicity, with a mean age of 60.1 (± 10.5) years and last measured hemoglobin A1c of 9.1% (± 1.1%).

Plans for Process Evaluation

Because our model is based on improving communication during the visit, we are also collecting information about the visit itself through post-visit telephone surveys (n = 400). This 35-item survey includes the following telephone-validated instruments: Stanford Communication with Physicians Scale,36 Consumer Assessment of Healthcare Providers and Systems,37 Patient Assessment of Chronic Care,38 Perceived Involvement in Care Scale,39 Perceived Efficacy in Patient – Physician Interactions,40 and the Effective Consumer Scale.41 In our initial experience, the survey requires less than 15 minutes to complete. Because we are collecting research data from patients, we obtained IRB approval for this component of the study. We will compare responses between intervention and control patients to test the hypothesis that patients using the pre-visit tool will have more effective communication experiences with their providers.

A key component of process evaluation is to examine in more detail how the intervention was implemented. To this end, we plan to obtain informed consent from patients and providers to audiotape 40 visits from patients in the intervention arm. Transcripts will be inductively coded to qualitatively examine whether and how providing patients the opportunity to define the visit priorities before the visit influences the content of the visit itself.42,43 In addition, we will quantitatively assess the number of secure messages sent to patients, the proportion opened by the patient, and the proportion of visit priorities submitted. Finally, we plan to interview a selected sub-set of intervention patients to identify reasons why patients submitted vs. did not submit their pre-visit priorities.

Discussion

The wide range of potential patient and provider concerns during a time-limited visit represents an important barrier to delivering effective evidence-based diabetes care. We present the design and initial implementation results of an NIH-funded randomized clinical trial to address this problem and discuss the challenges and potential solutions to conducting this study within an integrated health delivery system. The strategies we adopted to address these challenges may be generalizable to other investigators working in this area.

We identified three fundamental strategies required for successful implementation of large scale care delivery interventions.

  1. Navigating research vs. quality improvement requirements: Innovative research interventions to change care delivery require a strongly supportive environment. Because the health care system itself is the “laboratory” for research, it is imperative that the system leadership foster a learning health system culture.44 In initially developing the proposal, we needed to formulate our research question so that it was both scientifically innovative (and therefore potentially fundable by external grant funding agencies) yet close enough to standard of care that it could be widely tested within a health care system. We chose to focus on the primary care of complex patients because this represents a major challenge for care systems and is therefore amenable to innovation. We learned from our experience that investigators conducting delivery science research (defined broadly as any research directed towards changing the way care is delivered) need to carefully balance the extended timeline from grant submission to funding decision with the shorter time horizons typical of operational quality improvement efforts. For the Pre-Visit Prioritization Study, we engaged in timely communication with organizational leaders during the initial submission phase and again once a fundable score was obtained (both of which were well before the funding start date). A key facilitator for implementation was the strong support from our care system leaders for research on a problem of high priority to them that could help inform future quality improvement efforts.

    Because randomized trials require equipoise, translating our concept into an actual intervention also required that all stakeholders (research team, funders, clinical partners) be willing to take a chance on whether or not our Pre-Visit Prioritization strategy would be successful. This type of delivery science research requires that stakeholders and researchers be highly flexible and adaptable. Thus, it is more likely to be effectively conducted within organizations with a cultural emphasis on being a learning care system.45 Perhaps the most important facilitator for overcoming this implementation challenge was that the federally-funded research team was embedded within the care delivery system and had close working relationships with care system leaders who strongly supported the creation of new knowledge that could be of potential value for the care system.

  2. Informed consent considerations: The need for informed consent by patients within learning health systems must be carefully evaluated by criteria that are appropriate for this type of research. As leading ethicists in the field have argued, research focused on health care delivery must be evaluated in terms of the true risk posed to patients relative to the burden potentially imposed by stringent consent requirements.46 This is a relatively new paradigm for evaluating ethical research practices. By working within established clinical practices, involving patients’ primary care physicians in the screening process, and implementing an intervention that supports evidence-based care while leaving all clinical decision-making in the hands of the patient’s primary care physician, the Pre-Visit Prioritization Study demonstrates some of the key elements required for an ethical study that waives patient informed consent. Our decision to randomize physicians and to request waiver of informed consent for their patients required that our Institutional Review Board carefully examine the ethics and safety of allowing clinical research without direct patient consent. For delivery science studies such as the Pre-Visit Prioritization study, the experimental interventions test different models of care system organization and care delivery. The research team worked closely with our Institutional Review Board to establish that the intervention was very low risk and that the waiver of patient consent was important for the research to be feasible. In weighing the potential benefits and risks of waiving patient-level consent, our Institutional Review Board was guided by the ethical framework that individual patient informed consent could be waived if the potential risk to patients was very low and the study could not be feasibly conducted otherwise. This ethical framework is critical for conducting innovative delivery science research within learning health care systems.46

  3. Introducing a new tool into a highly time-constrained workflow: The work of patient care is highly complex, and care team members are severely pressed for time.19,47,48 More so than any other type of research endeavor, conducting research interventions that perturb the usual care of a health system requires early, active, and ongoing collaboration with all relevant stakeholders.49 This engagement extends from the operational leaders overseeing clinical practices to the physicians and staff whose workflow is affected by the study, the health IT designers and electronic medical record managers through whom the intervention is delivered, and the patients and the caregivers for whom the intervention is intended to benefit. Early strategic decisions about the health IT tool for our study required weighing the relative advantages of creating a new stand-alone tool vs. adapting existing functionality of the health system’s electronic health record. While creating a new stand-alone tool would allow more design input by the research team and greater flexibility in the development phase, the advantage of adapting our intervention model to existing electronic medical record functionality is the greater ease of dissemination if the study is successful. For the Pre-Visit Prioritization Study, we opted to work very closely with the electronic health record using existing functionalities (e.g. secure patient emails, visit-linked surveys). Although this limited the format options for delivering the intervention, the research team focused its energies on the content of the tool to create the most innovative and effective intervention. The research team spent the first year developing the content in conjunction with patients (via focus groups) and providers (via team meetings and individually among physician leaders). Involvement of providers in the development phase had the additional advantage that these physicians were subsequently more willing to participate (and encouraged their peers to participate) in the evaluation of the Pre-Visit Prioritization tool in our clinical trial.

Several potential limitations of our study design should be noted. Our intervention requires an electronic health record linked to a patient on-line web portal to allow transfer of patient-derived data from patient to provider. While this infrastructure allows us to test our hypothesis in a way that more effectively fits physician workflow, results are not immediately generalizable to patients without on-line access or to systems currently without robust electronic health records. We were also fortunate to work within a supportive integrated care system. While our care environment is somewhat unique, the basic concept of helping patients prioritize before visits can be tested in other care environments, particularly systems that have electronic medical records. In addition, because all patients (intervention and control) see non-physician health team members, there is a theoretical concern for contamination across study arms. However, because the intervention is focused specifically on the visit with the primary care provider and is closely temporally linked with this visit, it seems highly unlikely that non-physician team members would be a meaningful source of contamination between treatment arms.

A larger issue raised by the Pre-Visit Prioritization Study is the imperative faced by researchers to make care delivery innovations fit in with existing workflows rather than radically overturn existing care structures. For pragmatic reasons, we prioritized fitting into provider workflow as a critical requirement for successful intervention implementation. Such an incremental approach stands in contrast to the more radical changes that might be necessary to significantly re-model our care delivery system. To successfully test new models of care delivery that may have a more disruptive impact on current workflow practices, we will need more flexible practices or innovation labs. Some organizations have created space for this type of innovation.

In summary, the Pre-Visit Prioritization study exemplifies a relatively novel category of large-scale randomized clinical trial conducted to test different care delivery strategies for complex patients. This type of clinical trial addresses the final stage of research translation from clinical evidence into clinical practice, a critical (and underfunded) challenge that must be overcome before all patients can receive the full benefit of medical research advances. We found that conducting innovative clinical research within a learning health system requires strong institutional leadership support, a close working relationship with the Institutional Review Board, and energetic collaboration from the full range of stakeholders affected by the study intervention.

Figure 1.

Figure 1

Open in a new tab

Screen shot of electronic medical record showing the primary care physician’s view of patient-submitted priorities in the visit encounter progress note. Clicking on “Add To Note” brings patient responses into the open progress note being created for that visit.

Table 1.

Summary of Key Findings

Design Domain Key Finding Advice
Research vs. quality improvement requirements Clinical trial interventions to change care delivery require a strongly supportive environment Develop collaborations with key stakeholders very early in the grant writing process
Informed consent considerations The need for patient informed consent must be evaluated by criteria that are appropriate for delivery science research Collaborate with the IRB committee to determine whether an intervention is low risk for patients
Introducing a new tool into a highly time-constrained workflow The impact of care delivery interventions on existing workflow must be carefully considered Develop early, active, and ongoing collaboration with all relevant users (e.g. patients, providers, and staff)
Open in a new tab

Acknowledgments

We thank Tracy Lieu MD MPH and Karen Emmons PhD for reviewing earlier drafts of this manuscript.

Funding

This work is supported by the National Institute of Diabetes and Digestive and Kidney Diseases (R01DK099108 and P30DK092926)

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Clinical Trial Registration: NCT02375932

References

  • 1.McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635–2645. doi: 10.1056/NEJMsa022615. [DOI] [PubMed] [Google Scholar]
  • 2.Garber AM, Skinner J. Is American health care uniquely inefficient? J Econ Perspect. 2008;22:27–50. doi: 10.1257/jep.22.4.27. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Academy Health Funding Report. [Accessed 2015]; at https://www.academyhealth.org/files/publications/fundingreport.pdf. [Google Scholar]
  • 4.Kass NE, Faden RR, Goodman SN, Pronovost P, Tunis S, Beauchamp TL. The research-treatment distinction: a problematic approach for determining which activities should have ethical oversight. Hastings Cent Rep. 2013:S4–S15. doi: 10.1002/hast.133. Spec No. [DOI] [PubMed] [Google Scholar]
  • 5.2011 National Diabetes Fact Sheet. Centers for Disease Control and Prevention. [Accessed September 13, 2012]; at http://www.cdc.gov/diabetes/pubs/estimates11.htm.
  • 6.Grant RW, Kirkman MS. Trends in the Evidence Level for the American Diabetes Association's "Standards of Medical Care in Diabetes" from 2005 to 2014. Diabetes Care. 2015;38:6–8. doi: 10.2337/dc14-2142. [DOI] [PubMed] [Google Scholar]
  • 7.Ali MK, Bullard KM, Saaddine JB, Cowie CC, Imperatore G, Gregg EW. Achievement of Goals in U.S. Diabetes Care, 1999–2010. New England Journal of Medicine. 2013;368:1613–1624. doi: 10.1056/NEJMsa1213829. [DOI] [PubMed] [Google Scholar]
  • 8.Karter AJ, Moffet HH, Liu J, et al. Achieving good glycemic control: initiation of new antihyperglycemic therapies in patients with type 2 diabetes from the Kaiser Permanente Northern California Diabetes Registry. Am J Manag Care. 2005;11:262–270. [PMC free article] [PubMed] [Google Scholar]
  • 9.Grant RW, Schmittdiel JA. Building a Career as a Delivery Science Researcher in a Changing Health Care Landscape. Journal of General Internal Medicine. 2015;30:880–882. doi: 10.1007/s11606-015-3178-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Grant RW, McCarthy EP, Singer DE, Meigs JB. Frequent outpatient contact and decreasing medication affordability in patients with diabetes from 1997 to 2004. Diabetes Care. 2006;29:1386–1388. doi: 10.2337/dc06-0196. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Grant RW, Pirraglia PA, Meigs JB, Singer DE. Trends in complexity of diabetes care in the United States from 1991 to 2000. Arch Intern Med. 2004;164:1134–1139. doi: 10.1001/archinte.164.10.1134. [DOI] [PubMed] [Google Scholar]
  • 12.Selvin E, Coresh J, Brancati FL. The burden and treatment of diabetes in elderly individuals in the U.S. Diabetes Care. 2006;29:2415–2419. doi: 10.2337/dc06-1058. [DOI] [PubMed] [Google Scholar]
  • 13.Wolff JL, Starfield B, Anderson G. Prevalence, expenditures, and complications of multiple chronic conditions in the elderly. Arch Intern Med. 2002;162:2269–2276. doi: 10.1001/archinte.162.20.2269. [DOI] [PubMed] [Google Scholar]
  • 14.Kalyani RR, Saudek CD, Brancati FL, Selvin E. Association of diabetes, comorbidities, and A1C with functional disability in older adults: results from the National Health and Nutrition Examination Survey (NHANES), 1999–2006. Diabetes Care. 2010;33:1055–1060. doi: 10.2337/dc09-1597. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Kerr EA, Heisler M, Krein SL, et al. Beyond comorbidity counts: how do comorbidity type and severity influence diabetes patients' treatment priorities and self-management? Journal of General Internal Medicine. 2007;22:1635–1640. doi: 10.1007/s11606-007-0313-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Beverly EA, Wray LA, Chiu CJ, Weinger K. Perceived challenges and priorities in comorbidity management of older patients with Type 2 diabetes. Diabetic Medicine. 2011;28:781–784. doi: 10.1111/j.1464-5491.2011.03282.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Elliott RA, Ross-Degnan D, Adams AS, Safran DG, Soumerai SB. Strategies for coping in a complex world: adherence behavior among older adults with chronic illness. Journal of General Internal Medicine. 2007;22:805–810. doi: 10.1007/s11606-007-0193-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Reuben DB, Tinetti ME. Goal-oriented patient care--an alternative health outcomes paradigm. N Engl J Med. 2012;366:777–779. doi: 10.1056/NEJMp1113631. [DOI] [PubMed] [Google Scholar]
  • 19.Abbo ED, Zhang Q, Zelder M, Huang ES. The increasing number of clinical items addressed during the time of adult primary care visits. Journal of General Internal Medicine. 2008;23:2058–2065. doi: 10.1007/s11606-008-0805-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Pollak KI, Krause KM, Yarnall KS, Gradison M, Michener JL, Ostbye T. Estimated time spent on preventive services by primary care physicians. BMC Health Services Research. 2008;8:245. doi: 10.1186/1472-6963-8-245. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Weiner SJ, Schwartz A, Weaver F, et al. Contextual errors and failures in individualizing patient care: a multicenter study. Annals of Internal Medicine. 2010;153:69–75. doi: 10.7326/0003-4819-153-2-201007200-00002. [DOI] [PubMed] [Google Scholar]
  • 22.Barry CA, Bradley CP, Britten N, Stevenson FA, Barber N. Patients' unvoiced agendas in general practice consultations: qualitative study. Bmj. 2000;320:1246–1250. doi: 10.1136/bmj.320.7244.1246. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Wagner EH, Austin BT, Von Korff M. Organizing care for patients with chronic illness. Milbank Q. 1996;74:511–544. [PubMed] [Google Scholar]
  • 24.AHRQ: Defining the PCMH. (Accessed at https://pcmh.ahrq.gov/page/defining-pcmh.)
  • 25.Stein T, Frankel RM, Krupat E. Enhancing clinician communication skills in a large healthcare organization: a longitudinal case study. Patient education and counseling. 2005;58:4–12. doi: 10.1016/j.pec.2005.01.014. [DOI] [PubMed] [Google Scholar]
  • 26.Chuang JH, Hripcsak G, Heitjan DF. Design and analysis of controlled trials in naturally clustered environments: implications for medical informatics. Journal of the American Medical Informatics Association : JAMIA. 2002;9:230–238. doi: 10.1197/jamia.M0997. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Tricco AC, Ivers NM, Grimshaw JM, et al. Effectiveness of quality improvement strategies on the management of diabetes: a systematic review and meta-analysis. Lancet. 2012;379:2252–2261. doi: 10.1016/S0140-6736(12)60480-2. [DOI] [PubMed] [Google Scholar]
  • 28.Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials. 2013;14:15. doi: 10.1186/1745-6215-14-15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Grant RW, Adams AS, Bayliss EA, Heisler M. Establishing visit priorities for complex patients: A summary of the literature and conceptual model to guide innovative interventions. Healthcare. 2013;1:117–122. doi: 10.1016/j.hjdsi.2013.07.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Abras CM-K, D Preece J. User-Centered Design. In: Bainbridge, editor. Encyclopedia of Human-Computer Interaction. Thousand Oaks: Sage; 2004. [Google Scholar]
  • 31.Hofer TP, Zemencuk JK, Hayward RA. When there is too much to do: how practicing physicians prioritize among recommended interventions. Journal of General Internal Medicine. 2004;19:646–653. doi: 10.1007/s11606-004-0058-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Tinetti ME, McAvay GJ, Fried TR, et al. Health outcome priorities among competing cardiovascular, fall injury, and medication-related symptom outcomes. J Am Geriatr Soc. 2008;56:1409–1416. doi: 10.1111/j.1532-5415.2008.01815.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Fried TR, McGraw S, Agostini JV, Tinetti ME. Views of older persons with multiple morbidities on competing outcomes and clinical decision-making. J Am Geriatr Soc. 2008;56:1839–1844. doi: 10.1111/j.1532-5415.2008.01923.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Federal Policy for the Protection of Human Subjects ('Common Rule') (Accessed at http://www.hhs.gov/ohrp/humansubjects/commonrule.)
  • 35.George S, Duran N, Norris K. A systematic review of barriers and facilitators to minority research participation among African Americans, Latinos, Asian Americans, and Pacific Islanders. Am J Public Health. 2014;104:e16–e31. doi: 10.2105/AJPH.2013.301706. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Lorig KSA, Ritter P, González V, Laurent D, Lynch J. Outcome Measures for Health Education and other Health Care Interventions. Thousand Oaks CA: Sage Publications; 1996. [Google Scholar]
  • 37.CAHPS 2.0 Survey and Reporting Kit. Rockville MD: Agency for Health Care Policy and Research; 1999. [Google Scholar]
  • 38.Glasgow RE, Wagner EH, Schaefer J, Mahoney LD, Reid RJ, Greene SM. Development and validation of the Patient Assessment of Chronic Illness Care (PACIC) Medical care. 2005;43:436–444. doi: 10.1097/01.mlr.0000160375.47920.8c. [DOI] [PubMed] [Google Scholar]
  • 39.Lerman CE, Brody DS, Caputo GC, Smith DG, Lazaro CG, Wolfson HG. Patients' Perceived Involvement in Care Scale: relationship to attitudes about illness and medical care. Journal of general internal medicine. 1990;5:29–33. doi: 10.1007/BF02602306. [DOI] [PubMed] [Google Scholar]
  • 40.Maly RC, Frank JC, Marshall GN, DiMatteo MR, Reuben DB. Perceived efficacy in patient-physician interactions (PEPPI): validation of an instrument in older persons. J Am Geriatr Soc. 1998;46:889–894. doi: 10.1111/j.1532-5415.1998.tb02725.x. [DOI] [PubMed] [Google Scholar]
  • 41.Kristjansson E, Tugwell PS, Wilson AJ, et al. Development of the effective musculoskeletal consumer scale. J Rheumatol. 2007;34:1392–1400. [PubMed] [Google Scholar]
  • 42.Glaser BG, Strauss AL. The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago: Aldine; 1967. [Google Scholar]
  • 43.Miles MB, Huberman AM. Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: Sage Publications; 1994. [Google Scholar]
  • 44.Schmittdiel JA, Grumbach K, Selby JV. System-based participatory research in health care: an approach for sustainable translational research and quality improvement. Ann Fam Med. 2010;8:256–259. doi: 10.1370/afm.1117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Grossmann C, Goolsby WA, Olsen L, McGinnis JM for the Institute of Medicine. Engineering a Learning Healthcare System: A Look at the Future. Washington, D.C: National Academies Press; 2011. [Google Scholar]
  • 46.Faden RR, Beauchamp TL, Kass NE. Informed consent, comparative effectiveness, and learning health care. N Engl J Med. 2014;370:766–768. doi: 10.1056/NEJMhle1313674. [DOI] [PubMed] [Google Scholar]
  • 47.Morrison I. The future of physician's time. Annals of Internal Medicine. 2000;132:80–84. doi: 10.7326/0003-4819-132-1-200001040-00013. [DOI] [PubMed] [Google Scholar]
  • 48.Parchman ML, Romero RL, Pugh JA. Encounters by patients with type 2 diabetes--complex and demanding: an observational study. Ann Fam Med. 2006;4:40–45. doi: 10.1370/afm.422. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Schmittdiel JA, Desai J, Schroeder EB, et al. Methods for engaging stakeholders in comparative effectiveness research: A patient-centered approach to improving diabetes care. Healthcare. 2015;3:80–88. doi: 10.1016/j.hjdsi.2015.02.005. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES