Measuring diabetes guideline adherence with claims data: systematic construction of indicators and related challenges

Abstract

Objectives

Indicators of guideline adherence are frequently used to examine the appropriateness of healthcare services. Only some potential indicators are actually usable for research with routine administrative claims data, potentially leading to a biased selection of research questions. This study aimed at developing a systematic approach to extract potential indicators from clinical practice guidelines (CPG), evaluate their feasibility for research with claims data and assess how the extracted set reflected different types of healthcare services. Diabetes mellitus (DM), Swiss national guidelines and health insurance claims data were analysed as a model case.

Methods

CPG for diabetes patients were retrieved from the Swiss Endocrinology and Diabetes Society website. Recommendation statements involving a specific healthcare intervention for a defined patient population were translated into indicators of guideline adherence. Indicators were classified according to disease stage and healthcare service type. We assessed for all indicators whether they could be analysed with Swiss mandatory health insurance administrative claims data.

Results

A total of 93 indicators were derived from 15 CPG, representing all sectors of diabetes care. For 63 indicators, the target population could not be identified using claims data only. For 67 indicators, the intervention could not be identified. Nine (10%) of all indicators were feasible for research with claims data (three addressed gestational diabetes and screening, five screening for complications and one glucose measurement). Some types of healthcare services, eg, management of risk factors, treatment of the disease and secondary prevention, lacked corresponding indicators feasible for research.

Conclusions

Our systematic approach could identify a number of indicators of healthcare service utilisation, feasible for DM research with Swiss claims data. Some areas of healthcare were covered less well. The approach could be applied to other diseases and countries, helping to identify the potential bias in the selection of indicators and optimise research.

Strengths and limitations of this study.

• This is the first study to explore the limited information from claims data as a source of potential bias when selecting indicators for guideline adherence research.

• A systematic approach was used to develop a comprehensive set of adherence indicators from guidelines, that were then tested for feasibility with claims data.

• Only one clinical area (diabetes mellitus) in a specific country (Switzerland) was investigated as a model case, but the approach could be transferred to other areas of interest.

Introduction

Health insurance claims data are often used to evaluate the appropriateness and intensity of healthcare service utilisation in a country, region or hospital.^{1 2} The Dartmouth Atlas of Health Care in the USA³ is an example of such a healthcare service utilisation intensity map. Recently, a similar project, ‘Versorgungsatlas’, was launched in Switzerland.⁴ Healthcare service appropriateness may additionally be assessed with reference to the clinical indication for the service of interest. In this case, both the patient population with a clinical indication and the corresponding service need to be identifiable. Claims data are used for this purpose in a variety of clinical areas as a valid substitute for clinical data.^5–8 The resulting indicators, proportions of patients receiving the indicated healthcare services, can be regarded as clinical process quality indicators.⁹ They can be used for research as well as policy development—for example, to assess regional access to services or implementation of clinical practice guidelines (CPG).

The indicators are commonly extracted from CPG or Choosing Wisely lists.^{10 11} Indicators of healthcare services overuse have been explored particularly frequently.^12–14 However, the selection and development of the indicators are often non-systematic.¹⁵ Even when based on CPG and guided by clinical relevance, the selection of the analysed indicators is at the discretion of the researchers and may be strongly influenced by the availability of data.¹⁰ Feasibility is frequently used as a criterion to select healthcare quality indicators.¹⁶ As a result, there may be a danger that some healthcare services research and monitoring could be driven by opportunities and data available rather than by a systematic assessment of relevant research questions from a patient or societal perspective.

Specifically, potential bias in indicator selection may arise as claims data typically do not cover all aspects of healthcare. In fee-for-service and bundled payment healthcare reimbursement systems, clinical information is normally not recorded with the reimbursement claim.¹⁷ For example, in Swiss claims databases, outpatient diagnoses and indications are not recorded with the provided outpatient services.¹⁸ With limited clinical information, some patient populations may not be accurately identifiable, and consequently, appropriateness of healthcare service utilisation in these populations may not be assessed. This could lead to a bias of disproportionate application of more easily assessable indicators.

Systematic development and application of indicators of guideline adherence may reduce the possibility of imbalanced selection of topics in healthcare service utilisation research. A few recent studies have systematically extracted indicators of low-value care and evaluated their feasibility for research in primary care¹⁹ or inpatient settings.¹¹ In contrast to extracting only the feasible indicators, the aim of this study was to additionally quantify the indicators unavailable and healthcare services not reflected due to limitations of claims data. Our study aimed to develop a comprehensive, systematic approach to extract and assess guideline adherence indicators for a case model of diabetes mellitus (DM) in Switzerland. CPG were used to systematically develop indicators and mandatory health insurance data to test their feasibility for research. DM is an example of a highly prevalent chronic disease that has a comprehensive national guideline set in Switzerland, published by the Swiss Society for Endocrinology and Diabetes (Schweizerischen Gesellschaft für Endokrinologie und Diabetologie).²⁰ Research on specific DM services indicators based on claims data has already been shown to be feasible.^{21 22} However, the scope of all feasible indicators has not been assessed systematically in diabetes or any other single clinical area across all healthcare settings.

Methods

Systematic development of guideline adherence indicators

In the first step, we designed a systematic approach for the development of guideline adherence indicators for a specific patient population or clinical area. As we have found no previous published detailed approach, we designed the algorithm based on the agreement of all study authors. The final approach is illustrated in figure 1.

Figure 1.

Systematic approach for identifying indicators of guideline adherence for research with claims data.

Second, we applied this approach to DM and Swiss administrative claims data. We retrieved all current guideline and recommendation documents published online on the official website of the Swiss Endocrinology and Diabetes Society in German.²⁰ From the documents retrieved, DM-CPG were identified, resulting in 15 unique documents.

An indicator of guideline adherence was defined as the ratio of patients who actually receive a healthcare service, to all patients with an indication for that specific service. Therefore, to translate a recommendation statement to an indicator, both the patient population with the indication for a service and the patients receiving the service need to be defined. We selected recommendation statements that had both the indication (population) and healthcare service defined. Exclusion criteria for recommendation statements were as follows:

1. The recommendation does not result in any intervention directly affecting the individual patient (eg, a certain risk stratification tool recommended for a specific patient population; documentation of healthcare service delivery).

2. The recommendation expresses no preference for a specific service, in the presence of multiple options (eg, initial therapy for DM patients with symptomatic hyperglycaemia or metabolic decompensation should be started with insulin with or without metformin).

3. The recommendation is a non-specific principle (eg, pharmaceutical therapy should be started without unnecessary delay).

4. No recommendation signalling words (such as should [‘soll’, ‘sollte’], must [‘muss’], we recommend [‘wir empfehlen’], is not recommended [‘ist nicht empfohlen’] and so on) are used in the recommendation (eg, general statements of common current practice).

In order to maintain a comprehensive approach, we did not further exclude recommendation statements based on the nature of the recommended intervention (eg, interventions of general vs specialised practice, positive vs negative recommendations, recommendations for diabetes management vs comorbidities management in diabetic patients).

Eligible recommendation statements were then translated into indicators of guideline adherence. We classified the indicators according to: (1) disease stage (screening—general population; risk factor—patients with an established risk factor for diabetes; disease—patients with diabetes), (2) area of healthcare service (screening—testing of population without symptoms; diagnosis—interventions to confirm or reject the diagnosis; primary prevention—interventions to prevent the onset of disease; control—routine interventions for a patient with a disease; treatment—therapeutic measures; secondary prevention—interventions to prevent disease progression) and (3) intervention type (clinical test—a specific non-laboratory examination, such as blood pressure measurement, medical history taking or a questionnaire administration; laboratory test—blood or urine laboratory analysis; consultation—a consultation of a specialist, such as ophthalmologist, a teaching session or a specific medical advice given during a physician consultation; medication—direct pharmacological intervention; device use—prescription of a medical device, such as for continuous glucose measurement). This classification broadly covers all types, functions and modalities of healthcare services.⁹

Screening indicators for feasibility

As a next step, information from administrative mandatory health insurance claims data was identified and characterised in cooperation with the research team of a major healthcare insurance provider in Switzerland (BB, CB, EB). The claims database consists of information on reimbursed healthcare diagnostic and therapeutic services, deemed appropriate clinically and cost-effective, from all healthcare service providers settings. The information is coded according to international or national classification or tariff systems. Major types of patient information available in the claims database and the corresponding coding systems used are given in table 1. In addition, demographic data, such as sex and age, are available.

Table 1.

Major categories of patient data available in Swiss mandatory health insurance claims databases

Data category	Inpatient data	Outpatient data
Diagnosis	SwissDRG, ICD-10-GM	–
Procedures	SwissDRG, CHOP	Tarmed
Laboratory tests	–	Analysenliste
Medication	–	ATC
Materials and devices	–	MiGeL
Summary of services	SwissDRG	–

Abbreviations of data sources: SwissDRG (Swiss Diagnosis-Related Groups, a classification of inpatient cases, based on diagnoses, procedures and other clinical information), CHOP (Schweizerische Operationsklassifikation, a classification of inpatient procedures), ICD-10-GM (International Classification of Diseases, 10th revision, German Modification, codes for primary and secondary diagnoses for each hospitalisation episode of an inpatient), Tarmed (classification of outpatient procedures and services), Analysenliste (outpatient laboratory test codes), ATC (Anatomical Therapeutic Chemical Classification System, code and quantity of a prescription drug), MiGeL (Mittel und Gegenständeliste, classification of therapeutic materials and devices).

References for the listed data sources are provided in online supplementary file 1.

Next, it was checked if both the addressed population and intervention of the indicator could be identified in the claims data. Indicators were excluded as not feasible if either the patient population or intervention could not be identified. The reason for exclusion was noted. These steps were performed by two authors in duplicate (AU and CB), and instances of disagreement resolved by discussion with the other authors.

For example, one of the identified CPG in Switzerland, Criteria for Good Disease Management of Diabetes in Primary Care (Kriterien für ein „gutes “Disease Management Diabetes in der Grundversorgung), recommends that ‘to fulfil the score criteria, glycated haemoglobin (HbA_1c) should be measured at least two times per year (for diabetes patients)”. The recommendation identifies both the population (diabetes patients) and the intervention (HbA_1c at least biannually), therefore the translated indicator is the fraction of all diabetes patients who get the test done at least biannually. According to our classification, this is an indicator of a laboratory control test performed routinely for diabetes patients. Eligibility of the indicator can now be checked, and the required variables identified in the claims database (to define the population: data on the prescription of diabetes medication; to define the intervention: HbA_1c outpatient test performance).

Patient and public involvement

This study was performed as part of the National Research Programme 74 ‘Smarter Health Care’ of the Swiss National Science Foundation. Patients and public, including policy makers and healthcare services providers, are involved in interpreting, disseminating and translating the overall results of studies conducted under this programme. Patients were not directly involved in the planning and conducting of this study.

Results

From 15 CPG on DM, 93 explicit recommendation statements were translated into indicators (figure 2). Nine (10%) of these indicators could be identified from the administrative claims data available (ie, both the population and the indicated intervention could be identified) and would be principally feasible for further analysis (table 2). Specific codes to identify the patient population and the intervention for each feasible indicator from the claims data sources are provided in online supplementary file 2.

Figure 2.

Selection of clinical practice guidelines, recommendation statements and indicators of guideline adherence.

Table 2.

Feasible indicators based on diabetes recommendation statements

Reference clinical practice guideline		Population	Data source	Intervention	Data source
Primary Health Care Good Disease Management Criteria*	1	DM patients	ATC	HbA1c test twice yearly	Ana
	2	DM patients under 75 years	ATC	Low density lipoprotein (LDL) cholesterol test annually	Ana
	3	DM patients	ATC	Serum creatinine and microalbuminuria tests annually	Ana
	4	DM patients	ATC	Ophthalmologist consultation annually	TM
Gestational Diabetes Screening†	5/6‡	Pregnant women in 24–28 weeks / in the third trimester of pregnancy, if not screened earlier	DRG, Ana	Oral glucose tolerance test (OGTT) or fasting glucose test	Ana
	7	Patients with recent history of gestational diabetes	ICD, DRG	DM screening 4–8 weeks after delivery	Ana
Recommendations Concerning Insulin Therapy for Care-dependent Patients§	8	DM patients dependent on care and insulin therapy	ATC, TM	HbA1c four times and creatinine, serum electrolytes tests 1–2 times yearly	Ana
Blood Glucose Self-Monitoring¶	9	DM patients on intensive insulin therapy	ATC	7–8 blood glucose measurements daily	MiGeL

URL links to the referenced guidelines are provided in online supplementary file 3.

*DE: Kriterien für ein ‘gutes’ Disease Management Diabetes in der Grundversorgung.

†DE: Screening des Gestationsdiabetes.

‡Two recommendation statements are combined for this indicator.

§DE: Empfehlungen bezüglich der Insulintherapie bei unselbständigen Patienten;

¶DE: Selbst-Monitoring der Blutglukose.

Ana, Analyseliste (outpatient laboratory test codes); ATC, Anatomical Therapeutic Chemical Classification System; DM, diabetes mellitus; DRG, diagnosis-related group; HbA1c, glycated haemoglobin; ICD, International Statistical Classification of Diseases and Related Health Problems, 10th revision; MiGeL, Mittel und Gegenständeliste (classification of therapeutic materials and devices); TM, TARMED Tarif (classification of outpatient procedures and services).

For 63 indicators, the patient population could not be identified, and for 67 indicators, the intervention could not be identified with sufficient accuracy (figure 2). Missing clinical information (eg, clinical examination details) prevented the identification of 45 populations and 26 interventions. Missing test results (eg, HbA1c test value) resulted in a failure to identify another 17 populations. In other cases, populations and interventions could not be identified due to unavailable initial date of a diagnosis or a prescription, information on the utilisation frequency of diagnostic tests or devices (except when monthly utilisation rate could be used as a proxy for daily utilisation), and limited information on the use of some devices.

Of the indicators suitable for analysis with claims data, three addressed gestational diabetes and DM screening, five addressed routine screening for DM complications and one the mode of blood glucose measurement (table 2). Indicators suitable for analysis were distributed unequally across disease stages, areas of healthcare services and interventions (table 3). For some areas of healthcare services, no feasible indicators were available. Examples include interventions to diagnose and treat risk factors, medication and device use for disease treatment, and secondary prevention of complications.

Table 3.

Potential and feasible indicators of guideline adherence for diabetes mellitus patients

Disease stage	Area of healthcare service	Intervention type	No. of indicators	No. of feasible indicators
			93	9 (10%)
Screening	Screening	Clinical test	1	0
		Laboratory test	2	2
Risk factor	Screening	Laboratory test	1	0
	Diagnosis	Laboratory test	4	0
	Primary prevention	Consultation	1	0
Disease	Screening	Clinical test	1	0
	Diagnosis	Laboratory test	1	1
		Device use	1	0
	Control	Consultation	3	1
		Clinical test	4	0
		Laboratory test	4	4
	Treatment	Clinical test	3	0
		Laboratory test	12	1
		Consultation	6	0
		Medication	19	0
		Device use	25	0
	Secondary prevention	Consultation	3	0
		Medication	2	0

The categories used for the classification of disease stages, areas of healthcare services and intervention types are explained in the methods section.

Discussion

We developed a systematic approach and identified potential indicators of guideline adherence that can be used for research with Swiss health insurance claims data, using DM as a case model. Only a fraction of indicators generated from CPG recommendation statements was feasible for research, distributed unequally across different healthcare services types and areas.

Claims databases are a frequent source of data for research on healthcare utilisation,¹ increasingly used in Switzerland.18 21 23–26 Limitations of the claims data could lead to some of the potential indicators being over- or under-represented in research with this data source. Ultimately, such a selective set of indicators, when used to link healthcare funding to provider performance, could potentially distort the way healthcare is provided. In our study, no identified indicator was feasible to assess medication or continuous glucose monitoring device use, even though these service categories comprised the largest number of potential indicators (table 3). In contrast, all four identified indicators of control laboratory testing were feasible for research. Such imbalanced distribution confirms a risk of bias, if the indicator selection is guided by the available data rather than the clinical or public health relevance.

Despite potential shortcomings, research on healthcare service utilisation with claims data is attractive as they provide real-life, up-to-date information and may be a valid substitute for clinical data in some cases.⁵ They are gathered from all settings and providers and are recorded in standard variables (table 1). Therefore, they are easily comparable across patients and providers and can be used to construct country-wide healthcare utilisation intensity maps.^{3 4} Longitudinal data can be linked directly to the patient, as implemented in Sweden,²⁷ leading to a unified national database. In Switzerland, the clinical details are currently stored locally by the service provider, and the administrative information is linked to the patient through the insurer, with a strategy to implement a unified eHealth record database in a few years.²⁸

The number of feasible indicators could be increased by allowing more uncertainty in the population or intervention definitions. For example, DM type is not recorded in the claims database. However, it could be estimated from patient characteristics and prescription patterns.²⁹ Indicators requiring knowledge of DM type could be considered on this basis. Further sensitivity analyses for patients hospitalised in the years of interest could be based on inpatient diagnoses, which are recorded in the claims data. However, complex identification algorithms might decrease the sensitivity or specificity of the indicators, and render the retrieval and analysis of the data overly complicated.

In addition to reimbursement claims, other data sources could be explored for systematic application of guidelines adherence indicators. Electronic healthcare record data, gathered on a national level, could provide more clinical details. This could extend the spectrum of the feasible indicators, as major reasons for indicator exclusion in this study were missing clinical examination and laboratory test results. However, health records might face other problems—such as different interpretation across providers, missing data and various quality issues. Another alternative data source is disease registries, providing focused information on guideline adherence in specific diseases, such as myocardial infarction.³⁰

Although the general approach developed in our study is broadly applicable, the model case of diabetes care in Switzerland is limited to the specificities of the national claims data structure and the peculiarities of the analysed disease. Data collected in claims databases vary by country,^{1 31} limiting the possibility to apply the developed indicators internationally. Most of the reported indicators of healthcare service utilisation have been developed in the USA,¹² where unlike in Switzerland outpatient diagnoses in Medicare claims data may be available.³² DM patients in Swiss claims data could be identified by drug prescription; however, DM controlled by diet only would be missed.³³ Some of the indicators developed in this study would rely on indirect identification, for example, the stage of pregnancy would have to be estimated from the date of delivery. All indicators, especially relying on indirect identification algorithms, should ideally be assessed with sensitivity analysis and validated. Algorithms to identify specific patient populations with claims data are frequently validated; however, validation is rarely applied to indicators of utilisation.^{34 35}

DM is an example of a disease for which the medications prescribed are specific enough to suffice for patient identification. Only a limited range of other diseases can be identified in a similarly clear-cut way (eg, hypothyroidism, osteoporosis or tuberculosis). For many frequently prescribed drugs, the indications are wide and overlapping, precluding identification of a specific disease or at least rendering it more uncertain.³⁶ This limits the transferability of our approach to some diseases, although algorithms also considering non-drug outpatient services can provide solutions in some cases (eg, combining a prescription of a drug at a specific dosage by a certain specialty physician). We ensured a systematic approach by pre-specifying the inclusion and exclusion criteria for indicators, and with a concurrent assessment by two authors. However, a further limitation is related to the structure of the CPG used for indicator extraction. Inaccuracies may have arisen due to inherent uncertainties in the formulations of the recommendation statements in these CPG. Recommendation statements were usually not specifically highlighted in the analysed CPG texts and did not have standard formatting or wording, which could lead to a slightly different interpretation by different reviewers. The strength of recommendations or the level of evidence were rarely stated in the analysed guidelines, which would have been useful criteria to classify the indicators. One of the reasons for these shortcomings is that CPG are not specifically designed for indicator construction (just as claims databases are not designed for their assessment). These shortcomings could be minimised by analysing guidelines with clearly structured recommendation statements, such as in the Consensus Report by the American Diabetes Association and the European Association for the Study of Diabetes.³⁷

Some of the feasible DM indicators identified in this study have already been demonstrated to be fruitful for research. Huber et al ²¹ used medication prescriptions to identify a population of drug-treated diabetes patients from the claims database. The investigated interventions were the frequency of HbA_1c, lipid profile and nephropathy status testing and ophthalmologist visits. These indicators were identified as feasible in this study as well.

Conclusions

Using the systematic approach, a number of indicators could be identified for research on diabetes healthcare service utilisation with claims data. Not all healthcare services were represented equally. We believe that our approach could be applied to other diseases and databases in other countries, helping to identify the potential bias in the selection of indicators and optimise such research. In parallel, the structure and content of claims databases may be optimised to allow more efficient healthcare services research in the future. More clearly, structured guidelines with evident recommendation statements would also facilitate their extraction. Comparing the validity of indicators constructed for different national databases could further inform policymakers regarding the advantages of different data collection strategies.