Validation of a Pediatric Diabetes Case Definition Using Administrative Health Data in Manitoba, Canada

Abstract

OBJECTIVE

To validate a case definition for diabetes in the pediatric age-group using administrative health data.

RESEARCH DESIGN AND METHODS

Population-based administrative data from Manitoba, Canada for the years 2004–2006 were anonymously linked to a clinical registry to evaluate the validity of algorithms based on a combination of hospital claim, outpatient physician visit, and drug use data over 1–3 years in youth 1–18 years of age. Agreement between data sources, sensitivity, specificity, negative (NPV) and positive predictive value (PPV) were evaluated for each algorithm. In addition, ascertainment rate of each data source, prevalence, and differences between subtypes of diabetes were evaluated.

RESULTS

Agreement between data sources was very good. The diabetes definition including one or more hospitalizations or two or more outpatient claims over 2 years provided a sensitivity of 94.2%, specificity of 99.9%, PPV of 81.6% and NPV of 99.9%. The addition of one or more prescription claims to the same definition over 1 year provided similar results. Case ascertainment rates of both sources were very good to excellent and the ascertainment-corrected prevalence for youth-onset diabetes for the year 2006 was 2.4 per 1,000. It was not possible to distinguish between subtypes of diabetes within the administrative database; however, this limitation could be overcome with an anonymous linkage to the clinical registry.

CONCLUSIONS

Administrative data are a valid source for the determination of pediatric diabetes prevalence that can provide important information for health care planning and evaluation.

Diabetes prevalence continues to rise in all age-groups, including children and adolescents (1,2). In order to describe the burden of disease and evaluate strategies for disease prevention and treatment, reliable population-based data are required on an ongoing basis. Vital statistics files, disease specific registries, and surveys are all potentially useful and adaptable to this purpose (2). Health administrative data are an additional potential data source that is not as costly and time consuming to establish and maintain and is thus more feasible to study (3). Such data are also easily accessible on an ongoing basis and can thus provide cross-sectional and longitudinal information.

A case definition using administrative data has been validated for adult-onset diabetes (>20 years of age) and is used by Health Canada in the National Diabetes Surveillance System (NDSS), which is a network that gathers anonymous data for aggregate analysis from provincial surveillance systems across Canada. The NDSS diabetes case definition includes one hospitalization or two outpatient visits for diabetes over a 2-year period (4).

Because all data sources have potential limitations and the sensitivity and specificity of administrative databases have been reported to vary considerably according to the disease studied (5), validation of the data source for the given disease and population of interest is essential. Although a number of studies have evaluated the prevalence of youth-onset diabetes using administrative data (6–10) only one has attempted to systematically validate a pediatric specific case definition. Guttmann et al. (11) assessed the sensitivity and specificity of a number of definitions using hospital and outpatient claims over 1 or 2 years in youth aged <19 years in the province of Ontario, Canada. They found that the NDSS definition had a sensitivity of 100% and a specificity of 94.2% in this age-group. These results have not yet been confirmed with other datasets, nor have the contribution of drug use data and the determination of differences between type 1 and type 2 diabetes been assessed in youth (11).

RESEARCH DESIGN AND METHODS

Eighteen youth-onset (age 1–18 years) diabetes case definition algorithms were evaluated using combinations of physician, hospital and drug use data over 1, 2, and 3 years (Table 1). Validation of these algorithms was performed by comparing them to confirmed cases in a clinical registry. Case ascertainment rates were also determined and differences between type 1 and type 2 diabetes were evaluated. Finally, clinical differences between identified cases (true positives) and missed cases (false negatives) were evaluated.

Table 1.

Algorithms applied and validated to develop case definition for youth-onset diabetes using Manitoba administrative data

Years of data collection	Algorithm #	Hospital separation or	Physician claims or	DPIN records
1 (2006)	1	1 or more	1 or more	—
	2	1 or more	2 or more	—
	3	1 or more	1 or more	1 or more
	4	1 or more	2 or more	1 or more
	5	1 or more	1 or more	2 or more
	6	1 or more	2 or more	2 or more
2 (2005–2006)	7	1 or more	1 or more	—
	8	1 or more	2 or more	—
	9	1 or more	1 or more	1 or more
	10	1 or more	2 or more	1 or more
	11	1 or more	1 or more	2 or more
	12	1 or more	2 or more	2 or more
3 (2004–2006)	13	1 or more	1 or more	—
	14	1 or more	2 or more	—
	15	1 or more	1 or more	1 or more
	16	1 or more	2 or more	1 or more
	17	1 or more	1 or more	2 or more
	18	1 or more	2 or more	2 or more

Data sources

1) The Manitoba Health Services Insurance Plan (MHSIP) contains registration files, physician reimbursement claims, hospital discharge abstracts, and records of prescriptions dispensed. Canada has a single-payer health system, and nonparticipation is minimal because residents are not charged health care premiums. These data are stored for research purposes (University of Manitoba Research Ethics Board and Health Information Privacy Committee approval required) in de-identified form in the Population Health Research Data Repository (herein referred to as the Repository) housed at the Manitoba Centre for Health Policy in the University of Manitoba’s Faculty of Medicine. Physician reimbursement claims include International Classification of Diseases, 9th Revision, and Clinical Modification (ICD-9 CM) diagnostic codes at the 3-digit code level. Hospital records include ICD-9CM codes at the decimal level up until 1 April 2004 and ICD-10 Canadian version (ICD-10CA) codes thereafter. The ICD-9CM code 250 (diabetes mellitus) and ICD-10-CA codes E10 (insulin dependent diabetes) and E11 (noninsulin dependent diabetes) were evaluated in this study.

Data on all prescriptions dispensed by pharmacies are also available since 1995 in the pharmacy database, which is a subset of the Drug Programs Information Network (DPIN). The Anatomic Therapeutic Chemical (ATC) code A10 (drugs for diabetes) was used.

2) The Manitoba Diabetes Education Resource for Children and Adolescents (DER-CA) Registry is located in the only tertiary care pediatric referral center for Manitoba and is known to follow over 90% of children in the province with type 1 diabetes (10). The DER-CA also follows a large number of youth with type 2 diabetes. All youth followed in the DER-CA since 1986 have been prospectively entered into a clinical diabetes registry. The registry contains personal health identification numbers (PHIN) and validated diagnostic data that distinguish the subtypes of diabetes. The diagnosis of diabetes was made according to the criteria of the Canadian Diabetes Association (12), which are consistent with those of the American Diabetes Association (13). Using de-identified PHIN codes and through the creation of a cross-walk file to the Repository, youth can be linked between data sources at the person level. All youth with prevalent diabetes and valid Manitoba PHIN codes between 1 and 18 years of age from 1 January to 31 December 2006 were included in this study. Individuals that did not have coverage data available in the Repository for the full time period evaluated in each algorithm (i.e., 1, 2, or 3 years) were excluded from that particular analysis.

Validation methods and analysis

The κ statistic (κ) was used to measure the agreement between the two data sources for the presence or absence of diabetes. A κ of 0.80 was considered very good agreement. A sensitivity and specificity analysis of each proposed algorithm was conducted, using the DER-CA database as the “gold standard”. In addition, negative predictive value (NPV) and positive predictive value (PPV) were determined. The total midyear population of children 1–18 years of age in Manitoba in 2006 was 268,120, which was considered the total population at risk for diabetes. The two-source capture, recapture method was used to assess the ascertainment rate of the Repository and DER-CA for youth-onset diabetes in Manitoba as well as to determine the ascertainment-corrected prevalence of diabetes for 2006 (14).

Differences between type 1 and type 2 diabetes

Due to the fact that the ICD9-CM codes in the administrative physician visit data do not distinguish between the subtypes of diabetes, it was not possible to identify de novo the subtypes in the Repository using any of the proposed algorithms.

The clinical registry from the DER-CA contains validated diagnostic data which distinguish the subtypes of diabetes. It was used to determine the prevalence of each subtype of diabetes in Manitoba youth in 2006. Once individuals were identified as either having type 1 or type 2 diabetes, they were anonymously linked back to the Repository to determine what percentage of each subtype of diabetes was also present in the Repository data using each of the 18 algorithms. In addition, drug use (i.e., ATC code A10 from 2004–2006) was examined to determine if it would permit the differentiation of subtypes of youth-onset diabetes within the Repository.

Differences between identified cases and false negative cases

In order to identify factors potentially responsible for the cases missed by the NDSS definition (i.e., false negatives), difference in age, sex, urban versus rural residence, socioeconomic status and type of diabetes (type 2 vs. type 1) were compared between the true positives and the false negatives using t tests and χ² tests as appropriate.

RESULTS

Validation of diabetes algorithms

Table 2 contains the validation results for each youth-onset diabetes algorithm. Overall, the agreement between the two data sources was moderate to very good with κ values of 0.57–0.89. The sensitivity was high for all algorithms (88.9–98.5%). Not surprisingly, the sensitivity was higher if only one physician claim was required rather than two or more. However, these algorithms had much lower PPVs than those requiring two physician claims. The algorithm with the highest PPV was #2, which included at least one hospital claim or two physician claims over a 1-year period. The addition of DPIN data to the algorithm including 1 year of data (#4) improved the sensitivity from 88.9 to 94.2% as compared with algorithm #2 while maintaining a relatively good PPV of 82.4%. In contrast, the addition of DPIN data to the algorithms including 2 or 3 years of data did not significantly improve the sensitivity and decreased the PPV slightly. This suggests that if at least 2 years of data are included, outpatient claim data provide an adequate case ascertainment, not improved by the addition of drug use data. The specificity and NPV of all algorithms were between 99.7 and 99.9%.

Table 2.

Validation of pediatric diabetes algorithms compared with clinical DER-CA registry, ascertainment rates, and ascertainment-corrected prevalence of youth-onset diabetes

No. years	Alg. #	n (DER-CA)	n (MCHP)	Ascertainment rate of DER-CA based on case definition	Ascertainment rate of repository for youth-onset diabetes	Ascertainment-corrected prevalence of youth-onset diabetes (per 1,000 youth)	κ	MCHP case definition algorithm
								Sens	Spec	PPV	NPV
1	1	531	867	62.2%	95.4%	3.18	0.75	95.4	99.9	62.2	99.9
	2		579	86.1%	88.9%	2.30	0.87	88.9	99.9	86.1	99.9
	3		918	60.4%	96.8%	3.28	0.74	96.8	99.9	60.4	99.9
	4		662	82.4%	94.2%	2.40	0.89	94.2	99.9	82.4	99.9
	5		909	60.8%	96.8%	3.26	0.75	96.8	99.9	60.8	99.9
	6		651	83.1%	94.0%	2.38	0.88	94.0	99.9	83.1	99.9
2	7	531	1,048	48.9%	96.4%	4.07	0.65	96.4	99.8	48.9	99.9
	8		613	81.6%	94.2%	2.43	0.87	94.2	99.9	81.6	99.9
	9		1,096	47.4%	97.7%	4.19	0.64	97.7	99.8	47.4	99.9
	10		670	76.7%	96.8%	2.59	0.86	96.8	99.9	76.7	99.9
	11		1,087	47.7%	97.7%	4.15	0.64	97.7	99.8	47.7	99.9
	12		661	77.8%	96.8%	2.55	0.86	96.8	99.9	77.8	99.9
3	13	519	1,225	41.4%	97.9%	4.67	0.58	97.9	99.7	41.4	99.9
	14		650	77.4%	96.9%	2.50	0.86	96.9	99.9	77.4	99.9
	15		1,279	40.0%	98.5%	4.84	0.57	98.5	99.7	40.0	99.9
	16		708	71.8%	97.9%	2.70	0.83	97.9	99.9	71.8	99.9
	17		1,268	40.3%	98.4%	4.80	0.57	98.4	99.7	40.3	99.9
	18		697	72.9%	97.9%	2.66	0.84	97.9	99.9	72.9	99.9

Alg., algorithm; MCHP, Manitoba Centre for Health Policy; sens, sensitivity; spec, specificity.

The ascertainment rate of the Repository for youth-onset diabetes for all diabetes algorithms ranged from 88.9 to 98.5%. The ascertainment rate for the DER-CA had a wider range depending on the algorithm evaluated (41.4–86.1%).

Prevalence of youth-onset diabetes

The ascertainment-corrected prevalence of youth-onset diabetes in 2006 ranged between 2.30 and 4.84 per 1,000 children aged 1–18 years in Manitoba (Table 2).

Differences between type 1 and type 2 diabetes

There were a total of 531 prevalent youth with diabetes (420 with type 1 and 111 with type 2 diabetes) identified from the DER-CA registry that had full coverage from 1 January to 31 December 2006 in the Repository. The prevalence of type 1 diabetes was 1.57 per 1,000 and the prevalence of youth-onset type 2 diabetes was 0.41 per 1,000. These figures are probably underestimates (i.e., minimum prevalence rates).

The majority of the youth with type 1 diabetes were captured by all definitions (91.9–99.3%); however, the percentage of youth with type 2 diabetes captured was more varied (77.5–95.5%). The algorithms requiring two physician visits rather than one decreased the likelihood of a case being identified (77.5 vs. 92.8% over 1 year and 88.3 vs. 93.7% over 2 years), indicating that many youth with type 2 diabetes had only one billed physician encounter in that calendar year.

The drug data revealed that 96.8% of youth with type 1 diabetes (n = 396) and 63.6% of youth with type 2 diabetes (n = 70) were dispensed a medication for diabetes. The youth with type 1 diabetes were only dispensed insulin; however, 42.7% (n = 47) of the youth with type 2 diabetes were dispensed only insulin, 13.6% were dispensed only an oral hypoglycemic agent (n = 15), and 7.3% (n = 8) were dispensed both insulin and an oral hypoglycemic agent. The oral agents prescribed were metformin (60.0%), glyburide (36.3%), and other (3.8%).

Differences between identified cases (true positives) and false negatives

There was no difference between true positive and false negative cases with respect to age at diagnosis, sex, urban versus rural residence, or socioeconomic status (data not shown). There was, however, a statistically significant difference in the proportion of youth-onset type 2 diabetes (as a percentage of all diabetes) in the true positive group (98/500 = 19.6%) vs. the false negative group (13/31 = 41.9%); P = 0.003.

CONCLUSIONS

This study supports the use of population-based administrative data to evaluate diabetes prevalence in youth 1–18 years of age in single-payer health systems, such as the one that exists in Canada. Definitions that included 3 years of data were the most sensitive (97.9–98.5%), but had the worst PPV (40.0–77.4%), as more false positives were identified with the longer time period. The examination of 2 years of data rather than 1 year increased the sensitivity of the definition (94.2 vs. 88.9%), while sacrificing only slightly the PPV (81.6 vs. 86.1%). Algorithms that required at least two outpatient physician visits increased the PPV markedly over those that only required one, as more “true” cases were being detected with the more restrictive definitions. The addition of one or two prescription claims for a diabetes drug improved the sensitivity slightly (94.2 and 96.8 vs. 88.9%) for 1 year of data. Specificity and NPV were excellent (99.7–99.9%) for all definitions.

On balance, the overall best definitions for youth-onset diabetes were as follows:

1. One year of data (#4): Any one of: one or more hospitalizations or two or more physician claims or one or more prescription claims (sensitivity 94.2%, specificity 99.9%, PPV 82.4%, and NPV 99.9%).

2. Two years of data (#8): Any one of: one or more hospitalizations or two or more physician claims (sensitivity 94.2%, specificity 99.9%, PPV 81.6%, and NPV 99.9%).

The second definition is identical to the one used by the NDSS (4) and thus supports its use in children. Guttmann et al. (11) also recently evaluated the validity of different algorithms for pediatric diabetes using hospital chart audits in Ontario as the second data source. The NDSS definition in their study was found to have a sensitivity of 100% and a PPV of 97.6%; however, the specificity was only 94.2%. They also compared physician claim–based algorithms with and without hospital claims data and found that the addition of hospital data increased the number of false positives identified, and thus decreased the specificity of the diabetes definition. The authors hypothesized that this inaccuracy was caused by secondary causes of diabetes being coded as diabetes in the inpatient setting. This finding is difficult to interpret in the context of hospital chart review as the validation source. The most specific definition in their study was four outpatient visits over a 2-year period. However, two claims in a 2-year period had a higher sensitivity with only a marginal difference in specificity (98.6 vs. 98.9%).

A summary of the pediatric diabetes literature that has used administrative data to evaluate diabetes prevalence is provided in Table 3. Blanchard et al. (10) used Manitoba administrative data between 1985 and 1993 to evaluate 0- to 14-year-olds specifically with type 1 diabetes. The case definition used was five or more physician claims or a minimum of three physician claims if registered with Manitoba Health for less than 2 years (10). Individuals with treaty status were excluded from that analysis as a proxy in order to evaluate only children with type 1 diabetes (10). Treaty status refers to individuals of Aboriginal descent included in the Indian Registry, established by the Indian Act in 1876 (15). As the majority of youth with type 2 diabetes in Manitoba are Aboriginal, the availability of this information could allow the differentiation between youth with type 1 and type 2 diabetes in future studies. The ethnicity of youth with type 2 diabetes varies from population to population, and therefore this approach would not necessarily be generalizable.

Table 3.

Pediatric studies evaluating the prevalence of diabetes using administrative data

Author	Country	Data source/years	Code(s)	Age (years)	Algorithm (years of data evaluated)	Validation source
Guttmann 2010 (11)	Canada (Ontario)	Institute for Clinical Evaluative Sciences	ICD-9CM: 250.X	<19	1+ hospital or 2+ outpatient visits (2 years) vs.	Hospital charts
		1994–2005	ICD-10: 10–14		1–5 outpatient visits ± hospital claim (1 or 2 years)
Dabelea 2009 (18)	United States (Navajo Youth)	Indian Health Service Facilities 2001–2005	ICD-9CM 250.0–250.9	<20	1+ outpatient visit or hospitalization (3 years)	Medical records
Hsia 2009 (6)	United Kingdom	Prescription records 1998–2005	ATC code A10	0–18	1+ prescription claim	“General practice research database”
Cox 2008 (7)	United States (Missouri)	Prescription claims: Scripts Inc.	Antidiabetic drug	5–19	1+ prescription claim	None
		2002–2005
Kemper 2006 (17)	United States	MarketScan: commercial claims and encounter database (Inpatient, outpatient + drug data)	T1DM: 250.X1/X3	≤18	T1DM: 1+ inpatient or outpatient code ± insulin	Drug claim data
		1998–2002	T2DM: 250.X0/X2		T2DM: 1+ inpatient or outpatient code ± no insulin
Acton 2002 (9)	United States	Indian Health Service Facilities	ICD-9CM 250.0–250.9	<35	1+ outpatient code	None
		1990–1998
Blanchard 1997 (10)	Canada (Manitoba)	Outpatient visits 1985–1993	ICD-9CM 250	0–14	5 outpatient visits or 3–4 if <2 years coverage	Diabetes Education Registry

T1DM, type 1 diabetes mellitus; T2DM, type 2 diabetes mellitus.

Rhodes et al. (16) used a Boston Massachusetts database to evaluate inpatient or outpatient five-digit ICD-9CM codes in youth and young adults with diabetes. The PPV was 97% for type 1 diabetes but only 16% for type 2 diabetes, likely due to the fact that unspecified diabetes has the same five digit code. Kemper et al. (17) evaluated encounter and pharmacy claims in privately insured children in the U.S. over 1 year. They found that 10% of children with type 1 diabetes identified with claims data alone were not on insulin, suggesting that claims data alone may have a poor PPV. These results are in keeping with our finding that drug use data improved the PPV when using only 1 year of data.

Two studies used the Indian Health Service Facility Database in the U.S. (9,18). The study by Dabelea et al. (18) used a definition of one or more inpatient or outpatient records over 3 years to identify cases of diabetes, but reported that only 50% of cases identified by the Indian Health Service were confirmed by chart audit, suggesting that the PPV of the definition was very low. These results are supported by this study, which showed that algorithms with two or more outpatient records, and using only 1 or 2 years of data are more valid than those including only one outpatient record and/or 3 years of data. The study by Acton et al. (9) used only one outpatient code to identify cases with diabetes. The higher diabetes prevalence they identified therefore likely is also an overestimate (9). Two studies have evaluated prescription claim data alone in the U.K. (6). and U.S. (7). to evaluate the prevalence of diabetes. Neither study evaluated their methodology compared with other administrative algorithms. Further study is required to assess the validity of drug claim data alone.

Administrative databases have been criticized for being incomplete or inaccurate (19). There has been concern that health care encounters may not be billed for, and thus do not appear in the data. Alternatively, even though care may be billed for, the correct diagnosis may not be recorded (5). In addition, diagnoses listed may be a reflection of testing performed rather than diagnosis made. Despite these concerns, databases are extensively used in research. Database research offers the possibility of feasibly following individuals with chronic diseases such as diabetes cross-sectionally and longitudinally at a much reduced cost and improved feasibility as compared with other means (20). These concerns, however, highlight the importance of maximizing the performance of claims-based algorithms by assessing their validity, such as has been done in this study.

Ascertainment and diabetes prevalence

The ascertainment of the Repository for the diagnosis of youth-onset diabetes was very good for all evaluated algorithms (88.9–98.5%). This suggests that few cases are likely to be missed if this administrative database is used to evaluate youth-onset diabetes prevalence.

The two most favorable algorithms provided youth-onset diabetes prevalence estimates of 2.40 and 2.43 per 1,000 youth. This estimate is comparable to the crude prevalence of 1.82 per 1,000 in the U.S. evaluated by a recent active surveillance initiative (21) and the Canadian prevalence of 3 per 1,000 youth 1– 19 years of age reported by the NDSS (4).

Differences between type 1 and type 2 diabetes

The main limitation of administrative data in the evaluation of diabetes is the inability to identify the subtypes of diabetes with three-digit ICD-9CM codes. This is less of an issue in adult populations where type 2 diabetes accounts for the vast majority of cases. In the past, type 1 diabetes could be assumed on the basis of pediatric age; however, with the increasing prevalence of type 2 diabetes in children, this is no longer the case (22).

In addition, even if five-digit ICD-9CM coding is available, the PPV for the diagnosis of type 2 diabetes may be as low as 16% (16). The current study evaluated drug use in both subtypes of diabetes as a potential means to differentiate type 1 from type 2 diabetes. However, lifestyle management is the mainstay of treatment in type 2 diabetes and is not captured in the drug use database. In addition, insulin remains the only approved pharmacologic agent for the treatment of diabetes in children in Canada and is the predominant form of recommended pharmacologic therapy for both type 1 and type 2 diabetes in youth (12,23). Despite these recommendations, however, clinical practice is quite variable. A recent national surveillance study in Canada revealed that 33% of individuals with type 2 diabetes were treated with lifestyle alone, 27% with lifestyle and insulin, 22% with lifestyle and an oral hypoglycemic agent, and 16% were treated with lifestyle, an oral hypoglycemic agent, and insulin (24). Thus, pharmaceutical use cannot be used as a reliable means to distinguish one subtype of diabetes from the other. Possible mechanisms to overcome this limitation include linkages to clinical registries such as the DER-CA, which contain diagnostic data differentiating the subtypes of diabetes or medical record reviews or audits.

Of particular concern in this study is the finding that youth with type 2 diabetes are over-represented in the false negative group. This may be explained by encounters being coded with an alternate diagnosis or missed as the result of lack of billing in an alternatively funded arrangement or a nurse-only visit in a local health center, or it may suggest a possible service inequity that should be addressed.

CONCLUSIONS

This study supports the use of administrative data in a single-payer health care system to determine the overall prevalence of diabetes in youth at a population level. This data source can feasibly provide cross-sectional and longitudinal data which are important in planning for health care use as well as evaluating disease prevention and treatment strategies. In order to differentiate type 1 from type 2 diabetes, clinical correlation is required.