Abstract
Background
Stroke registries contribute to the conduct of clinical research and to the assessment of health care quality control. Efforts to compare clinical outcomes and quality indicators between centers are dependent on standardized data elements, but it is unknown how stroke physicians define common data elements, such as hypertension or diabetes, when collecting data for registries at their centers.
Methods
We conducted an internet-based survey of 91 centers affiliated with a university to assess their definitions of common data elements (CDEs) and compared their responses with standardized definitions, including those from the American College of Cardiology (ACC).
Results
More than half (52%) of centers completed the survey. There was only modest agreement among respondents regarding definitions of CDEs in the survey and even less agreement on how the respondents’ definitions compared to ACC standards.
Conclusions
Surveyed respondents do not agree on the definitions of CDEs, making comparisons between centers problematic. Standardized definitions of CDEs are needed to improve data collection for patient care and clinical research.
Key Words: Clinical data standards, Common data elements, Consensus guidelines, Health care quality control, Standardized definitions, Stroke registry
Introduction
Stroke registries are prospectively maintained collections of patient data that are used for clinical research and health care quality, federal- and state-mandated programs for hospital certification, and NIH-sponsored research networks. In research focused on cerebrovascular disease, registries soon will be used for verification of adherence to Centers for Medicare/Medicaid pay-for-performance indicators. Historically, registries in stroke have provided important data on the prevalence and severity of vascular risk factors in the stroke population, approaches to acute stroke treatment, complications, and outcomes. Observed associations among variables in stroke registries have proven useful in the design of quality improvement projects and prospective clinical trials.
In 2001, the American College of Cardiology's (ACC) Task Force on Clinical Data Standards published ‘Key Data Elements and Definitions for Measuring the Clinical Management and Outcomes of Patients with Acute Coronary Syndromes’ [1, 2]. These definitions were proposed to provide standardization for all centers involved in the collection of clinical data related to cardiovascular disease and allow for effective pooling of information from different centers. Unfortunately, no equivalent published guidelines exist in the literature on cerebrovascular disease. The NINDS is in the midst of proposing a library of CDEs or ‘common data elements’ (http://www.commondataelements.ninds.nih.gov/) for use in stroke studies; however, while these elements include factors such as hypertension and diabetes, the NINDS standards currently do not offer definitions for the elements but rather only specify classification for these factors as ‘yes’, ‘no’ or ‘unknown’ [3, 4]. Since cardiovascular and cerebrovascular diseases share risk factors, the ACC standards could provide specific definitions for use in stroke studies as well. These ACC standards can be used to assess the validity and reliability of registry data from stroke centers.
If data elements and definitions for research in cerebrovascular disease are agreed upon and standardized, registry level data have the potential to promote research by allowing for comparison and pooled analyses of both clinical trial data and multi-site observational data. Registries often provide important data and observations upon which clinical trials are designed and clinical trials often provide the basis for clinical care decisions. Furthermore, without registries, it may not be possible to readily understand the prevalence of specific characteristics and co-morbid diseases among an otherwise captive cohort of patients. Finally, with the impending quality performance measurement initiatives such as pay-for-performance, providers would benefit from registry data that are supported by a common taxonomy, enhancing both validity and reliability.
The purpose of this study was to compare the current use of definitions for data elements among US stroke centers that have stroke registries, and compare the use of existing stroke registry definitions with those already established by the ACC. We hypothesized that vascular neurologists using registry data were not making use of existing definitions published in the cardiology and internal medicine literature.
Materials and Methods
Study Design, Participants, and Administration
From September to December 2009, we conducted a cross-sectional, internet-based survey of academic vascular neurologists in the United States at university-affiliated hospitals using a modified Dillman's tailored design method [5, 6]. The survey was designed and conducted at www.surveymonkey.com [7] and used our previously published methods for obtaining a complete list of US academic vascular neurologists [8]. Vascular neurologists were targeted, as they are typically the primary investigators of registry-based clinical research, and as such are responsible for supervising the research process, assuring study personnel are appropriately trained, and that they adhere to the study protocol. After an exhaustive effort to identify all stroke centers affiliated with a university in the United States, surveys were distributed to a total of 144 vascular neurologists. Seven of the 144 individuals had e-mail addresses that were nonfunctional and the correct address could not be found. This yielded a total sample number of 137 initial e-mails that were successfully sent to the target population at 91 universities.
After β-testing by three of the authors (K.C.A., S.M.-S., and S.I.S.) and two additional volunteers, the survey was formally released. Participants received a pre-survey cover letter explaining the objectives of the study and its importance. A request to participate in the study, with detailed instructions, followed which provided a link to the electronic survey. Non-responders were sent four reminders. No incentives were provided. This study was coordinated by the Vascular Neurology Program at the University of Texas School of Medicine in Houston.
Survey Content
The survey consisted of 3 general questions and 10 multi-part definition questions. All survey questions as they were presented to respondents are shown in the online supplementary material (www.karger.com/doi/10.1159/000334146). Respondents were asked to indicate (1) if their institution had an active stroke registry, (2) who abstracted data at their site, and (3) if their site made use of a registry codebook or data dictionary. The survey focused on 10 variables (i.e., data elements) commonly collected in stroke registries, many of which are specifically defined in the ACC standards: hypertension, diabetes, hyperlipidemia/dyslipidemia, atrial fibrillation, tobacco use, myocardial infarction, coronary artery disease, congestive heart failure, carotid disease, and peripheral vascular disease [1, 2, 9, 10, 11]. After an exhaustive search of available guidelines failed to reveal a formal definition of ‘carotid disease’, potential definitions were generated. Participants were provided choices and instructed to indicate if their center currently uses each provided choice as part of their definition for each variable (yes or no). Respondents were then instructed to indicate if they found each of the provided choices ‘acceptable’ as part of their definition for each variable (yes or no). Risk factor data elements included (but did not explicitly identify) criteria listed in definitions previously published by the ACC and American Diabetes Association (ADA) [1, 2, 11, 12].
Data Collection and Analysis
All data was entered electronically by the participants. Analysis was performed using SPSS version 15.0 (SPSS Inc., Chicago, Ill., USA). The anticipated response rate was estimated at 50% (95% CI 37–63%, adjusted Wald method) based on prior studies [12, 13, 14].
Results
Sample
Participants completed 55 self-administered surveys, representing 47 universities. Vascular neurologists reported that registry data were abstracted by trained abstractors (29%), medical assistants (4%), nurses (29%), physician assistants (2%), fellows (2%), attending physicians (6%), or a combination of medical professionals (29%). The response rate for individuals was 40.1% (55/137) compared to 51.6% (47/91) for universities. There was no significant difference in response rates based on Census Bureau regions (Northeast 25.5%, Midwest 29.1%, South 27.3%, and West 18.2%, p = 0.700).
Data Elements
The majority of respondents (80%, 44/55) reported using a stroke registry; however, only 46% of respondents using a registry reported using a data dictionary to standardize definitions within their center. Table 1 illustrates reported current use of examined definition criteria for 10 data elements. Patient or family report of having a data element was the most commonly used definition criterion, ranging from 55.8% for hyperlipidemia to 74.4% for hypertension and carotid endarterectomy as a surrogate for carotid disease. Current use of prior documentation of a condition ranged from 53.5% for tobacco use within the last year to 72.1% for hypertension. Current use of definition criteria related to patient medication ranged from 7.0% for peripheral vascular disease to 65.1% for diabetes. The use of laboratory findings to define data elements ranged from 11.6% in ‘high-density lipoprotein <40 mg/dl’ to 44.2% in ‘glycosylated hemoglobin (HbA1c)’ for diabetes. Use of data elements derived from prior diagnostic studies ranged from 11.6% for ‘left ventricular hypertrophy on admission electrocardiogram (ECG)’ for hypertension to 76.7% for demonstration of atrial fibrillation on admission ECG. Also depicted in table 1 is the reported acceptability for each data element. There was poor concordance between current use and acceptability of most of the definition choices, especially for the many choices offered for the data elements hypertension, diabetes, and carotid disease. For the most commonly used definition criteria of each of the 10 data elements, at least 20% fewer respondents found definition criteria to be acceptable.
Table 1.
Current use and acceptability of definition criteria for 10 stroke registry data elements
| Current use (n = 43) |
Also acceptable (n = 46) |
|
|---|---|---|
| % (n) | % (n) | |
| Hypertension (HTN) | ||
|
74.4 (32) | 47.8 (22) |
|
72.1 (31) | 47.8 (22) |
|
11.6 (5) | 43.5 (20) |
|
14.0 (6) | 37.0 (17) |
|
55.8 (24) | 30.4 (14) |
|
2.3 (1) | 4.3 (2) |
|
25.6 (11) | 4.3 (2) |
| Diabetes mellitus (DM) | ||
|
69.8 (30) | 41.3 (19) |
|
72.1 (31) | 41.3 (19) |
|
34.9 (15) | 47.8 (22) |
|
27.9 (12) | 47.8 (22) |
|
27.9 (12) | 56.5 (26) |
|
44.2 (19) | 37.0 (17) |
|
65.1 (28) | 43.5 (20) |
|
|
2.2 (1) |
|
23.3 (10) | 4.3 (2) |
| Hyperlipidemia (HLP) or dyslipidemia | ||
|
55.8 (24) | 34.8 (16) |
|
58.1 (25) | 34.8 (16) |
|
65.1 (28) | 37.0 (17) |
|
55.8 (24) | 32.6 (15) |
|
25.6 (11) | 32.6 (15) |
|
34.9 (15) | 47.8 (22) |
|
30.2 (13) | 30.4 (14) |
|
11.6 (5) | 28.3 (13) |
|
51.2 (22) | 37.0 (17) |
|
2.3 (1) | 2.2 (1) |
|
23.3 (10) | 4.3 (2) |
| Atrial fibrillation (AF) | ||
|
62.8 (27) | 39.1 (18) |
|
74.4 (32) | 39.1 (18) |
|
76.7 (33) | 39.1 (18) |
|
39.5 (17) | 30.4 (14) |
|
25.6 (11) | 23.9 (11) |
|
62.8 (27) | 43.5 (20) |
|
39.5 (17) | 32.6 (15) |
|
2.3 (1) | 2.2 (1) |
|
|
2.2 (1) |
|
16.3 (7) | 2.2 (1) |
| Tobacco use | ||
|
72.1 (31) | 41.3 (19) |
|
67.4 (29) | 37.0 (17) |
|
53.5 (23) | 37.0 (17) |
|
|
2.2 (1) |
|
16.3 (7) | 2.2 (1) |
| Myocardial infarction (MI) | ||
|
62.8 (27) | 34.8 (16) |
|
67.4 (29) | 34.8 (16) |
|
25.6 (11) | 23.9 (11) |
|
20.9 (9) | 21.7 (10) |
|
27.9 (12) | 32.6 (15) |
|
9.3 (4) | 10.9 (5) |
|
27.9 (12) | 28.3 (13) |
|
25.6 (11) | 32.6 (15) |
|
|
2.2 (1) |
|
23.3 (10) | 2.2 (1) |
| Coronary artery disease (CAD) | ||
|
69.8 (30) | 39.1 (18) |
|
67.4 (29) | 37.0 (17) |
|
18.6 (8) | 17.4 (8) |
|
11.6 (5) | 17.4 (8) |
|
23.3 (10) | 15.2 (7) |
|
27.9 (12) | 19.6 (9) |
|
|
2.2 (1) |
|
23.3 (10) | 4.3 (2) |
| Congestive heart failure (CHF) | ||
|
58.1 (25) | 30.4 (14) |
|
62.8 (27) | 34.8 (16) |
|
25.6 (11) | 21.7 (10) |
|
30.2 (13) | 13.0 (6) |
|
20.9 (9) | 19.6 (9) |
|
25.6 (11) | 28.3 (13) |
|
23.3 (10) | 23.9 (11) |
|
11.6 (5) | 15.2 (7) |
|
2.3 (1) | 6.5 (3) |
|
2.3 (1) | 4.3 (2) |
|
23.3 (10) | 4.3 (2) |
| Carotid disease | ||
|
58.1 (25) | 23.8 (13) |
|
72.1 (31) | 32.6 (15) |
|
74.4 (32) | 39.1 (18) |
|
72.1 (31) | 34.8 (16) |
|
72.1 (31) | 37.0 (17) |
|
74.4 (32) | 37.0 (17) |
|
60.5 (26) | 34.8 (16) |
|
32.6 (14) | 30.4 (14) |
|
46.5 (20) | 37.0 (17) |
|
|
2.2 (1) |
|
18.6 (8) | 2.2 (1) |
| Peripheral vascular disease (PVD) | ||
|
58.1 (25) | 28.3 (13) |
|
58.1 (25) | 32.6 (15) |
|
41.9 (18) | 34.8 (16) |
|
48.8 (21) | 37.0 (17) |
|
48.8 (21) | 39.1 (18) |
|
16.3 (7) | 21.7 (10) |
|
37.2 (16) | 43.5 (20) |
|
20.9 (9) | 26.1 (12) |
|
|
13.0 (6) |
|
39.5 (17) | 37.0 (17) |
|
|
2.2 (1) |
|
32.6 (14) | 2.2 (1) |
Discussion
The majority of centers surveyed are actively collecting stroke registry data on key elements such as hypertension or diabetes. However, few centers appear to have formal guidelines in place (e.g., codebook/data dictionary) to assist them with the within-site standardization of definitions for these variables. No more than three fourths of respondents were using or found any provided definition criteria to be acceptable for any of the data elements. For each data element, far more respondents were using a stated history of the data element to define its presence, than those that actually found a stated history to be an acceptable criterion. Researchers using and combining registry data may not agree on the definitions or even know the definitions used in the acquisition of the data they house in their registries.
The results of this survey suggest there is little consensus among US centers regarding the actual definition of CDEs in cardiovascular and cerebrovascular diseases. The prevalence of these data elements and association of these data elements with other patient characteristics or outcomes will be substantially affected by the definitions applied, making the interpretation of results from these registries problematic. There are various possible reasons for the lack of consensus. There may be a lack of agreement about the purpose of the registry, i.e., what question or questions it will answer, or a belief that a registry can be created that will be able to answer a wide variety of questions without compromising quality. Without a prospective and limited list of questions to answer, one can understand why it is hard to agree on how data fields are defined.
In addition, our study found that vascular neurologists are not currently using published ACC or ADA definition criteria for data elements. It is possible that stroke neurologists are not familiar with the ACC guidelines; however, our survey found that they often do not agree with the data element definitions proposed by the ACC.
The evolving acceptance of definitions by specialty organizations (e.g., ACC or ADA) is one barrier to the adoption of definitions for stroke registries. For example, in 2010 the ADA approved the use of HbA1c measurement of ≥6.5% to meet the criteria for diabetes mellitus. If stroke registries uniformly incorporate this criterion, the prevalence of diabetes mellitus may increase from rates previously demonstrated. If the criterion is not uniformly incorporated, data regarding diabetes mellitus cannot be compared nor combined from center to center. As other methods of diagnosing stroke risk factors are proposed and validated, centers using registries need to have a plan for how to adapt and incorporate new criteria. Similar to the standardized definitions that support evidence-based process measures endorsed by the National Quality Forum for adoption by the Centers for Medicare/Medicaid Services, definitions for medical diagnoses or the presence/absence of symptoms or disease characteristics are essential. Without this standardization, the validity of findings from the best process measures is ultimately threatened due to measurement error.
Our study has a number of limitations. First, the generalizability of this study is limited by its 40.1% response rate; however, our response rate is in keeping with a review of 31 e-mail surveys that reported a mean response rate of 36.8% and recent stroke surveys assessing opinions of university-affiliated stroke neurologists [8, 12, 13, 14]. While self-administered web-based surveys have been shown to result in lower response rates and more missing data, they do have the advantage of diminishing social desirability bias and response order effects. Secondly, despite our efforts to β-test survey questions prior to disseminating the survey, we cannot be certain that the responses regarding currently used and acceptable definitions of CDEs accurately represent the opinions of other neurologists within the same stroke center. For example, respondents at centers that use a prespecified protocol with standardized definitions to perform retrospective chart abstraction may have indicated that they did not currently use a registry codebook or a data dictionary, despite using standardized definitions. Further, we cannot disregard the potential influence of close-ended, leading, or loaded questions on respondents, despite our efforts to avoid double-barrel questions by separating the ‘current use’ and ‘also acceptable’ answer choices. It is possible that our target population did not represent all hospitals collecting registry level data, as many hospitals not affiliated with universities may also collect data to support their programs of research as well as stroke center certification. Lastly, we wish to point out that the results of this study should not be misconstrued to equate registries with well-designed epidemiological studies and controlled clinical trials.
To the best of our knowledge, this is the first study to provide quantitative evidence supporting the NINDS proposal for creating a library of CDEs. The implications of our findings are practical, as existing inconsistencies in definitions may substantially impair our ability to detect associations between variables, limit the ability to generalize the observational data of a single center, and confound multi-site collaboration. Given the concerted efforts by academic institutions to collect data for clinical research, standardization of data elements is of paramount importance. This study supports the argument that consensus guidelines need to be developed to outline and define key data elements, to optimize and standardize data collection for the purpose of clinical research.
Disclosure Statement
The authors have no conflicts of interest to report.
Supplementary Material
Acknowledgements
The authors would like to thank Virginia Howard, PhD, for her invaluable assistance in reviewing and editing the manuscript, and Sharyl Martini, MD, PhD, for her support and assistance. We would also like to acknowledge Nakia Lee, BA, for her administrative support. This study was supported by P50 NS044227 and R21HD060978.
References
- 1.Cannon CP, Battler A, Brindis RG, Cox JL, Ellis SG, Every NR, Flaherty JT, Harrington RA, Krumholz HM, Simoons ML, Van De Werf FJJ, Weintraub WS, Mitchell KR, Morrisson SL, Brindis RG, Anderson HV, Cannom DS, Chitwood WR, Jr, Cigarroa JE, Collins-Nakai RL, Ellis SG, Gibbons RJ, Grover FL, Heidenreich PA, Khandheria BK, Knoebel SB, Krumholz HL, Malenka DJ, Mark DB, McKay CR, Passamani ER, Radford MJ, Riner RN, Schwartz JB, Shaw RE, Shemin RJ, Van Fossen DB, Verrier ED, Watkins MW, Phoubandith DR, Furnelli T. American College of Cardiology key data elements and definitions for measuring the clinical management and outcomes of patients with acute coronary syndromes. A report of the American College of Cardiology Task Force on Clinical Data Standards (Acute Coronary Syndromes Writing Committee) endorsed by the American Association of Cardiovascular and Pulmonary Rehabilitation, American College of Emergency Physicians, American Heart Association, Cardiac Society of Australia & New Zealand, National Heart Foundation of Australia, Society for Cardiac Angiography and Interventions, and the Taiwan Society of Cardiology. J Am Coll Cardiol. 2001;38:2114–2130. doi: 10.1016/s0735-1097(01)01702-8. [DOI] [PubMed] [Google Scholar]
- 2.Radford MJ, Heidenreich PA, Bailey SR, Goff DC, Grover FL, Havranek EP, Kuntz RE, Malenka DJ, Peterson ED, Redberg RF, Roger VL. ACC/AHA 2007 methodology for the development of clinical data standards: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Data Standards. J Am Coll Cardiol. 2007;49:830–837. doi: 10.1016/j.jacc.2007.01.002. [DOI] [PubMed] [Google Scholar]
- 3.NINDS: NINDS common data elements, 2010 (http://www.commondataelements.ninds.nih.gov/).
- 4.Stone K. NINDS common data element project: a long-awaited breakthrough in streamlining trials. Ann Neurol. 2010;68:A11–A13. doi: 10.1002/ana.22114. [DOI] [PubMed] [Google Scholar]
- 5.Dillman D, editor. Mail and Internet Surveys: The Tailored Design Method. New York: Wiley; 2000. [Google Scholar]
- 6.Dillman D, editor. Mail and Internet Surveys: The Tailored Design Method − 2007 Update with New Internet, Visual, and Mixed-Mode Guide. Hoboken: Wiley; 2007. [Google Scholar]
- 7.SurveyMonkey, 2010 (www.surveymonkey.com).
- 8.Savitz SI, Benatar M, Saver JL, Fisher M. Outcome analysis in clinical trial design for acute stroke: physicians' attitudes and choices. Cerebrovasc Dis. 2008;26:156–162. doi: 10.1159/000139663. [DOI] [PubMed] [Google Scholar]
- 9.McNamara RL, Brass LM, Drozda JP, Jr, Go AS, Halperin JL, Kerr CR, Levy S, Malenka DJ, Mittal S, Pelosi F, Jr, Rosenberg Y, Stryer D, Wyse DG, Radford MJ, Goff DC, Jr, Grover FL, Heidenreich PA, Peterson ED, Redberg RF, American College of Cardiology, American Heart Association ACC/AHA key data elements and definitions for measuring the clinical management and outcomes of patients with atrial fibrillation: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Data Standards (Writing Commitee to Develop Data Standards on Atrial Fibrillation) J Am Coll Cardiol. 2004;44:475–495. doi: 10.1016/j.jacc.2004.06.041. [DOI] [PubMed] [Google Scholar]
- 10.Fuster V, Ryden LE, Asinger RW, Cannom DS, Crijns HJ, Frye RL, Halperin JL, Kay GN, Klein WW, Levy S, McNamara RL, Prystowsky EN, Wann LS, Wyse DG, Gibbons RJ, Antman EM, Alpert JS, Faxon DP, Gregoratos G, Hiratzka LF, Jacobs AK, Russell RO, Smith SC, Klein WW, Alonso-Garcia A, Blomstrom-Lundqvist C, de Backer G, Flather M, Hradec J, Oto A, Parkhomenko A, Silber S, Torbicki A, American College of Cardiology/American Heart Association/European Society of Cardiology Board ACC/AHA/ESC guidelines for the management of patients with atrial fibrillation: executive summary. A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines and the European Society of Cardiology Committee for Practice Guidelines and Policy Conferences (Committee to Develop Guidelines for the Management of Patients with Atrial Fibrillation): developed in collaboration with the North American Society of Pacing and Electrophysiology. J Am Coll Cardiol. 2001;38:1231–1266. doi: 10.1016/s0735-1097(01)01587-x. [DOI] [PubMed] [Google Scholar]
- 11.American DA. Standards of medical care in diabetes − 2010. Diabetes Care. 2010;33(suppl 1):S11–S61. doi: 10.2337/dc10-S011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Moskowitz A, Chan YF, Bruns J, Levine SR. Emergency physician and stroke specialist beliefs and expectations regarding telestroke. Stroke. 2010;41:805–809. doi: 10.1161/STROKEAHA.109.574137. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.El Khoury R, Fisher M, Savitz SI. Current practice versus willingness to enroll in clinical trials: paradox among vascular neurologists about treatment for acute ischemic stroke. Stroke. 2010;41:2038–2043. doi: 10.1161/STROKEAHA.110.586511. [DOI] [PubMed] [Google Scholar]
- 14.Sheehan KB. E-mail survey response rates: a review. J Comput Mediat Commun. 2001;6:0. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.