Systematic review of scope and quality of electronic patient record data in primary care

Abstract

Objective To systematically review measures of data quality in electronic patient records (EPRs) in primary care.

Design Systematic review of English language publications, 1980-2001.

Data sources Bibliographic searches of medical databases, specialist medical informatics databases, conference proceedings, and institutional contacts.

Study selection Studies selected according to a predefined framework for categorising review papers.

Data extraction Reference standards and measurements used to judge quality.

Results Bibliographic searches identified 4589 publications. After primary exclusions 174 articles were classified, 52 of which met the inclusion criteria for review. Selected studies were primarily descriptive surveys. Variability in methods prevented meta-analysis of results. Forty eight publications were concerned with diagnostic data, 37 studies measured data quality, and 15 scoped EPR quality. Reliability of data was assessed with rate comparison. Measures of sensitivity were highly dependent on the element of EPR data being investigated, while the positive predictive value was consistently high, indicating good validity. Prescribing data were generally of better quality than diagnostic or lifestyle data.

Conclusion The lack of standardised methods for assessment of quality of data in electronic patient records makes it difficult to compare results between studies. Studies should present data quality measures with clear numerators, denominators, and confidence intervals. Ambiguous terms such as “accuracy” should be avoided unless precisely defined.

Introduction

The NHS is becoming increasingly accountable for the services it provides. One element of that accountability is clinical governance, which, in turn, depends crucially on the availability of high quality clinical information. This relies on the data collected. ¹ A clear message emerging from government policy initiatives is the need for high quality data on health accessible through electronic patient record (EPR) systems. In this context, such systems will inevitably replace their paper based predecessors. They represent a fundamental change in how health professionals approach the management of clinical information. As the service acclimatises to new technology, the need for assessment of quality and improvement of primary care datasets has been repeatedly emphasised. ² However, the criteria against which quality should be judged remain unclear.

We identified one review of mainly secondary care studies that described system and organisational factors that affect quality of the data in EPR. ³ We carried out a similar review but in primary care.

Methods

We searched all major bibliographic databases and several specialist datasets during the last quarter of 2001 (see bmj.com for databases and sources and web table A for search criteria). Under our primary exclusion criteria we excluded duplicate publications, editorials, letters, poster presentations, and coding studies and publications based on EPRs set in health maintenance organisations, administration, and single variable databases (such as, prescribing, disease registers). We searched for citations of papers that used a reference standard for assessment of quality. When relevance was ambiguous (for example, if we were unable to deduce whether the study involved EPR or paper records) we checked the abstract and MeSH headings through PubMed. When ambiguity remained we obtained the full paper and made a collective decision.

We established a framework for categorising and selecting review papers; defined the reference standards and measurements used to judge quality; and examined the quality of EPRs in primary care (box). We extracted data on study design, countries involved, number of sites, measurement criteria, description of reference standard, research topic, main results, name of EPR, and data structure. Eligible papers had to satisfy at least one aspect (numbered) of each category (A-C) within the box.

Results

We identified 4589 abstracts and categorised 174 documents after primary exclusions. Of these, we included 47 journal publications, four reports, ^{4– 7} and one thesis ⁸ from 1980-2001. Thirty seven studies measured data quality, and 15 used electronic patient records and commented on quality in the presence of a reference standard (scoping). These were analysed separately ( table 1). Forty eight studies assessed diagnostic data, 20 assessed management information, and 13 examined wider aspects of routine data.

Table 1.

Proportions of data type being investigated, reference standards used to assess quality, and commonest measures of quality. Figures are numbers (percentage) of studies

	Measuring data quality (n=37)	Scoping data quality (n=15)
Data type:
Diagnostic or symptom state data	34 (92)	14 (93)
Management data	16 (43)	4 (27)
Wider aspect of care	10 (43)	3 (20)
Reference standard
Video, direct examination:	1 (2)	1 (7)
Surveys	7 (19)	3 (20)
Routine data	24 (65)	12 (80)
National statistics, rates	18 (49)	8 (53)
Quality measures used:
Rate comparison	27 (73)	10 (67)
Sensitivity	20 (54)	7 (47)
Positive predictive value	7 (19)	8 (53)
Specificity	3 (8)	1 (7)

Measuring data quality

Thirty one publications were from the United Kingdom. A similar proportion had been published since 1995. Table 2 shows characteristics of the 37 studies that measured quality of data. Table B on bmj.com gives full details of categorisation (according to that shown in the box) and characteristics. Eight studies were prospective, in which a network of practices was established from which to extract data. Although these studies were prospective, the data extraction was primarily cross sectional. The remaining articles were cross sectional or retrospective surveys. Two studies were interventional: one a case-control study involving onsite training and the other a before and after software update study. Both showed substantial improvements in recording levels after the intervention. ⁹,¹⁰ A retrospective cohort study of data conscious practices that took advantage of generic national services also showed an increase in completeness and accuracy of EPRs over five years. ¹¹

Table 2.

Characteristics of 37 studies that measured data quality

	No of publications
Type of study:
Prospective	8
Cross sectional	6
Retrospective	23
Countries involved:
UK	26
Netherlands	3
UK/USA	3
UK/Canada	1
UK/Netherlands	1
Iceland	2
Nordic collaboration *	1
No of practices:
Single	3
2-10	11
11-100	13
200-750	4
Undefined	6
Data type:
Read codes	17
International classification of diseases (ICD)	6
Oxford medical information systems (OXMIS)	5
International classification of primary care (ICPC)	3
International Classification of Health Problems in Primary Care (ICHPPC)	1
Terms and textual	2
Undefined	12

^*Study included data from five Nordic countries including Iceland.

Framework for assessing eligibility of publications for review

All three categories (A-C) needed to be satisfied for a paper to be selected

A Reference standard

A modification of the “distance from patient” concept, which classified the reference standard used to judge quality ³

1. Studies that used objective “close to patient” standards by using techniques such as video recording or direct examination

2. Studies that used interviews or questionnaire surveys of patient, next of kin, or their immediate carers as reference standard

3. Studies that used routine consultation data (databases, EPRs, paper records, discharge letter, etc) as standard reference

4. Studies that used national statistics or equivalent survey results as their reference standard

B Study objectives

1. Studies that measured change in EPR data quality or those that measured EPR data quality were classified as measuring data quality

2. Studies that used EPRs and commented on their quality were classified as scoping data quality

C Data types

Publications that investigate:

1. Diagnostic or symptom state of the patient

2. Patient management data—for example, health promotion, drug treatment, referrals, tests

3. Wider aspects of patient and practice management—for example, family history, ethnicity, socioeconomic status, immunisation, hospital episodes, consultation rates

Structured data (codes, classifications, and nomenclatures) were most commonly investigated. Although textual data were mentioned, they rarely received detailed attention. ¹² Only one study considered textual data in any detail. ¹³ Twelve documents did not present their data structure (that is, coding system name) while most did not present the precise codes being investigated ( table 2). UK publications generally used Read and OXMIS (Oxford medical information systems) codes. In other countries the ICPC (international classification of primary care) codes were more widely used. ICD (international classification of diseases) codes act as a referencing standard for these primary care coding systems. When there were deficits in descriptive ability of a coding strategy, subsidiary codes (for example, chapter headings from British National Formulary; Prescription Pricing Authority) were used to enhance the data. ⁶,^{15– 18}

Quality of data (reliability) was usually measured with rate comparisons. Data validity was expressed under a range of terms (completeness, correctness, accuracy, consistency, and appropriateness), which were rarely defined. Sensitivity (completeness) was the commonest such index ( table 1, webextra table B on bmj.com). One study used video recording of the consultation to evaluate the EPR content compared with the use of notes and UK national statistics (fourth national study of morbidity in general practice, MSGP4) for comparative measures. ¹⁸ Seven studies carried out questionnaire and telephone surveying for a reference standard with data gathered from the patient, carer, or both. ¹³,¹⁴,¹⁹,²⁰,^{22– 24} These studies involved the sampling of a study population from the database for subsequent validation through questionnaires. The reference standard varied from “life time experience of morbidity” to more structured investigation of diagnostic status through validated questionnaires. ²⁰,²² Triangulation with multiple sources (prescription data, clinician diagnosis in EPR, or notes) was used for further validation. ²³

Twenty four studies used clinical information gathered during the consultation as a reference standard ( table 1). Seventeen publications used triangulation within the EPR to test internal consistency of data. Fifteen studies were conducted after 1994. Twelve relied on medication data as the internal reference standard. Sixteen used paper based information as the reference standards. Often EPR diagnostic status was appraised through electronic prescribing information and subsequently validated against the paper notes. Hospital discharge details have also been used to evaluate EPR diagnostic status through practitioner responses, discharge summaries, and consultants' letters. ¹²,¹⁴,¹⁷,¹⁹Time of diagnosis and referral data were also evaluated under this reference standard. Dissonance between data from secondary and primary care has been documented, though the presence of hospital diagnosis and procedural data have been found to improve the quality of data in primary care. ¹²,¹⁷,¹⁹,²⁰,²⁵ Eighteen studies used national statistics or survey data as a reference standard for data reliability. ^{4– 7},¹¹,¹³,²¹,²³,^{25– 34} A third of UK studies used MSGP4 as a reference standard for rate comparisons.

Scoping data

We identified 15 studies that used EPR data for research or practice management. Although the intention of these studies was not to measure data quality, they gave insight into issues of data validation. These studies relied more on measures of positive predictive value than on measures of sensitivity ( table 1) to meet their needs. Fourteen studies considered the diagnostic status of the patient, with 10 publications dealing primarily with information on patient identification and case validation. ^{35– 44} Three used survey techniques to establish diagnostic status. ⁴⁰,⁴¹,⁴⁵ Of the 12 retrospective investigations, seven used centralised datasets. ^{36– 38},⁴⁰,⁴¹,⁴⁴,⁴⁶ These “scoping” studies were more than twice as likely to present confidence intervals than studies that measured data quality (10/15 (67%) ν 11/37 (30%)).

Levels of data recording

Table 3 shows that prescribing data are generally the most sensitive. The ability to link prescriptions with diagnosis was the favoured means of identifying patients and establishing the predictive validity of diagnostic codes. The sensitivity of other EPR elements was wide ranging, while positive predictive value was consistently high. Those diseases with clear diagnostic criteria were generally better recorded, as were data on specific procedures. ²⁰ Lifestyle and socioeconomic data were rarely studied and then only in terms of sensitivity. Results indicated lower recording levels than for diagnosis and medication. ¹¹,¹⁸,⁴⁷

Table 3.

Selected sensitivities and positive predictive values of data from electronic patient records

Data	Sensitivity (%)	Positive predictive value (%)
Prescribing
Nazareth et al 15	100	100
Pringle et al 18	100	100
Scobie et al 47	93	—
Jick et al 12	95	—
Martin 31	95	—
Njalsson and Sigurdsson 51	96-58	—
Referral
Jick et al 12	93	—
Consultation
Nazareth et al 15	74	—
Neal et al 50	95-99	—
Scobie et al 47	91	—
Coronary heart disease
Hassey et al 29	95	97
Van Staa and Abenhaim 19	93	100
Angina
Whitelaw et al 20	40	100
Myocardial infarction
Whitelaw et al 20	100	100
Van Staa and Abenhaim 19	89	85
Diabetes
Whitelaw et al 23	100	100
Van Staa and Abenhaim 19	67-96 *	—
Whitelaw et al 23	74	—
Van Staa and Abenhaim 19	91	99
Hassey et al 29	98	99
Schizophrenia
Nazareth et al 15	88, 71 †	71, 81 †
Depression
Van Staa and Abenhaim 19	100	100
Whitelaw et al 20	89	100
Life style—smoking
Pringle et al 18	52	—
Scobie et al 47	61	—
Van Staa and Abenhaim 19	95	—
Life style—alcohol
Pringle et al 18	37	—
Scobie et al 47	62	—
Van Staa and Abenhaim 19	97	—
Socioeconomic data
Scobie et al 47	26	—
Summary of all conditions considered
Hassey et al 29	87	96
Whitelaw et al 20	75	100
Pringle et al 18	55	—
Jick et al 12	87	—
Jick et al 17	96	—

^*Dependent on time of diagnosis.

^†Different definitions of diagnostic status.

Discussion

We believe this is the first systematic review to investigate the measurement of quality of data in primary care. Most of the research has been published since 1995, reflecting the increasing importance and use of EPRs. The categorisation provided a framework for selecting and describing the most important publications. This showed that patient identification and diagnostic data were the focus of most studies of data quality. Mostly studies were descriptive surveys. This would seem ideal for an environment where external forces set the direction of change (for example, pace of technological development). The scarcity of interventional studies reflects the passivity and inability of researchers to control their study environment. The appraisal of data quality has favoured the selection of practices that embrace technology and was the likely reason for purposive sampling in many studies. Consequently, the EPR quality reported in the literature is likely to be an overestimate of the general picture.

The dominance of UK publications is unsurprising given the scope of this review. However, it also suggests that UK researchers understand the importance of the quality of EPR data in terms of health policy and validated research databases such as the general practice research database (GPRD), the doctor independent network (DIN) database, and the medicines monitoring unit (MEMO). Centrally maintained and quality assured, these databases act as a rich source of data for epidemiological research. Their size and success with pharmacological data are the prime attraction to overseas collaborators.

Publications from non-English speaking countries were disadvantaged under our selection criteria. Those that were identified used similar techniques to measure data quality. Like early UK publications, non-UK studies emphasise where, how, and by whom data are collected and the reliability of the process. This focus is now not present in UK primary care, where clinicians directly collect data during the consultation.

Measuring Quality

The element of the EPR being investigated (numerator) and the components of the reference standard used to appraise its quality (denominator) were often not clearly defined within the literature (for instance, diagnostic code/diagnostic criteria). When they were defined there was inconsistency between studies. This makes comparisons risky and meta-analytical interpretation of results impossible. It may be a reflection of the immaturity of the discipline that more standardised approaches have not yet evolved.

Measurement theory requires that both the concepts of validity and reliability be addressed. Reliability (a precursor to validity) is a measure of stability and is appraised through the subjective comparison of rates and prevalence. Many studies used old statistics (for example, MSGP4) or variations between practices to make judgments on the reliability of “live” data. Such methods cannot measure validity of the EPR in reflecting the “truth.” Sensitivity and positive predictive value, the most widespread measures of data validity, presuppose that the selected denominator is an adequate representation of this truth. Surveys and questionnaires can be of dubious accuracy. Reference standards that emanate from the patient and carers present different but important perspectives on morbidity or concordance with treatment. What is the real health status of the patient? The answer exists in subjective (perceived), objective, and diagnostic dimensions. Each needs to be measured by different techniques and its appropriateness for EPR validation considered. To aid interpretation of the resulting proportions and to facilitate comparisons between populations confidence intervals should be provided.

What is already known on this topic

The demonstration of quality is central to the NHS strategic agenda

Data from electronic records are expected to have a central role within healthcare commissioning, quality control, clinical governance, and the new GP contract

No standard methods of measuring data quality have been described

What this study adds

A framework for categorising and selecting papers which report data quality in primary care

Reliability of data was measured through rate comparison in 73% of studies, while validity was calculated mostly through measures of sensitivity

Markers of quality should comprise internal reference standards based on objective and diagnostic EPR elements that have high positive predictive value

When the opportunity to record clinical data in different forms (paper and computer) exists, this inevitably decreases validity of any one to act as a true reference standard. The use of paper notes to assess EPR validity will become increasingly inaccurate as clinicians migrate to electronic systems. In the medium term it is best to consider several independent markers of quality, and those studies that used several explicit reference standards (triangulation) were more likely to reflect the true quality of electronic data (see table B on bmj.com).

To facilitate comparisons of data quality across sites and systems, it is essential to have a reference standard. User friendly “point of service” technologies have ensured that electronic prescription data has rapidly become accepted as sensitive and highly predictive when used appropriately for diagnosis validation. Similarly, record linkage and automated population of EPRs with investigations and test results will offer alternative objective markers against which to test the internal consistency of EPRs. The sensitivity of these markers for a reference standard may be varied but their predictive abilities are likely to be high. In the longer term we recommend the establishment of internal reference standards based on those objective and diagnostic EPR elements recognised as having high positive predictive value (that is, diagnostic codes, prescriptions, test results, referral outcomes, procedural codes). Such reference standards can then be used to explore measures of sensitivity.