Patients' own assessments of quality of primary care compared with objective records based measures of technical quality of care: cross sectional study

Abstract

Objective To investigate the relation between older patients' assessments of the quality of their primary care and measures of good clinical practice on the basis of data from administrative and clinical records.

Design Cross sectional population based study using the general practice assessment survey.

Setting 18 general practices in the Basildon primary care trust area, south east England.

Participants 3487 people aged 65 or more.

Main outcome measures Correlations between mean practice scores on the general practice assessment survey and three evidence based measures on survey of case records (monitoring for, and control of, hypertension, and vaccination against influenza).

Results 76% of people (3487/4563) responded to the general practice assessment survey. Correlations between patient assessed survey scores for technical quality and the objective records based measures of good clinical practice were 0.22 (95% confidence interval -0.28 to 0.62) for hypertension monitored, 0.30 (-0.19 to 0.67) for hypertension controlled, and -0.05 (-0.50 to 0.43) for influenza vaccination.

Conclusions Older patients' assessments are not a sufficient basis for assessing the technical quality of their primary care. For an overall assessment both patient based and records based measures are required.

Introduction

The complex, multidimensional nature of the quality of health care has long been recognised.^1-3 Since the outcomes movement of the 1980s much effort has gone into developing rigorous methods of assessment. Meanwhile the growth of the consumer movement has led to a view of patients as being potentially the best judges of the quality of their care.⁴,⁵ This is implicit in the introduction of the agenda about choice in the UK NHS.⁶,⁷

Research in the United States suggests that patient reports can be used to identify health plans that offer care of higher clinical quality.⁸ The general practice assessment survey is a patient questionnaire developed in the United States and adapted for use in the United Kingdom.⁹,¹⁰ It has been tested in a variety of settings. More recently, the general practice assessment survey has been modified to the general practice assessment questionnaire (www.gpaq.info/). We used the general practice assessment survey to test whether older patients' assessments of the technical quality of their care in general practice were related to evidence based good clinical practice as indicated by data from medical records.

Participants and methods

We invited 23 general practices in Basildon to participate in the study. Sample size calculations suggested that 19 practices would allow for the detection of a correlation coefficient of 0.60 with 80% power at the 5% significance level, and 17 practices, a value of 0.63 or more.¹¹ (In the general practice assessment survey validation study correlations between survey assessments of technical care and four of the other domains were 0.63 or more.¹⁰) The sampling frame consisted of all patients of participating practices aged 65 or more, registered on 1 September 2000. We excluded patients identified by their doctor as too ill to participate.

Patient assessment of quality

The general practice assessment survey covers nine domains of patient assessed quality, including quality of care provided by practice nursing or reception staff, satisfaction with practice premises, and the technical quality of care. Each domain includes several items, which are weighted to provide an overall index. The technical quality domain includes items on medical knowledge, thoroughness of physical examination, arrangement of tests when needed, making the right diagnosis, and prescribing the right treatment. The general practice assessment survey also includes personal information and indicators of socioeconomic status, including car ownership and housing tenure. We used a postal version of the general practice assessment survey. The questionnaires were bar coded and anonymised. Previous studies suggested response rates of around 66% in the general population.¹⁰ Questionnaires were sent to 300 randomly selected people aged 65 years or more registered with each practice. For practices with fewer than 300 such people, we selected all registered people. After two weeks we sent non-respondents another copy of the questionnaire.

Record based measures of technical quality

We chose three indicators of the technical quality of clinical care using the following criteria: clinical guidelines were available on the care that should be provided to older people, based on evidence of benefit; adherence to guidelines could be determined from patient records; and the conditions concerned were sufficiently common for differences to be detectable between practices in adherence to guidelines.¹²

Two indicators were based on adherence to the British Hypertension Society guidelines.¹³ We extracted medical records with sampling fractions dependent on estimated numbers of patients aged 65-79 with hypertension: 50% for practices estimated to have more than 400 such patients; 75% for practices with 250-399; and 100% for all other practices. The sample size for this part of the study was 5473. We developed an algorithm for data extraction. Two trained research nurses established whether blood pressure had been measured within the past five years (hypertension monitored) and whether hypertension was controlled to the standards of the British Hypertension Society (hypertension controlled).

Both nurses audited one practice as part of their training, and interobserver agreement was estimated. One hundred per cent matches were found for 160 of 192 records audited (κ = 0.89). All except one disagreement resulted from illegible handwriting. As a result one nurse ascertained hypertension measures for seven practices where the research nurses were confident about the doctors' (mainly computerised) record keeping. Both nurses undertook assessment for the 11 practices where records were mainly hand written. When the diagnosis of hypertension was uncertain, cases were discussed and agreement was reached.

The third indicator of technical quality of care was coverage of influenza vaccination. At the time of the survey the guideline recommended influenza vaccination for all patients aged 75 years or more.¹⁴ The research nurses extracted data on the vaccination status of patients aged 75 years or more (4961 people) in each of the practices using case notes, computer records, or practice vaccination registers. They recorded vaccination status on a structured assessment form.

Data analysis

We used prescribed scoring methods to estimate mean general practice assessment survey scores for each domain in each practice.⁹,¹⁰ We treated data from each practice as if sampled from infinite populations. Analysis of variance was used to assess variation in domain scores between and within practices.

Earlier studies suggested that general practice assessment survey scores were related to age and social class but not sex.¹⁵ We constructed four socioeconomic groups from two questionnaire variables as follows: access to car, owns or is buying home; access to car, renting home; no access to car, owns or is buying home; no access to car, renting home. This ordinal scale indicator has been found to correlate with reported incomes of elderly people in data from the 1998 health survey for England.¹⁶ For each domain we derived a regression equation, with, as independent variables, five age groups, four socioeconomic groups, and sex. We used these equations to produce practice scores adjusted for age, socioeconomic status, and sex.

We estimated the proportions of participants in the relevant age groups in each practice who had had their blood pressure monitored and influenza vaccination. For those with a diagnosis of hypertension, we also estimated the proportions whose blood pressure was controlled.

We produced two matrices of correlation coefficients between general practice assessment survey domain scores and records based measures of technical quality. One was unweighted. For the other we used weights inversely proportional to the variance of the estimated practice mean to account for the varying survey sample sizes. We used the method described by Bland to estimate confidence intervals for the coefficients.¹¹ Data manipulation was carried out using Excel, and statistical analysis was done using Stata 8.1.

Results

Nineteen of 23 practices agreed to participate. These varied in size, from eight single handed practices to two with five doctors. The numbers of people aged 65 or more in the practices ranged from 155 to 1695. The smallest practice had only 39 such patients and was excluded from further analysis.

The demographic structure of the practices was similar to that of the rest of England (table 1). In total, 3487 out of a possible 4563 patients in the 18 included practices responded to the general practice assessment survey (76% response rate). Response rates were lower in those aged 85 years or more (60% for those more than 85; 78% for those less than 85; χ² =1, df = 62.2, P < 0.0001) but were not related to sex. Response rates by practice varied from 67% to 82% (χ² = 17, df = 41.1, P = 0.001).

Table 1.

Respondents to general practice assessment survey questionnaire by age and sex compared with general population in England

Variable	No aged ≥65	% according to age group					% of population
		65-69	70-74	75-79	80-84	≥85
Men:
England	3 337 3000	31.4	26.8	20.5	13.2	8.1	42.3
Basildon practices	5029	34.2	28.7	18.5	12.9	5.7	41.1
General practice assessment survey sample	1897	34.4	28.1	18.8	13.2	5.4	41.6
Respondents	1469	34.5	28.1	19.3	13.6	4.7	42.1
Women:
England	4 543 4000	24.8	23.3	20.6	16.2	15.1	57.7
Basildon practices	7196	28.0	24.7	19.2	16.5	11.6	58.9
General practice assessment survey sample	2666	27.5	25.5	19.7	16.4	10.9	58.4
Respondents	2018	28.6	27.1	19.8	16.1	8.4	57.9

Respondents were of lower socioeconomic status than the population aged 65 or more in the health survey for England.¹⁶ The proportion of respondents who owned or were buying their home and had access to a car was 37.7% compared with 47.8% in the health survey for England, and the proportion who lived in rented or part rented accommodation with no access to a car was 28.5% compared with 22.1% in the health survey for England. Data on satisfaction with practice nursing were missing for 35% of respondents (table 2). Several patients commented that they had not had sufficient contact with the nurse to be able to judge, and one practice had no nurse.

Table 2.

General practice assessment survey results for each domain

General practice assessment survey item	% of respondents with missing data (%)	Mean score overall*	Estimated SD between practices†	Estimated SD within practices	Range of practice mean scores*
Access to practice	0.2	64.0	7.2	15.4	47.5-77.0
Satisfaction with receptionists	1.0	76.9	8.1	19.2	61.3-90.8
Satisfaction with continuity of care	7.3	70.9	5.6	20.3	64.2-82.0
Satisfaction with communication	4.1	71.0	5.9	19.7	51.6-79.1
Satisfaction with interpersonal care	4.5	67.3	5.9	20.8	48.5-75.6
Trust in general practitioner	3.3	73.6	4.6	17.4	58.8-79.8
General practitioner's knowledge	6.6	61.4	5.8	21.8	41.8-70.5
Satisfaction with practice nursing	34.6	76.0	2.7	17.3	70.6-81.4
Satisfaction with technical care	5.5	74.0	5.9	19.2	54.5-82.6

^*Possible range 0-100.

^†In analyses of variance, P values for F tests of significance of variation between practices were <0.0001 for all items.

Case notes were retrieved for 97.5% (n = 5336) of the 5473 patients eligible for hypertension monitoring. The number per practice ranged from 90 to 602. Overall, 4332 (81.2%) of patients had had their blood pressure measured within the past five years, with practice rates ranging from 51% to 95%. In total, 2166 people in the 18 practices had a diagnosis of hypertension (number per practice ranged from 24 to 232), and 1473 (68.0%) of these had a systolic blood pressure of less than 160 mm Hg (values per practice ranged from 39% to 79%). Overall, 94.4% (n = 4683) of the 4961 records sampled for investigation of influenza vaccination rates were retrieved (number per practice ranged from 66 to 753); 2282 (46.0%) had been vaccinated (proportions per practice ranged from 16% to 63%).

Adjustment of mean general practice assessment survey scores for age, sex, and socioeconomic status made little difference. The largest change in score for any of the 162 practice and domain combinations was less than 1%, with 97% of the changes less than 0.5% and 61% less than 0.2%. Unadjusted scores were therefore used in the analyses presented here. Table 2 shows mean general practice assessment survey scores, standard deviations, and ranges for each of the domains across practices. With a possible range of scores for each domain of 0 to100, overall mean scores ranged from 61.4 for doctors' knowledge to 76.9 for satisfaction with receptionists. Variation between practices for most domains was broadly similar, except for satisfaction with nursing, which was relatively low.

The figure gives scatter plots for general practice assessment survey scores for technical care against each of the records based indicators. Each circle represents a practice. One practice's technical quality score seems to be an outlier.

Figure 1.

Scatter plots for general practice assessment survey scores for technical quality compared with three records based measures. Size of marker indicates confidence interval around survey score estimate

Table 3 gives weighted correlation coefficients with 95% confidence intervals for 17 practices, excluding this outlier. Strong and significant correlations were found between general practice assessment survey technical quality and three other general practice assessment survey domain scores: communication skills of doctors (0.85, 95% confidence interval 0.64 to 0.94), interpersonal skills of doctors (0.88, 0.69 to 0.95), and trust in doctors (0.87, 0.69 to 0.95). The correlations between technical quality and the records based measures were, however, much weaker and not significant (technical care and hypertension monitored 0.22, -0.28 to 0.62; technical care and hypertension controlled 0.30, -0.19 to 0.67; and technical care and influenza vaccination -0.05, -0.50 to 0.43). When the outlier was included, coefficients changed little (0.22, 0.30, -0.05, 0.52, -0.08, and 0.27 became 0.36, 0.23, -0.32, 0.51, -0.19, and 0.23, respectively) and none became significant.

Influenza vaccination rate was negatively related to continuity of care and access: -0.44, -0.75 to 0.03 and -0.46 -0.76 to 0.01, respectively. Some evidence was found for a correlation between monitoring and control of hypertension (0.52, 0.07 to 0.79; P = 0.028). However, the correlations between influenza vaccination rates and the measures of hypertension care were weak: hypertension monitored -0.08 (-0.53 to 0.40) and hypertension controlled 0.27 (-0.22 to 0.66).

Discussion

In this survey, older patients in primary care did not distinguish between technical quality of care and other aspects of doctor quality. We found evidence of variation between practices in all our measures of good clinical practice, but weak correlations between the technical care measure on the general practice assessment survey and three records based measures (hypertension monitored, hypertension controlled, influenza vaccination), and also between the records based measures themselves. Strong correlations were, however, shown between patients' own assessments of technical care using the general practice assessment survey and their assessments of other characteristics of their doctors, such as interpersonal skills, trust, and communication.

Table 3.

Pearson's correlation coefficients (95% confidence intervals) for each domain of general practice assessment survey, with general practice assessment survey “technical care” and three records based measures, unadjusted, but weighted by number sampled in each practice

	General practice assessment survey measures									Records based measures
	Access	Receptionists	Continuity of care	Communication	Interpersonal skills	Trust	Knowledge of patient	Practice nursing	Technical care	Hypertension monitored	Hypertension controlled
General practice assessment survey measures:
Technical care	0.33 (−0.16 to 0.69)	0.33 (−0.16 to 0.69)	0.33 (−0.17 to 0.69)	0.85 (0.64 to 0.94)	0.88 (0.69 to 0.95)	0.87 (0.69 to 0.95)	0.84 (0.62 to 0.94)	0.32 (−0.17 to 0.68)	1	—	—
Records based measures:
Hypertension monitored	0.16 (−0.33 to 0.58)	0.30 (−0.19 to 0.67)	−0.03 (−0.49 to 0.44)	−0.01 (−0.47 to 0.46)	0.15 (−0.34 to 0.58)	0.05 (−0.42 to 0.51)	−0.04 (−0.50 to 0.43)	0.08 (−0.40 to 0.52)	0.22 (−0.28 to 0.62)	1	—
Hypertension controlled	−0.09 (−0.53 to 0.40)	−0.11 (−0.55 to 0.38)	−0.38 (−0.72 to 0.11)	0.29 (−0.21 to 0.66)	0.37 (−0.11 to 0.72)	0.29 (−0.21 to 0.66)	0.28 (−0.22 to 0.66)	0.36 (−0.13 to 0.71)	0.30 (−0.19 to 0.67)	0.52 (0.07 to 0.79)	1
Influenza vaccination	−0.46 (−0.76 to 0.01)	−0.35 (−0.70 to 0.14)	−0.44 (−0.75 to 0.03)	0.03 (−0.44 to 0.49)	0.05 (−0.43 to 0.50)	0.08 (−0.40 to 0.53)	−0.10 (−0.54 to 0.39	−0.02 (−0.48 to 0.45)	−0.05 (−0.50 to 0.43)	−0.08 (−0.53 to 0.40)	0.27 (−0.22 to 0.66)

The general practice assessment survey has been extensively tested for validity and reliability.⁹,¹⁰ Response rates in our study were good except in the very oldest age group (85 years or more), and general practice assessment survey scores are close to (within 5%) those found previously in the United Kingdom for patients from the full range of ages.¹⁰ Our three records based measures were based on explicit criteria, and we believe that they were reliably ascertained. Basildon is a medium sized town in south east England with fairly typical demography, and although the population has below average home ownership and access to a car, there are few pockets of serious deprivation. Recruitment of practices to the study was good. Therefore our findings are likely to be generalisable to other parts of the United Kingdom and potentially more widely.

One possible explanation for our results is that patients are right; technical quality of care offered by doctors in primary care may be closely related to communication skills, interpersonal skills, and trustworthiness and it is our records based indicators that are misleading. If so, patient assessments would be sufficient. A parallel qualitative study of patients who had been asked to complete the general practice assessment survey, however, found that “relatively few patients had enough knowledge about their own particular illnesses or about possible alternative treatments to make informed judgements.”¹⁷ Also, in a study of 120 practices in England and Scotland,¹⁸ a practice performance index based on 16 subindicators from routine NHS data covering prescribing, prevention, and external quality was not significantly correlated with a questionnaire based patient enablement index. Alazri and Neal¹⁹ found a significant correlation of -0.27 between individual overall general practice assessment survey scores and HbA_1c levels in 106 patients (mean age 68) with type 2 diabetes. However, the correlation with the technical quality element of the score was, at -0.126, one of the lowest. These results are hard to reconcile with the proposition that patient questionnaires are all that is necessary for an assessment of the quality of clinical practice.

It seems more likely that when patients are asked about technical quality they base their judgments on those aspects of care that they feel better able to judge. These may be only weakly related to recognised standards of good clinical practice. The question then is how to construct a valid instrument for assessing clinical quality. We used only three measures. It is reassuring that the two related to hypertension were correlated with each other but a matter of concern that they were not correlated to influenza vaccination rates. An approach that covered more aspects of care would be more valid in principle, but a more comprehensive approach might possibly involve more controversial components and problems with collecting and interpreting data.

In an era of strong advocacy for quality of health care driven by choice and “assessment and accountability,”²⁰ valid and reliable indicators are needed. Our findings suggest that in general, older patients do not distinguish between technical quality and other aspects of doctor quality and that the domain of technical quality of care itself may have many subdimensions. We conclude that the evidence for relying solely on patient assessed measures of quality of primary care for older people is weak and that a broader range of measurements is needed.²¹ The challenge for schemes such as the UK Quality and Outcomes Framework²² will be to find measures that fulfil our criteria: based on carefully evaluated evidence, clear practice guidelines, and events or characteristics that are well defined, frequent enough, and recorded sufficiently accurately for values to be reliably estimated and compared at a practice level.

What is already known on this topic

Different dimensions need to be taken into consideration when measuring quality of care in general practice

Patients can appropriately assess some aspects of quality of health care

What this study adds

Some aspects of technical quality of primary care can be measured using practice records; these measures indicate substantial variations between practices

Patients' own assessments of technical quality are not closely related to independently ascertained records based measures of technical quality

Assessment of the technical quality of primary care should not rely on patient based assessments alone