Comparison of patients' assessments of the quality of stroke care with audit findings

Abstract

Objective

To determine the extent of correlation between stroke patients' experiences of hospital care with the quality of services assessed in a national audit.

Methods

Patients' assessments of their care derived from survey data were linked to data obtained in the National Sentinel Stroke Audit 2004 for 670 patients in 51 English NHS trusts. A measure of patients' experience of hospital stroke care was derived by summing responses to 31 survey items and grouping these into three broad concept domains: quality of care; information; and relationships with staff. Audit data were extracted from hospital admissions data and management information to assess the organisation of services, and obtained retrospectively from patient records to evaluate the delivery of care. Patient survey responses were compared with audit measures of organisation of care and compliance with clinical process standards.

Results

Patient experience scores were positively correlated with clinicians' assessment of the organisational quality of stroke care, but were largely unrelated to clinical process standards. Responses to individual questions regarding communication about diagnosis revealed a discrepancy between clinicians' and patients' reports.

Conclusions

Better organised stroke care is associated with more positive patient experiences. Examining areas of disparity between patients' and clinicians' reports is important for understanding the complex nature of healthcare and for identifying areas for quality improvement. Future evaluations of the quality of stroke services should include a validated patient experience survey in addition to audit of clinical records.

Stroke is one of the main causes of death and disability in England, and is a major source of expenditure for the National Health Service (NHS) and the wider economy.¹ Development of stroke services and improvement in the quality of care is increasingly recognised as a priority. The National Service Framework (NSF) for Older People set out standards to ensure that those who have had a stroke have prompt access to integrated stroke care services.² The National Clinical Guidelines for Stroke, prepared by the Intercollegiate Stroke Working Party (ICWP), provide a detailed evidence‐based framework for delivering the best care for stroke patients in hospital and in the community.³ The quality of hospital stroke services has been evaluated on a 2‐year cycle since 1998 using clinical audit methods developed by the ICWP.⁴ Results of the national audits have enabled trusts to assess the quality of their stroke services and monitor progress against standards set out in the NSF and the National Clinical Guidelines for Stroke. Performance is assessed based on the provision of stroke services featuring at least four of five key features identified by the Stroke Unit Trialists' Collaboration as defining a specialist stroke unit⁵,⁶:

• Having staff with a specialist interest in stroke or rehabilitation

• Involving carers in the rehabilitation process

• Multidisciplinary team meetings taking place at least weekly

• Provision of information to patients and carers

• Regular education and training for staff.

Besides evaluating the quality of care through clinical audit, there is a growing interest in assessing patients' experiences of care. Patient satisfaction surveys have traditionally been the main approach used to obtain their views of healthcare. However, these surveys have been criticised because the concept of satisfaction is ill‐defined and multidimensional,⁷ and because such surveys typically elicit high undifferentiated levels of satisfaction.⁷,⁸,⁹ Measuring patients' experiences of care by asking them to provide reports of “what happened” during a specific episode of care is considered to be a more valid approach. Responses to such “report” style questions are intended to be factual rather than evaluative and provide results that are more useful for quality improvement.¹⁰,¹¹ As part of the Healthcare Commission's national patient survey programme, a survey was conducted in England to evaluate stroke patients' experiences of care in hospital and in the period immediately after discharge.¹²

There is a growing body of research on patients' experiences of stroke care,¹³,¹⁴,¹⁵,¹⁶,¹⁷,¹⁸ yet few studies have directly compared clinical audit with patients' own assessments of the quality of stroke care. Research undertaken in the USA showed that the process of stroke care—measured by compliance with stroke guidelines—was positively and highly associated with greater patient satisfaction.¹⁹ To assess hospital‐based stroke rehabilitation, Tyson and Turner²⁰ combined a process audit with surveys of patient and staff opinions, arguing that by comprehensively examining many aspects of the service together, inadequacies in the quality of care could be identified more readily. Similarly, Durieux and colleagues²¹ compared patients' and healthcare professionals' views of standards of care and concluded that both patients' and healthcare professionals' evaluations must be considered if high quality care is to be achieved.

Methods

The 2004 national sentinel stroke audit had two parts: the first audited the organisation of care and the second audited the care of up to 40 patients per site consecutively admitted to hospital between April and June 2004 based on reports from clinical records. Reliability analyses, where the first five cases per site were audited by two different auditors, gave good levels of agreement across all the audit data items, with κ values above 0.60 dominating.⁴ Further details of audit forms and the methods can be found at the Royal College of Physicians website (www.rcplondon.ac.uk/pubs/books/strokeaudit/).

The questionnaire used in the patient survey was based on published qualitative studies of stroke patients' and carers' experiences of care, the NSF for older people and the National Clinical Guidelines for Stroke. Content validity and sampling issues were considered by an advisory group of experts in stroke. Cognitive interviews with 29 stroke patients supported the face validity of the questionnaire, which was further tested in a pilot study involving 187 patients in three trusts. Details about the development of the questionnaire are available online (http://www.nhssurveys.org/docs/Stroke_Development_Report.pdf).

All NHS acute trusts in England participated in the audit. Approximately a third (n = 51) of the acute trusts were randomly selected for participation in the patient survey. We asked trusts to sample patients admitted between April and June 2004 with a primary diagnosis of stroke. Between October and December 2004, the surviving sample patients were sent a questionnaire and up to two reminder letters. Linkage between the survey data and the audit data was undertaken by matching on the basis of NHS trust, gender, year of birth, and dates of admission and discharge.

Data analysis

We calculated an overall patient experience score for each respondent on the basis of their responses to 31 questions. Individual responses reflecting quality of care on each question were scored in equal increments on a scale of 0 to 100, with higher scores reflecting better experience. The overall score was a simple average of all applicable responses. We chose to derive a simple mean score in this way in order to recognise the fact that many different elements of care could potentially affect the quality of patients' experiences overall. Although differences in the overall quality of organisation within trusts might manifest in different ways, we considered this approach to remain sensitive to such disparities by broadly including all measured aspects of the patient experience. In addition, the 31 survey items were grouped into three broad concept domains, with mean scores calculated for “quality of care” (12 items), “information” (11 items) and “relationships with staff” (8 items) to have a more focused way of assessing certain key areas of organisational quality.

Data for the organisational audit were extracted from hospital admissions data and management information. We used an a priori algorithm developed by the ICWP to derive nine organisational domain scores, and from these an overall organisation score. These domains covered a broad spectrum of the organisation of care including the interdisciplinary nature of services both on and off the stroke unit, staff knowledge and skills, team working on records, meetings and agreed measures of assessment, availability of information, and communication with patients and carers. The clinical process of care audit was based on information obtained retrospectively from patient records. Eleven “yes/no” indicators were analysed, which had been identified in the 2001 national stroke audit to best represent the total clinical process.⁴

The distribution of organisation of care and overall patient experience scores were negatively skewed. Relationships between hospital‐level audit data on the organisation of care and patients' reports were tested with the Spearman correlation test, whereas the association between these and clinical process standards was assessed with the χ² test. Associations of each indicator with overall patient experience scores were assessed using the Mann–Whitney test.

Results

Selection

A total of 2729 patients were sampled for inclusion in the experience survey from the 51 trusts, of whom 1042 were also included in the audit (table 1 (A)). (Another 57 who were initially included in the patient survey were excluded from analysis as they were subsequently found to have died since their stroke.) Completed questionnaires were obtained from 670 (64%) of the 1042 patients. Audit patients in the survey had similar baseline characteristics to audit patients not in the survey, though they did have slightly better health status at discharge and a shorter stay in hospital (table 1 (A, B)). Audit patients from trusts participating in the survey were similar to audit patients from other trusts although they were slightly more likely to have been treated in a stroke unit (table 1 (A/B, D)). Survey responses from audit and non‐audit patients were similar (results not shown). However, the survey sample was not completely representative of all stroke patients with some demographic groups being under‐represented.¹²

Table 1 Characteristics of the sample.

	(A) National audit and patient survey	(B) National audit but not patient survey	(C) Patient survey but not national audit	(D) National audit patients in sites not in patient survey
	n = 1042	n = 680	n = 1687	n = 3625
Gender, % (n) male	52 (547)	48 (327)	52 (885)	51 (1851)
Age
Median (IQR)	74 (65–82)	77 (67–83)	75 (65–82)	76 (66–83)
% (n) ⩾75 years	50 (520)	56 (382)	51 (863)	53 (1934)
IMD 2000†, median (IQR) n	19 (11–34) n = 953		21 (12–36) n = 1585
LOS in hospital
Patient survey, mean (median)	24.4 (13)		20.5 (13)
National audit, mean (median)	27.2 (14)	37.0 (23)		31.6 (18)
Ethnic group, % (n) non‐white	7 (67/923)		6 (84/1476)
Stroke audit 2004, % (n)
Treated in stroke unit	61 (637)	59 (402)	Not applicable	51 (1841)
Spent most of stay in a stroke unit	55 (577)	50 (337)		45 (1616)
Pre‐stroke independent housing/warden controlled	95 (990)	93 (635)		94 (3413)
Discharge independent housing/warden controlled	82 (847/1031)	75 (495/663)		76 (2713/3553)
Three or more comorbidities*	22 (232)	22 (151)		21 (774)
On medication before admission	68 (704/1031)	66 (439/669)		69 (2484/3612)
Independent Barthel score of 20 at discharge	40 (349/881)	29 (166/563)		38 (1117/2930)
At time of maximum severity, % (n)
Orientated and could talk	72 (716/998)	66 (417/635)		68 (2322/3393)
Able to walk without help	38 (362/947)	30 (190/630)		33 (1108/3352)
Fully conscious	86 (892/1036)	78 (520/670)		81 (2898/3590)

IQR, interquartile range; LOS, length of stay.

*Atrial fibrillation, previous stroke/transient ischaemic attack, diabetes, hyperlipidaemia, hypertension, myocardial infarction/angina, vascular disease, valvular heart disease.

†The Index of Multiple Deprivation 2000 (IMD2000) is a composite index of relative deprivation at small area level, based on six domains of deprivation: income; employment; geographical access to services; health and disability; education, skills and training; and housing. Higher scores represent greater levels of deprivation.

Organisation of services

Patient experience scores for those sites with better organisation were higher both overall and within all three domains (table 2). Of the 58 sites that had a specialist unit, 38 had all five basic features of organisation that define a stroke unit. Two sites had two features, one had three features and 17 sites had four features. Patients in sites with all five organisational features were more positive about their experience of stroke care (table 2).

Table 2 Patients' experiences of care compared with audit assessments of organisational quality of stroke services.

	Hospitals n = 63	Overall organisational score		Five key features† of stroke unit organisation (58 sites with stroke unit)
		<median	⩾median	<5 features	All 5 features
		n = 31	n = 32	n = 20	n = 38
Overall patient experience score	Median score	71	75	71	76
	IQR score	65–77	70–81	65–76	70–80
	Spearman,* r (95% CI)	0.32 (0.07 to 0.53)		0.26 (0.01 to 0.48)
	p Value	0.01		0.05
Patient “quality of care” domain score	Median score	77	81	77	82
	IQR score	73–84	77–88	73–84	77–88
	Spearman, r (95% CI)	0.41 (0.18 to 0.60)		0.26 (0.01 to 0.48)
	p Value	0.001		0.05
Patient “information” domain score	Median score	62	66	62	65
	IQR score	56–65	58–75	57–66	58–73
	Spearman, r (95% CI)	0.27 (0.02 to 0.49)		0.17 (−0.09 to 0.40)
	p Value	0.03		0.21
Patient “relationships with staff” domain score	Median score	76	78	76	79
	IQR score	69–81	73–85	72–80	72–86
	Spearman, r (95% CI)	0.27 (0.02 to 0.49)		0.18 (−0.08 to 0.41)
	p Value	0.03		0.18

Results for the 670 responders were aggregated by hospital.

*Spearman coefficients were computed using actual overall organisational scores and the number of standard features of stroke unit organisation.

†These comprise: (1) consultant with specialist knowledge of stroke formally recognised as having main responsibility for stroke services; (2) stroke services has formal links with patient and carer organisations for communication on service provision, audit and future plans; (3) team meetings at least once a week for the interchange of information about individual patients; (4) patient‐orientated information literature displayed on unit/ward—either stroke‐specific or national/local guidelines/standards; and (5) inhouse programme for the continuing education of qualified staff in the management of stroke.

Process of care

The overall patient experience score was significantly higher for patients who, according to the audit, had (1) stayed in a stroke unit for more than half of their stay; (2) been screened for swallowing difficulties; (3) been assessed by a physiotherapist within 72 h of admission; and (4) been assessed by an occupational therapist within 7 days of admission (table 3; see table for p values). Similar patterns were seen for each of the three patient domains (results not shown). More specific cross‐matching of survey and audit questions showed weak trends in anticipated directions or no clear trends at all (table 4). Relationships between audit and survey responses were observed for the provision of information on secondary prevention.

Table 3 Patients' experiences of care compared with audit data on clinical process standards.

Audit result		Overall patient experience score
		n	Median (IQR)	p Value*
Over 50% of stay in stroke unit	Yes	371	81 (63–92)	<0.001
	No	297	73 (54–88)
Screening for swallowing <24 h	Yes	429	80 (61–92)	0.02
	No	200	76 (52–90)
Brain scan <24 h stroke	Yes	279	78 (61–92)	0.38
	No	229	81 (63–91)
Commenced aspirin <48 h of stroke	Yes	414	79 (61–91)	0.46
	No	126	81 (61–92)
Assessed by physiotherapist <72 h of admission	Yes	455	80 (54–89)	0.002
	No	190	71 (61–92)
Assessed by occupational therapist <7 days of admission	Yes	340	80 (61–92)	0.008
	No	178	76 (50–88)
Weighed at least once during admission	Yes	328	77 (60–91)	0.43
	No	335	80 (60–91)
Mood assessed	Yes	296	77 (60–90)	0.34
	No	367	80 (61–92)
On antithrombotic therapy by discharge	Yes	585	78 (61–91)	0.33
	No	23	75 (55–86)
Rehabilitation goals agreed by multi‐disciplinary team	Yes	373	78 (61–92)	0.29
	No	164	76 (55–90)
Home visit performed	Yes	198	76 (61–90)	0.42
	No	73	81 (57–93)

*Mann‐Whitney test.

Table 4 Comparison between specific patient experience and clinical audit questions.

Patient survey question	Patient survey response	Audit result			χ2 p value
Do you think your stroke was diagnosed quickly enough?	Yes (424)	55% (234) had brain scan <24 hours of stroke			0.22
	No (54)	46% (25) had brain scan <24 hours of stroke
Were you admitted to hospital quickly enough?	As soon as I thought necessary (535)	75% (400) same day as stroke, 15% (80) next day, 10% (55) later			0.15
	Should have been a bit/lot sooner (53)	70% (37) same day as stroke, 11% (6) next day, 19% (10) later
		Discussion with patient about:
Were you involved as much as you wanted to be in decisions about your care and treatment in hospital?		Diagnosis (p = 0.15)	Prognosis (p = 0.09)	Therapy goals (p<0.001)
	Yes, definitely	69% (225/326)	60% (195/324)	57% (181/315)
	Yes, to some extent	76% (133/176)	69% (123/178)	74% (130/175)
	No not involved in decisions about my care and treatment	66% (81/123)	59% (71/121)	55% (66/119)
		Discussion with carer about:
If a member of your family or someone else close to you wanted to talk to the staff, did they have enough opportunity to do so?		Diagnosis (p = 0.38)	Prognosis (p = 0.30)	Therapy goals (p = 0.19)
	Yes, definitely	58% (166/284)	55% (157/284)	53% (147/280)
	Yes, to some extent	63% (91/144)	62% (90/145)	62% (89/144)
	No but they wanted to talk to the staff	50% (13/26)	50%(13/26)	54% (13/24)
Was your stroke diagnosis discussed with you?	Yes, completely (305)	71% (216) Discussion with patient about diagnosis			0.98
	Yes, to some extent (196)	71% (139) Discussion with patient about diagnosis
	No it was not discussed (78)	72% (56) Discussion with patient about diagnosis
Before you left hospital did a member of staff give you information about stopping smoking?	Yes (104)	78% (81) Discussed other risk factors with patient			0.004
	No (44)	55% (24) Discussed other risk factors with patient
Did hospital staff give you information about changes in your diet that might help to prevent another stroke?	Yes (256)	58% (148) Discussed other risk factors with patient			0.08
	No (227)	50% (113) Discussed other risk factors with patient
Did hospital staff give you information about physical exercise that might help prevent another stroke?	Yes (328)	56% (182) Discussed other risk factors with patient			0.07
	No (170)	47% (80) Discussed other risk factors with patient

Discussion

Our analysis shows areas of both correlation and disparity between audit and patient survey data. The patient‐reported experience of hospital care correlated with audit assessments of the organisational quality of stroke services, showing that well‐organised stroke services are important for managing patients effectively³ and for improving their experiences. Furthermore, the finding that patients are more likely to report better experiences in sites with all five features considered essential for effective stroke unit organisation, emphasises the importance of implementing these organisational standards. This is consistent with previous research showing that some of the dimensions of care evaluated in the service organisational audit relate closely to patients' satisfaction with care, such as: doctor–patient communication; information provision; staffing levels; and service coordination.¹³,¹⁴,¹⁵,¹⁶,¹⁷,¹⁸

There was less correlation between patients' experiences of hospital care and the clinical process indicators. Only four of the 11 indicators were significantly associated with a more positive experience. There are a number of possible reasons why certain indicators correlated with patients' reports of their experiences and others did not. The four standards of care that did correlate may have been more visible or salient to patients: assessments by a physiotherapist and/or an occupational therapist represent additional contact from health services, demonstrating to patients that a multidisciplinary approach is being taken to their care. Being cared for on a stroke unit—one of the most positively correlated indicators—is associated with many clinical and organisational standards that probably have a positive influence on patients' experiences of care. A previous study also found that it was the dimensions of stroke care most visible to the patient which correlated most highly with their overall satisfaction.¹⁹ In contrast, some indicators may not have correlated with positive experiences due to their being less salient, from the patient's perspective, to the quality of care: patients may not always be cognisant of the value of being weighed or having their mood assessed. Other process of care standards may simply not be visible or memorable to patients, particularly if they take place immediately following admission, such as a brain scan, when patients might be less likely to subsequently recollect them. The overall patient experience score did not include questions on care following discharge, which might explain why the performance of a home visit was not seen to be significantly associated with a more positive patient experience.

Some studies in other areas of healthcare have reported a lack of correlation between patients' assessments of care and clinical‐based measures.²²,²³ A limitation of these studies, however, was their use of global rating scales instead of patient reports to assess patients' satisfaction with care. By using patients' reports of what actually happened during their hospital stay, it should be possible to gain a greater insight into the causes of satisfaction or dissatisfaction, and to identify specific events that did or did not occur in the care process. A difficulty in comparing audit and patient survey data is that many of the dimensions of care that are important to patients, and therefore covered in the questionnaire, are either not assessed or are measured differently in the audit. For instance, evidence that the patients' rehabilitation goals have been agreed by the multidisciplinary team—one of the key process standards—was not significantly correlated with patients' overall satisfaction with care. It is possible that if the audit had recorded whether the patients' own needs and wishes had been taken into account when setting such goals, this would have been associated with a positive patient experience. We should not, therefore, be surprised when patient reports differ from those recorded through clinical audit because the reports present two different methods of assessing the quality of care. Audit has an important role in assuring adherence to clinical standards, yet a patient‐centred system of healthcare is equally dependent on the ability to systematically collect and use feedback from patients on their views of the process: human elements of care such as effective communication simply cannot be quantified in any other way. This study demonstrates, then, that neither audit nor survey data is sufficient in isolation to present a comprehensive overview of quality of care.

Where it was possible to make direct comparisons between specific survey and audit questions, very little correlation was found. The provision of information on smoking cessation was the only survey question significantly correlated with a recording in the audit that risk factors had been discussed with the patient, although information provision on other aspects of secondary prevention (diet and physical exercise) were approaching statistical significance. Notable differences between clinicians' and patients' reports were evident with regard to communication about diagnosis. For almost three‐quarters of respondents who reported that their diagnosis had not been discussed with them, the audit recorded that such a discussion had taken place. This type of discrepancy has been found in other studies,²⁰ and a likely cause is that recording whether a diagnosis was discussed with the patient does not tell us anything about the quality of that interaction. A number of explanations for why information might not be effectively communicated to patients are cited in the literature: health professionals may feel they lack the time; the explanations given are too complicated; information is provided at the wrong time (such as when the patients' comprehension or recall was impaired); or it is too general and does not address the patients' own concerns.²⁵,²⁶,²⁷,²⁸ A further reason given is that health professionals avoid giving detailed explanations to patients due to the risk of encouraging over‐optimistic expectations of recovery following stroke.²⁵,²⁸,²⁹ Identifying areas where patients' and clinicians' reports do not concur is important for a better understanding of how the quality of care can be improved, and if this research were conducted at the level of the individual clinician or clinical team, it could be a very effective teaching aid for communication skills.

We believe this study is the first to have compared findings from a clinical audit and a patient survey at a national level. The ability to link directly the experiences of patients to clinical records on several dimensions of stroke care within the acute setting was a clear strength of the study. There were, however, some limitations. As previously stated, comparing responses from the patient survey to audit data was difficult because there was not an exact match in the dimensions of care being measured. Also, the survey results may have been biased in a positive direction because patients with poorer self‐reported health status were less likely to have responded.¹² It is known that patients with poorer health tend to give more negative assessments of their care.²⁴ Similarly, our analysis did not control for other factors that could have influenced patients' reports on the quality of care, such as stroke severity or comorbidity.

Conclusion

This study has shown that patients cared for in better organised stroke units, as measured by a clinical audit, reported a more positive experience of care in the patient survey. This strengthens the rationale for ensuring that all the recognised organisational standards are implemented in stroke units. In some areas disparities were found between patients and clinicians' reports on the processes of care, and this emphasises the intrinsic differences between the two sources of feedback. Indeed, an important reason for conducting patient experience surveys is that they measure standards that cannot be reliably measured by other means. Future evaluations of stroke services should therefore undertake both an audit of clinical records and a patient survey for a complete assessment of care.

Abbreviations

ICWP - Intercollegiate Stroke Working Party

NSF - National Service Framework