Primary care spirometry: test quality and the feasibility and usefulness of specialist reporting

Abstract

Background

Provision of spirometry for chronic obstructive pulmonary disease (COPD) is a new requirement in primary care. Effective spirometry requires that tests and interpretations meet international criteria.

Aim

To assess the feasibility and usefulness of remote specialist reporting of primary care spirometry.

Design of study

Comparison of reporting by primary care clinicians and respiratory specialists of consecutive primary care spirometry tests.

Setting

South London primary care teams with patient lists ≥6000.

Method

Feasibility of remote reporting of spirometry was assessed by the frequency of electronic mailing of tests. Usefulness of remote reporting was defined by the frequency that specialist reports made a clinically significant addition. Usefulness was assessed by measuring agreement (κ) between primary care reports and those of specialists. Clinically significant disagreements were analysed with respect to test quality, diagnosis, and severity.

Results

Six practices emailed 312 tests over 3 months. Forty-nine tests sent without indices or curves (flow volume and time volume) were excluded. Mean age of patients tested was 65 years and 52% were female. Mean predicted forced expiratory volume in the first second (FEV1) was 69%. Clinically significant disagreements were identified in the interpretation of acceptability (quality) of 67/212 (32%) tests (κ = 0.07; 95% confidence interval [CI] = 0 to 0.24), of diagnosis in 49/168 (29%) tests (κ = 0.39; 95% CI = 0.25 to 0.55), and of severity in 62/191 (32%) tests (κ = 0.53; 95% CI = 0.43 to 0.63).

Conclusion

Remote reporting of primary care spirometry was feasible. Its usefulness was confirmed by the high rate of additional clinically significant information to the reports of primary care clinicians. The quality of primary care spirometry was so unsatisfactory that remote reporting of tests may be a means of establishing adequate spirometry.

INTRODUCTION

Spirometry is the key test of diagnosis and severity in chronic obstructive pulmonary disease (COPD).¹ There is growing evidence that spirometry is needed to challenge a presumed diagnosis of COPD in a primary care setting.^2,3 The provision of spirometry for patients with COPD is demanding for primary care teams. It has been included in the International Primary Care Respiratory Group (IPCRG) guidelines for the diagnosis of respiratory diseases within primary care.⁴ It has also been included as an incentive in the new contract for GPs in the UK NHS.⁵ This new contract does not require the achievement of a quality threshold in spirometry. The unstated assumption is that the professionalism of primary care clinicians will ensure that tests are performed to an acceptable standard.

However, this assumption is not well supported since spirometry is not a core skill in primary care, direct access to specialist provision of spirometry is unusual for primary care teams in the UK, and there are no mechanisms for quality control in primary care spirometry provision.

Spirometry can be provided by primary care clinicians in their surgeries, or by referral to a locality-based service, or to a hospital lung function laboratory.^2,6,7 The most convenient and cost-effective way of providing spirometry for patients is within primary care, but while the quality of primary care tests can be excellent, the accuracy of the interpretation in primary care may also give cause for concern.^7,8 The basic skills required for training in spirometry should be readily available in primary care teams and indeed primary care spirometry can compare favourably with spirometry performed in specialist units.⁷ There is currently no standard for the training and conduct of primary care spirometry. Special expertise in spirometry interpretation and reporting is likely to be unusual among primary care clinicians.

How this fits in

Recent guidelines have recommended that spirometry should be undertaken in primary care for people suspected of having chronic obstructive pulmonary disease (COPD). GPs in the UK receive incentive payments for doing spirometry but the quality of the spirometry done is uncertain. This study shows that remote electronic reporting by specialists of spirometry undertaken in primary care was feasible. It was also useful because the quality of the spirometry done in primary care was unsatisfactory. Remote reporting may be a means to establishing acceptable spirometry in a primary care setting.

Added to these difficulties is the relative infrequency with which spirometry is required in primary care and which, in turn, may prolong the time over which adequate experience is acquired. These obstacles to the introduction of primary care spirometry may be overcome by providing effective remote supervision. In this study, the feasibility and usefulness to primary care teams of remote (electronic) specialist support of primary care-based spirometry was tested.

METHOD

Six general practices were selected at random from 29 practices who expressed an interest in the study. Forty-four practices with list sizes greater than 6000 were approached in the London Boroughs of Lambeth and Southwark. Participating practices agreed to conduct spirometry tests on at least 50 patients selected from their COPD registers. Practices were provided with an electronic hand-held spirometer (MicroLoop®, Micro Medical, UK) with PC-based software, training in spirometry and the recording of spirometry tests, and reimbursement of £10 for the cost of each spirometry test. The built-in electronic spirometry interpreter was switched off for the duration of the study. Tests were carried out by practice nurses in five of the practices and by a care assistant in one. Training was provided at the lung function laboratory of the local respiratory medicine unit where a group classroom session of 2 hours was followed by two 3-hour individual clinical tuition sessions with laboratory subjects. Spirometry testers and their clinical colleagues who did the test reporting were invited to attend. A final individual tuition session was provided to each spirometry tester by the manufacturer of the spirometer.

Practices were asked to perform spirometry tests according to American Thoracic Society criteria.⁹ Tests were electronically downloaded onto a practice computer, and exported as Microsoft® Word documents. Consecutive test results were emailed as attachments to the relevant primary care clinician (GPs in five practices and a practice nurse in one) and the local respiratory specialists for interpretation, and to the research team. Interpretations were written on a pro forma grid attached to the spirometry report under the headings:

• Acceptability of the test;

• Diagnosis;

• Severity of impairment;

• Management advice; and

• Confidence in the interpretation of the test.

Spirometry tests and their attached reports from the practice clinicians and from the respiratory specialists were forwarded by email to the researchers.

Tests were excluded from the analysis where spirometry indices (forced expiratory volume in the first second; [FEV1], forced vital capacity, [FVC]; FEV1 as a percentage of that predicted for age and height; [FEV1%], and ratio of FEV1 to FVC; [FEV1/FVC]), or the associated flow-volume and time-volume curves were omitted. The feasibility of a remote electronic approach to lung function reporting was assessed by the frequency with which the practices completed the tests and produced electronic reports. The usefulness of remote reporting was defined by the frequency with which specialist reports made a significant addition to the primary care clinician's reports. It was assessed by measuring the agreement (Cohen's κ and weighted κ) between the specialist reports and the primary care reports of the spirometry in terms of the acceptability of the test done, the diagnosis, and the severity of disease. Agreement assessed by κ was interpreted as slight (0.01–0.2), fair (0.21–0.4), moderate (0.41–0.6), substantial (0.61–0.8), or almost perfect (0.81–1.0).

Analysis of agreement on acceptability, diagnosis, and severity was completed on emailed tests for which the relevant question in the pro forma was answered by both the primary care clinician and the specialist. Acceptability of the spirometry test was categorised as: acceptable/unacceptable/don't know. Diagnoses given were categorised as: normal/obstructive airways disease/restrictive lung disease/mixed disease/COPD/asthma/not COPD/not reversible airways disease/other/don't know. The diagnoses made were not prompted on the pro forma grid and the rationale for the diagnosis offered was not requested. Severity was categorised as severe/moderate/mild/normal/don't know. For the testing of agreement, diagnosis was compressed to the categories: obstructive/other. Obstructive lung disease was defined as the combination of FEV1 <80% predicted for age and height, and FEV1/FVC ≤70%.¹

In addition to measuring κ, a threshold rate of disagreement at which clinically significant disagreement between the primary care team and the specialist was judged to indicate that remote reporting was useful, was set at 15% of tests. This was based on an arbitrary judgement that if one of six or fewer reports provided by the specialists generated additional clinically useful information, the incentive for primary care teams to use the remote reporting service as a routine would be small. Clinically significant disagreements between the interpretations of the primary care clinicians and the specialists were assessed by a multidisciplinary panel comprising a chest physician, a respiratory physiologist, a GP, and a researcher. Disagreements were assessed with respect to the acceptability of the test, the diagnosis of obstructive airways disease, and the severity of the disease. Disagreements were defined as clinically significant if the judgement of unacceptability would lead to a test having to be repeated, if the judgement of diagnosis would lead to incorrect attribution of the diagnosis of obstructive airways disease, or if the judgement of severity would lead to incorrect attribution of the level of risk. Interpretations by the respiratory specialists were used as the benchmark for comparative purposes but had to be in alignment with international criteria and to be acceptable to the multidisciplinary panel, which included a GP with a special interest (GPwSI) in respiratory disease.⁹

RESULTS

Six practices carried out 312 tests over 3 months (July, August, and September 2005). Mean practice list size was 10 224 patients (6912–18 323), and mean number of patients per full-time equivalent GP was 1866 (1562–2562). Two of the practices were engaged in routine spirometry prior to the study. None of the practices had a GPwSI in COPD providing spirometry interpretation. Subjects were patients with a clinical diagnosis or clinical suspicion of COPD.

Of the 312 tests, 49 were excluded because they did not have either the spirometry indices (FEV1, FVC, and FEV1/FVC ratio) or the spirometry curves (flow-volume curve and time-volume curve). The analysis was performed on tests from 263 subjects: mean age 65 years (standard deviation 13.8; range 19–94 years); 52% female; mean % predicted FEV1 69%; mean % predicted FVC 85%; mean FEV1/FVC ratio 0.68. Severity of airways obstruction, based on % predicted FEV1 and FEV1/FVC ratio and graded according to the NICE guidelines,¹ was categorised as: normal — 31%; mild — 48%; moderate — 17%; and severe — 4%. Although provision had been made for practices to obtain specialist reporting on a continuous basis throughout the study, most primary care teams sent the reports in batches. This was due largely to the unfamiliarity of the team member performing the test with the process of attaching Microsoft® Word documents to email correspondence.

Reports on 219 of the 263 eligible tests were received from primary care clinicians. Electronic copies of 239 were sent to, and reported on, by specialists. A total of 212 tests were reported on by both groups of clinicians. Explanations for the reporting of only 219 tests by primary care clinicians and the failed transmission of 24 reports to the specialists, were not obtained and could not be investigated due to the transmission of results in batches. Agreement (Cohen's κ) was sought between the primary care clinicians and the specialists regarding tests on which both had reported (Table 1). A total of 212 tests had been reported with respect to acceptability by both primary care clinicians and specialists, 168 with respect to diagnosis, and 191 with respect to severity. Agreement between the primary care clinicians and the specialists was slight with respect to acceptability (quality) of the test (n = 212; κ = 0.07; 95% confidence interval [CI] = 0 to 0.24) (Table 1). Clinically significant disagreements, excluding ‘don't know’ categories, were identified in the interpretation of 67/212 (32%) tests. The specialists' reasons for the tests being unacceptable were that the blow was too short (less than 6 seconds) in 39 cases (18%); that the subject had failed to exhale as quickly as possible at the beginning of the blow in 18 cases (8%); or that the subject had taken a second breath or cough during exhalation in 38 cases (18%). These judgments were confirmed by the multidisciplinary panel.

Table 1.

Comparison of spirometry interpretations and assessment of agreement (Cohen's κ) between primary care clinicians and respiratory specialists in tests on which both reported.

	Specialist
Primary care clinician	Yes	No	Don't know		κ (95% CI)
Acceptable? (n = 212)
Yes	109	63	6
No	4	7	0		0.07
Don't know	14	6	3		(0 to 0.24)
	Obstructive	Other
Diagnosis? (n = 168)
Obstructive	79	15			0.39
Other	34	40			(0.25 to 0.55)
	Normal	Mild	Moderate	Severe
Severity? (n = 191)
Normal	49	20	1	0
Mild	7	50	3	1
Moderate	1	19	23	0	0.53
Severe	0	5	5	7	(0.43 to 0.63)

Agreement with regard to diagnosis was fair (n = 168; κ = 0.39; 95% CI = 0.25 to 0.55). Clinically significant disagreements were identified in the interpretation of the diagnosis of obstructive lung disease in 49/168 (29%) tests. The main reason for the disagreements was that obstructive lung disease should have been diagnosed by primary care clinicians where the indices (FEV1, FVC and FEV1/FVC ratio) were diagnostic according to international criteria. In 18 cases, the primary care clinicians reported a restrictive defect where the criteria for a restrictive defect were not met.

Agreement was moderate with respect to severity (n = 191; κ = 0.53; 95% CI = 0.43 to 0.63). Disagreements were identified in the interpretation of severity in 62/191 (32%) tests. The specialists' disagreed with the primary care clinicians where the primary care clinicians did not adhere to national criteria for the categorisation of severity in obstructive lung disease.¹

The level of agreement (Cohen's κ) between the primary care clinicians and the specialists varied in each practice with respect to each of these three categories of assessment. Agreement on acceptability ranged from 0.77 in the practice with the highest level of agreement (three tests on which the clinicians disagreed; n = 26), to 0.02 in the practice with the lowest (23 tests on which they disagreed; n = 40). Agreement on diagnosis varied from 0.51 in the practice with the highest level of agreement (seven tests on which the clinicians disagreed; n = 30) to 0.04 in the practice with the lowest (11 tests on which they disagreed; n = 23). Agreement on severity varied from 0.87 in the practice with the highest level of agreement (three tests on which the clinicians disagreed; n = 32) to 0.40 in the practice with the lowest (17 tests on which they disagreed; n = 38).

Primary care clinicians reported on the certainty of their interpretations in 211 tests. In 136 (64.5%), the primary care clinicians reported being certain about their interpretation. They were significantly more likely to have had confidence in their interpretation with respect to tests that they had judged to be acceptable (χ² = 30.2; degrees of freedom = 4; P<0.001). The management decisions were not analysed because of the high level of disagreement about acceptability of the tests.

DISCUSSION

Summary of main findings

The quality of spirometry performed in participating practices was low, with over 15% of tests sent for reporting without complete data, and almost 40% of those that were complete reported by specialists to have been unacceptable. Agreement (Cohen's κ) between specialists and primary care clinicians on acceptability was slight, on diagnosis was fair, and on severity was moderate. Clinically significant disagreements were observed between the specialist and the primary care interpretation of these tests in terms of the acceptability of the test (32%), the diagnoses (29%), and severity (32%). The specialist reports were in keeping with international criteria and support the potential role of specialist reporting of primary care-based spirometry in the quality control of test acceptability and interpretation, especially in the initial stages of establishing spirometry recording within a primary care setting.

The study was designed to assess feasibility and usefulness of remote specialist reporting, but, in addition, the results point to unacceptable quality in the provision of spirometry within primary care for patients with COPD. The clinical interpretation of spirometry tests requires competence in determining if the test was done adequately and in assessing the implications of the findings. The training provided for the clinical staff who performed the spirometry tests in this study was at least as comprehensive as that obtained by most primary care teams in the UK. Despite this, 49 tests were sent to the primary care clinicians and the specialists without spirometry indices, without the essential spirometry curves, or without both. If these incomplete tests are added to those found to be unacceptable by the specialists, then the overall rate of unacceptable tests is 52%. It is not justifiable to invite a patient for a test which may have a 50:50 chance of being conducted inadequately. Even allowing for concerns about the underlying quality of the tests done in primary care, in almost 30% of reports the specialists and primary care clinicians disagreed about the diagnosis, in each example of which the specialists' interpretations were in keeping with international criteria.

Strengths and limitations of the study

The study was based on a small sample of practices and cannot claim to be representative of primary care as a whole. Its aim was to test the feasibility and usefulness of electronic reporting of spirometry in a setting that was characteristic of the average NHS general practice. The main obstacle in sending electronic reports to the primary care clinicians and to the specialists for interpretation was unfamiliarity with attaching documents to emails.

There was wide variation between the practices in the rate of disagreement with the specialists. The process could have been used to identify practices whose spirometry interpretation was accurate enough to allow independent reporting of the tests. The spirometry reports were returned to the practices by the specialist within 1–2 days of their receipt, so there was no attempt to conceal the specialist interpretations from the primary care teams.

Using this model, the ratio of agreements to disagreements for each participating practice could be used as a benchmark for the quality of primary care spirometry where an external reviewer of spirometry tests is employed. A specialist respiratory department was used to provide the remote reporting, but this approach could be supported by GPwSIs in spirometry.⁶ A further level of analysis comparing the primary care and specialist reporting with the built-in electronic interpretation in the hand-held spirometers was beyond the scope of this study. Separate testing of automatic electronic interpretation of spirometry for accuracy and reliability is needed before it can be proposed as a possible substitute for immediate or remote specialist reporting.

Comparison with existing literature

Previous studies have shown that spirometry can be performed in primary care to the highest quality.⁷ However, reservations have also been expressed that the quality of primary care spirometry may be unacceptably poor.⁸ The quality of spirometry is likely to be determined by several factors including the quality and length of spirometry training, the aptitude of the spirometry tester, supervision after completion of training, and the quality of the interpretation of tests carried out. The findings suggest that current training for NHS primary care spirometry will not in itself produce tests of acceptable quality. It is possible that if the spirometry tests had been sent to the specialists as they were performed, rather than in batches, an improvement in the acceptability and the quality of interpretation over time would have been observed. Expert interpretation of reports may help to assure the quality of the tests by providing immediate feedback.

Implication for future research and clinical practice

Electronic reporting of spirometry presents a model of specialist feedback which could be applied to a variety of near-patient tests in primary care including electrocardiography and imaging.^10–13 Systems for remote consulting using digital images have already been widely tested. Electronic reporting is an approach that offers efficiencies in time and human resources and promotes the ‘devolution’ of services to community centres.

The feasibility and usefulness of remote electronic reporting of primary care-based spirometry has been demonstrated. The next step is to discover if the quality of spirometry testing and interpretation in primary care can be improved by the availability of remote electronic reporting.

Funding body

This study was funded by the Research Support Unit of Lambeth, Southwark and Lewisham Primary Care Trusts (RDLS 79)

Ethics committee

Ethical approval was obtained from King's College Hospital Local Research Ethics Committee (04/Q0703/117)

Competing interests

The authors have stated that there are none