Abstract
Background/aims: To assess reliability and validity of the QUOTE-cataract, a questionnaire that measures the quality of care from the perspective of cataract patients.
Methods: The QUOTE-cataract was tested in a multicentre study among 540 cataract patients in three different hospitals. Reliability was represented by internal consistency (Cronbach’s α), and repeatability (intraclass correlation coefficient (ICC)). Validity was evaluated qualitatively and by factor analyses.
Results: A strong internal consistency coefficient (0.89), and high repeatability (ICC = 0.76) demonstrated good reliability. Content validity was assured by involvement of patients in the development of the questionnaire. Factor analysis confirmed an underlying taxonomy of generic and disease specific items.
Conclusion: The QUOTE-cataract has good reliability and provides a valid assessment of quality of care in cataract surgery.
Keywords: cataract, quality of health care, QUOTE-Cataract
In studies assessing the effectiveness of cataract surgery the importance of the patient’s perspective has been recognised. Besides clinical outcomes like visual acuity, assessments are directed at subjective measures of vision related functioning.1,2 However, a valid assessment of the quality of ophthalmic services from the patient’s perspective should consider all components that contribute to the quality of these services—objective outcome criteria (functional tests), subjective outcome criteria (patient satisfaction), and criteria related to the way the services are provided (structure and process of care).3–6 Previous research examining quality of care from the user’s perspective has been dominated by patient satisfaction surveys.7 Doubts have been cast on the validity of such research for two main reasons. Firstly, these studies usually do not involve patients in the development of instruments. Therefore, outcomes tend to reflect the perspective of managers and/or professionals and/or researchers, rather than the distinct view of the patient.8 Secondly, results show little insight into user views because satisfaction ratings are usually highly skewed, while no recognition is given to the fact that some quality aspects of care are more important than others.9,10 This has led reviewers to conclude that research into user views of quality of care has suffered from methodological weakness, and low specificity of results.11–13 We developed the QUOTE-cataract questionnaire (QUality Of care Through the patient’s Eyes) that measures the quality of care from a cataract patient’s perspective, which attempts to overcome these problems. The QUOTE-cataract closely resembles the work that was carried out on other QUOTE-instruments (for example, QUOTE-Rheumatic-Patients and QUOTE-Elderly) as described by Sixma et al.8,14 This instrument produces data on the importance and performance of healthcare services, which are related to the specific needs and expectations of cataract patients. Besides being useful and applicable to clinical practice, an instrument like the QUOTE-cataract should satisfy scientific quality standards. The aim of this study was to assess the reliability and validity of the QUOTE-cataract questionnaire.
PATIENTS AND METHODS
Patients
This study consisted of patients who were treated at three settings: University Hospital Maastricht (UHM, n = 166), University Hospital Groningen (UHG, n = 130), and the Rotterdam Eye Hospital (REH, n = 244). The mean age (SD) of the patients was 71.8 (8.0) years (UHM); 73.9 (9.2) years (UHG), and 71.9 (9.3) years (REH), respectively. The male: female ratio was 0.6 for all three settings. At REH more patients were categorised as “medium/higher educated” in comparison with UHM and UHG (Kruskal-Wallis, p<0.01). General health was reported as “good” by most patients, but differed significantly between the three settings with the highest scores at REH (Kruskal-Wallis, p<0.01). Criteria for participation in the study were first or second eye surgery for age related cataract 2–8 months earlier, and absence of ocular comorbidity. All cataract surgeries took place on an outpatient basis using a standardised phacoemulsification technique with implantation of an intraocular lens.
The questionnaire
The QUOTE-cataract includes a generic and a disease specific part (see Table 1). To generate these items focus group interviews with cataract patients and concept mapping were performed.15,16 Generic questions (i = 15) are applicable to a range of users of health care (that is, patients) and refer to process and structure related care aspects, whereas disease specific items (i = 16) are tailored to cataract patients. Four disease specific items were not included in the QUOTE-cataract at UHG because they were characteristic for UHM and REH. The items incorporated in the questionnaire measure the relative importance of the different quality aspects (for example, “My ophthalmologist should inform me clearly about what I may and may not do after cataract surgery”) and the performance of the healthcare services on each of the quality aspects (for example, “My ophthalmologist informed me clearly about what I may and may not do after cataract surgery”). Response options of the importance categories are “not important,” “fairly important,” “important,” and “extremely important.” Scores were calculated by linear transformation of standardised values (Z scores) to values between 0 (“not important”) and 10 (“extremely important”).14,17 Response options for perceived performances (1 = “no,” 2 = “not really,” 3 = “on the whole, yes,” and 4 = “yes”) were expressed as proportions of respondents who did not (score 1 or 2) perceive the particular quality aspect as being performed.8 Individual (i) performance (P) and importance scores (I) on different quality of care aspects (j) are used to calculate quality impact indices (Q), applying the formula Qij = Pij × Iij. Theoretically a quality impact score can vary from 0 (best possible quality of care) to 10 (all respondents think that this aspect is extremely important and 100% of the patients report that it needs improvement).
Table 1.
Quality aspects listed in the QUOTE-cataract
| Generic indicators | |
| 1.1 | Allow me to decide on which treatment/help to get |
| 1.2 | Take me seriously |
| 1.3 | Keep appointments punctually |
| 1.4 | Never let me wait in the waiting room longer than 15 minutes |
| 1.5 | Give me good directions for use of the prescribed medicine |
| 1.6 | Take care that prescribed medicines are free of charge |
| 1.7 | Be easily accessible by phone |
| 1.8 | Take care that I can consult a specialist within 2 weeks |
| 1.9 | Take care of fine tuned care of different caregivers |
| 1.10 | Know my health status very well |
| 1.11 | Tell me what to do in case of emergency |
| 1.12 | Have a room accessible for wheelchair users/disabled |
| 1.13 | Always inform me about the risks of a treatment |
| 1.14 | Always take enough time for me |
| 1.15 | Be willing to talk about affairs I did not like |
| Cataract specific indicators | |
| 2.1 | Inform me clearly what I may and may not do after cataract surgery |
| 2.2 | Inform me in writing about the day and time of my cataract surgery* |
| 2.3 | Inform me in the meantime how long it will take before surgery* |
| 2.4 | Provide an information brochure about cataract surgery |
| 2.5 | Tell me during cataract surgery exactly what he/she is doing |
| 2.6 | Always be calm and quiet while working |
| 2.7 | Always consider my personal desires |
| 2.8 | Always be friendly |
| 2.9 | Always take all my questions related to cataract surgery seriously |
| 2.10 | Always be the same person |
| 2.11 | Take care that the wait for surgery is not longer than 2 months |
| 2.12 | Take care that all preoperative examinations will take place at the same day* |
| 2.13 | Have at one’s disposal a desk with enough privacy |
| 2.14 | Take care of food and drinks at the ward/in the waiting room* |
| 2.15 | Take care that patient education is regulated among the different doctors |
| 2.16 | Give me the opportunity to come first occasionally |
*Quality of care aspects not included in the QUOTE-cataract at UHG.
Reliability and validity testing
A postal survey involved 540 patients who had undergone cataract surgery. Cronbach’s α was used to represent the internal consistency of the QUOTE-cataract based on one rating. A “benchmark” of 0.80 was considered as an acceptable value. In accordance with previous reports on the assessment of test-retest stability (repeatability), scoring of the QUOTE-cataract instrument was repeated after 2–3 weeks by 289 cataract patients from the UHG and REH.18 A Bland-Altman plot of agreement and the intraclass correlation coefficient demonstrated variation in test-retest.19,20
Content validity was optimised by involving cataract patients in the development process. Moreover, a question was included to check if patients felt that particular aspects of relevance to them were missing in the questionnaire. Furthermore, factor analysis was performed to examine the taxonomy/structure of the quality of care concept.
Statistical analysis/statistics
Importance scores were used in statistical tests to assess the reliability and validity of the scales. While performance scores will change when situations in healthcare services change, importance scores are less subject to situational changes as they are linked to the attitudes and opinions of patients.8 Cases with more than 10% missing values on the importance scores were excluded from reliability and factor analyses, remaining missing values were replaced by the mean. The Bland-Altman plot indicated stability of test results by demonstrating variation in scores as part of unreliability. One way ANOVA (random effect model) was used to obtain the intraclass correlation coefficient for retest reliability according to the following formula: ICC = 1 − (within subject variance/between subject variance). Factor analysis (principal component) was carried out to represent the internal structure of the questionnaire. The appropriateness of the factor analytic model was tested using the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy and Bartlett’s test of sphericity. Analyses were performed using spss 10.0 for Windows (SPSS Inc, Chicago, IL, USA) software.
RESULTS
Response to questionnaires
The total number of completed questionnaires returned was 166 (response rate 92.7%), 130 (response rate 65%), and 244 (response rate 74.4%) for UHM, UHG, and REH respectively.
The mean (SD) of the importance scores was similar comparing the three settings: 6.1 (1.6), 6.2 (1.2), and 5.8 (1.2) for UHM, UHG, and REH respectively (ANOVA, p = 0.13). Performance sores ranged from 0.01 to 0.49, from 0.01 to 0.49, and from 0.00 to 0.44 for UHM, UHG, and REH respectively. Importance scores were less skewed in our data set than the perception scores (range of skewness scores −0.58 to 0.30 and 0.43 to 11.3, respectively). Quality impact factors ranged from 0.05 to 3.39 (UHM), from 0.05 to 2.79 (UHG), and from 0.00 to 3.00 (REH), with skewness scores ranging from 0.70 to 1.58.
Reliability
The mean internal consistency coefficient of the three hospitals was 0.89, 0.84, and 0.85 for the QUOTE-cataract in total, and generic, and disease specific subscales, respectively. The mean (SD) importance scores for first and second assessments were 6.2 (1.2) and 6.2 (1.3) for UHG, and 5.8 (1.2) and 5.9 (1.4) for REH (paired t test, p = 0.89/0.61). The differences in first and second assessments were plotted against the mean QUOTE-cataract score for each patient (see Fig 1), which showed no relation between the measurement error and the true value. The mean of the differences between the pairs was negligible (mean difference −0.04). Furthermore, the limits of agreement were small, an individual measurement difference is expected within a range from −2.34 to 2.34. A strong intraclass correlation coefficient confirmed this consistency (average measure ICC = 0.76).
Figure 1.
Bland–Altman plot of agreement in importance scores between test and retest. The mean of the test and retest scores is plotted on the x axis and the differences between the two scores on the y axis. The horizontal interrupted lines represent the limits of agreement (within 2 SD from the mean).
Validity
The appropriateness of the factor analytic model was assessed by KMO (0.92) and Bartlett (χ2= 4131.5, p<0.01), which demonstrated legitimacy of the procedure. Unrotated factor analysis identified two common factors that explained 35.3% of the variance. When analysing generic and specific items separately, both generic and disease specific items clustered into one factor with moderate factor loadings (range 0.45–0.69). Explained variances were 31.6% and 33.7% respectively. The percentage of patients who responded to the question “Were any quality of care aspects missing in this questionnaire?” was 27.1, 26.9, and 22.5 for UHM, UHG, and REH respectively (χ2 test, p= 0.49). However, most answers related to alternative expressions of items already included in the QUOTE-cataract. Real new aspects, formulated by 6,7% of the patients, concerned the anaesthesiology before cataract surgery (n = 3), follow up visits (prescription of glasses and information about secondary cataract or other complications; n = 22), reassurance (n = 4), costs (n = 4), and technical competence (n = 3).
DISCUSSION
Former research emphasised the relevance of process and structure related quality of care with respect to satisfaction after cataract surgery.1 However, the reliability and validity in measuring the multidimensional concept satisfaction has been criticised frequently based upon the global approach and problems regarding methodological weakness (for example, highly skewed).11–13 The QUOTE-cataract instrument was developed to assess patient’s interests in a more specific and direct way, as recommended by Fitzpatrick.21 We showed that the QUOTE-cataract had low skewness scores in comparison with distributions of 80–90% generally categorised as satisfied.1 The internal consistency of the questionnaire was shown to be as good as the QUOTE-Rheumatic-Patients and QUOTE-Elderly with Cronbach’s α values of 0.92 and 0.93 respectively.8,14 Test-retest data for the QUOTE-cataract confirmed repeatability, there being negligible differences between pairs using the method of Bland and Altman, and a high intraclass correlation coefficient (ICC = 0.76). Factor analysis showed that category specific items clustered into one component, which was also comparable with the QUOTE-Rheumatic-Patients and QUOTE-Elderly.8,14 Four per cent of the patients missed an aspect regarding “follow up visits” in the questionnaire. Future research should indicate whether this missing aspect is of additional value for the questionnaire. The questionnaire as developed at UHM and UHG was shown to be applicable to the REH although four aspects were not included in the QUOTE-cataract at UHG. These aspects seemed to be institute specific for UHM and REH. One should recognise that subtle differences in communities require differences in assessment.22 Therefore, it is recommended that validation of the QUOTE-cataract is carried out at each specific setting and location before its administration. Since process and structure related quality of care (for example, patient education and counselling) have been proved to be highly correlated with patient satisfaction,1 we believe that the QUOTE-cataract may be an adequate instrument to measure patient satisfaction after cataract surgery in a more reliable and valid way.
Acknowledgments
Anne Lems and Maria Dolders, MHSc, are acknowledged for their contribution to the field work. The authors wish to thank Christine Markham, PhD, for helpful linguistic comments on this manuscript.
Footnotes
The QUOTE-cataract questionnaire is available upon request.
REFERENCES
- 1.Nijkamp MD, Nuijts RM, Borne B, et al. Determinants of patient satisfaction after cataract surgery in 3 settings. J Cataract Refract Surg 2000;26:1379–88. [DOI] [PubMed] [Google Scholar]
- 2.Steinberg EP, Tielsch JM, Schein OD, et al. The VF-14. An index of functional impairment in patients with cataract. Arch Ophthalmol 1994;112:630–8. [DOI] [PubMed] [Google Scholar]
- 3.Desai P. Assessing ophthalmic services. Eye 1995;9:677–8. [DOI] [PubMed] [Google Scholar]
- 4.Spaeth GL, MD. Why measuring quality of care, a vitally important task, is presently being done improperly. Ophthalmic Surg 1989;20:165–7. [PubMed] [Google Scholar]
- 5.Lledo R, Rodriguez T, Fontenla JR, et al. Cataract surgery: an analysis of patient satisfaction with medical care. Int Ophthalmol 1999;22:227–32. [DOI] [PubMed] [Google Scholar]
- 6.Pankow L. Seeing through the patient’s eyes. J Am Optom Assoc 1992;63:678–9. [PubMed] [Google Scholar]
- 7.Campen Cv, Sixma H, Friele RD, et al. Quality of care and patient satisfaction: a review of measuring instruments. Med Care Res Rev 1995;52:109–33. [DOI] [PubMed] [Google Scholar]
- 8.Campen Cv, Sixma HJ, Kerssens JJ, et al. Assessing patients’ priorities and perceptions of the quality of health care: the development of the QUOTE-rheumatic-patients instrument. Br J Rheumatol 1998;37:362–7. [DOI] [PubMed] [Google Scholar]
- 9.Sitzia J, Wood N. Patient satisfaction: a review of issues and concepts. Soc Sci Med 1997;45:1829–43. [DOI] [PubMed] [Google Scholar]
- 10.Zastowny TR, Stratmann WC, Adams EH, et al. Patient satisfaction and experience with health services and quality of care. Quality Management in Health Care 1995;3:50–61. [DOI] [PubMed] [Google Scholar]
- 11.Rubin H. Patient evaluations of hospital care: a review of literature. Med Care 1990;28:S3–9. [DOI] [PubMed] [Google Scholar]
- 12.Williams B. Patient satisfaction: a valid concept? Soc Sci Med 1994;38:509–16. [DOI] [PubMed] [Google Scholar]
- 13.Ross C, Steward C, Sinacore J. A comparative study of seven measures of patient satisfaction. Med Care 1995;33:392–406. [DOI] [PubMed] [Google Scholar]
- 14.Sixma H, Campen Cv, Kerssens J, et al. Quality of care from the perspective of elderly people: the QUOTE-Elderly instrument. Age Ageing 2000;29:173–8. [DOI] [PubMed] [Google Scholar]
- 15.Morgan DL. Focus groups as qualitative research. Beverly Hills, CA: Sage Publications Inc, 1988.
- 16.Trochim WMK. The concept system. Ithaca: Concept Systems, 1993.
- 17.Swanborn PG. De Likertschaal. In: Schaaltechnieken: theorie en praktijk van acht eenvoudige procedures. Meppel: Boom, 1982:96–103.
- 18.Deyo RA, Diehr P, Patrick DL. Reproducibility and responsiveness of health status measures. Statistics and strategies for evaluation. Control Clin Trials 1991;12:142s–58s. [DOI] [PubMed] [Google Scholar]
- 19.Bravo G, Potvin L. Estimating the reliability of continuous measures with Cronbach’s alpha or the intraclass correlation coefficient: toward the integration of two traditions. J Clin Epidemiol 1991;44:381–90. [DOI] [PubMed] [Google Scholar]
- 20.Bland J, Altman D. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;1:307–10. [PubMed] [Google Scholar]
- 21.Fitzpatrick R, Hopkins A. Problems in the conceptual framework of patient satisfaction research: an empirical exploration. Sociology of Health and Illness 1983;5:297–311. [DOI] [PubMed] [Google Scholar]
- 22.Pesudovs K, Coster DJ. An instrument for assessment of subjective visual disability in cataract patients. Br J Ophthalmol 1998;82:617–24. [DOI] [PMC free article] [PubMed] [Google Scholar]