An official website of the United States government
Here's how you know
Official websites use .gov
A
.gov website belongs to an official
government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you've safely
connected to the .gov website. Share sensitive
information only on official, secure websites.
As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with,
the contents by NLM or the National Institutes of Health.
Learn more:
PMC Disclaimer
|
PMC Copyright Notice
. 2000 Jun 24;320(7251):1730.
Open access follow up for inflammatory bowel disease
Would have been better to use t test than Mann-Whitney U test
Editor—Williams et al undertook a randomised trial to evaluate whether follow up of patients with inflammatory bowel disease is better with open access than with routine appointments.1 They compared primary and secondary care resource use and costs and concluded that open access follow up saves secondary care resources. This conclusion, however, is mistaken because they used inappropriate statistical methods.
Resource use and cost data tend to have highly skewed distributions. As a result, the authors decided that standard parametric statistical methods were not appropriate and assessed significance by using a Mann-Whitney U test. Although this is consistent with conventional statistical guidelines,2 it does not address the question of interest in economic evaluations. As the authors themselves state, “economic analysis is mainly concerned with a comparison of means.” Use of a Mann-Whitney U test, however, makes an overall comparison of distributions in the two groups, in terms of both shape and location,3 and does not specifically test for a difference in means.
The most appropriate simple method for comparing mean costs is the ordinary t test. By using the means and standard deviations in each group reported by the authors, we have calculated P values from t tests (table). The conclusions are dramatically different from the authors'. In particular, one of the authors' main conclusions—that open access follow up used fewer resources in secondary care—is not supported: the P value from the t test is 0.79. Other related conclusions are also misleading (table).
Although t test methods are only strictly valid for data that are normally distributed, they are fairly robust and give a reliable comparison of means, provided that skewness is not too extreme and sample sizes are moderately large.4 Using the raw data (unavailable to us), the t test results can be checked by non-parametric bootstrapping, an approach to compare means without the need for assumptions of normality.4
Use of inappropriate methods for the analysis of cost data is all too common.5 As this example shows, inappropriate analysis can lead to seriously misleading conclusions, which could influence important policy decisions in health care. Health service researchers, health economists, statisticians, and others concerned with analysis and interpretation in economic evaluations need to be aware of this important issue.
Table.
Mean (SD) costs (£) per patient in hospitals over 24 months
Open access (n=77)
Routine visit (n=78)
P value from Mann-Whitney U test (as given by authors)
4.Barber JA, Thompson SG. Analysis of cost data in randomised controlled trials: an application of the non parametric bootstrap. Stat Med (in press). [DOI] [PubMed]
5.Barber JA, Thompson SG. Analysis and interpretation of cost data in randomised controlled trials: review of published studies. BMJ. 1998;317:1195–1200. doi: 10.1136/bmj.317.7167.1195. [DOI] [PMC free article] [PubMed] [Google Scholar]
BMJ. 2000 Jun 24;320(7251):1730.
Ability of any method of follow up to detect cancer must be stated
Editor—Williams et al did not address an important clinical outcome—the risk of cancer developing—in their paper on inflammatory bowel disease.1-1 The sample size would have had to be much larger and the study conducted for much longer for it to have the power to pick up such a rare event.
Without knowing whether detection of cancer would be adversely affected by the open access policy, it would not be prudent to recommend this strategy wholeheartedly. The authors suggest that nurse practitioners should call patients regularly for assessment if necessary to reduce the risk of gastrointestinal malignancy, but they should have included the cost of hiring, training, housing, and sustaining such staff in the cost considerations.1-2 In an economic analysis it is important to consider all the costs as well as the outcomes.
We were puzzled by the statement that semistructured interviews were undertaken by general practitioners during audit visits to minimise bias. An independent blinded interviewer would eliminate or reduce bias; a general practitioner is unlikely to.
We were disappointed that sensitivity analysis was not performed as this is an important part of any cost effectiveness analysis.
As the disease specific questionnaire was not validated, drawing any conclusion from it would not be valid. Yet the table comparing quality of life (table 1 in the paper) was interesting: although the results of all results were not significant, there was a clear trend with negative numbers predominating, indicating a “better change” in the routine follow up patients compared with the open access patients. This raises the question of whether the non-significance was due to lack of power of the study to pick up a true difference when one existed.
Given these weaknesses, we do not think that an open access strategy should be recommended other than in a more powerful and longer study to answer these important issues.
References
1-1.Williams JG, Cheung WY, Russell IT, Cohen DR, Longo M, Lervy B. Open access follow up for inflammatory bowel disease: pragmatic randomised trial and cost effectiveness study. BMJ. 2000;320:544–548. doi: 10.1136/bmj.320.7234.544. . (26 February.) [DOI] [PMC free article] [PubMed] [Google Scholar]
1-2.Lockett T. Evidence-based and cost-effective medicine. Oxford: Radcliffe Medical; 1997. [Google Scholar]
Editor—We agree with Barber and Thompson that highly skewed cost data are best analysed by non-parametric bootstrapping. However, the BMJ set a deadline for papers submitted for its issue on managing chronic diseases. As our bootstrapping was not complete we followed conventional statistical guidelines and used the Mann-Whitney U test. Because economic analysis focuses on means we also reported these. As expected, our completed bootstrapping, to be published elsewhere, confirms the findings reported in the BMJ. In particular, open access greatly reduces secondary care costs.
We regret that Barber and Thompson, frustrated by our interim analysis, ignored the conventional statistical guidelines they cited by applying the t test to our data. The substantial differences between their findings and those achieved with the U test confirm that the data are highly skewed. Thus the findings of the t test are misleading.
We agree with Coomarasamy and Van Der Berg about the need to screen patients at high risk of developing malignancy. A gastrointestinal nurse practitioner could do this screening. As we do not yet have such a post it was not appropriate to include the costs in our study. We intend to evaluate this role in a further randomised trial.
General practitioners who were familiar with the principles of open access follow up but independent of the service at both hospitals undertook the semistructured interviews with study practitioners. Since the study practitioners had patients in both arms of the trial, however, the suggestion that the interviewers should have been blind is meaningless.
Although there was no disease specific quality of life questionnaire valid for the United Kingdom when the trial was designed, we have since developed and validated such a questionnaire.2-1 The trend towards a greater improvement in quality of life with routine care was balanced by a trend towards greater improvements in other variables with open access. It would indeed have required a very large sample to classify as significant differences as small as those observed. Furthermore, we judge that they are not clinically significant.
In short, we believe that our published conclusions stand in the face of the comments from the authors of these letters.
References
2-1.Cheung WY, Garratt AM, Russell IT, Williams JG. The UK IBDQ: development and validation of a British version of the inflammatory bowel disease questionnaire. J Clin Epidemiol. 2000;53:41–50. doi: 10.1016/s0895-4356(99)00152-3. [DOI] [PubMed] [Google Scholar]
