What is missing from descriptions of treatment in trials and reviews?

Abstract

Replicating non-pharmacological treatments in practice depends on how well they have been described in research studies

Have you ever read a trial or review and wondered exactly how to carry out treatments such as a “behavioural intervention,” “salt reduction,” or “exercise programme”? Although CONSORT and related initiatives have focused on the assessment of validity and presentation of results,^{1 2} less attention has been given to the adequacy of the description of the treatment used. For pharmacological treatments the description would need to include the dose, titration, route, timing, duration, and any monitoring used. For complex treatments the problems are even greater.

Why are full descriptions of treatment important?

The uptake of positive findings from trials is often slow and sometimes negligible.³ Reasons for this slow uptake include clinicians not becoming aware of the results, perceiving the results as either invalid or not relevant to their patients, or simply not remembering to use the treatment.^{4 5} An additional barrier, which has received less attention, is clinicians’ ability to carry out the treatment on the basis of the information provided in the published reports. For example, after receiving numerous requests for additional details from doctors and patients, the author of a randomised trial on graded exercise for chronic fatigue syndrome⁶ subsequently published a supplementary article with a more detailed “prescription.”⁷ Similarly, it is not possible to set upa stroke unit, offer low fat diets, or give smoking cessation advice without sufficient details on the components that were planned and delivered.⁸

Extent of the problem

To assess the extent of problems with descriptions of treatment we prospectively assessed 80 consecutive studies selected for abstraction in the journal Evidence-Based Medicine from October 2005 to October 2006. The journal is aimed specifically at doctors working in primary care and general medicine, and it provides summaries of research that is highly relevant to clinical practice. To select studies, the staff of the journal hand search 140 or so high impact clinical journals, selecting only articles of sufficient validity and relevance to warrant changes in clinical practice.⁹ The 5% of articles that pass the validity criteria are scored for clinical relevance by active primary care (and appropriate specialty) clinicians. A dozen or so clinicians, from a pool of several thousand, score each article. The articles that were scored as most relevant to practice are then abstracted (fig 1). For each study two general practitioners (PG, CH) were independently asked whether they could use this treatment with a patient if they saw them tomorrow.

Fig 1 Selecting studies for inclusion in one year’s issues of Evidence-Based Medicine

Of the 80 published reports of treatment, 55 were single randomised trials and 25 were systematic reviews; they were published in New England Journal of Medicine (10), Cochrane Database of Systematic Reviews (9), Lancet (7), JAMA (7), Archives of Internal Medicine (6), BMJ (5), Annals of Internal Medicine (5), and several other journals (31). Most (65) were of treatments directly applicable in general practice; the remainder were relevant to general practice but were targeted at surgery (6), emergency medicine (5), internal medicine (3), and dental medicine (1). More than half (44/80) were of drug treatments. Non-drug treatments were education and training (15), devices or surgery (10), psychological treatments (4), service delivery (3), and a mix of other interventions (4).

Elements of the intervention were missing in 41 of 80 of the published descriptions. Information was better in reports of individual trials than in systematic reviews, and for drug treatments than for non-drug treatments. Information was also better for control interventions, with 58 being sufficiently well described (22 as drugs and 36 as non-drug controls); the remaining 22 were simply described as placebos.

What elements of treatment do authors miss?

The missing element was most often the description of the process, but several studies were missing handouts or booklets—for example, the name or source for a self help booklet trialled for irritable bowel syndrome. Table 1 shows the missing elements encountered and their resolutions.

Table 1.

 Some reasons for non-reproducibility for studies

Type of problem and examples	Additional information obtained
Description of method
Craniocervical training programme for tension-type headache: home schedule not provided in publication	Authors provided details of the recommended regimen of exercises at home
Comprehensive lifestyle modification on diet, weight, physical fitness, and blood pressure control	Author provided website with protocol which included details and handbooks used
ACE inhibitor in coronary artery disease: review with several combinations of drug and dose	Author recommended two similar, acceptable regimens
Systematic review of probiotics for preventing antibiotic associated diarrhoea and treating Clostridium difficile disease	Author provided suggestions on appropriate types and doses of probiotic, plus websites
Lifestyle measures effective in patients with gastroesophageal reflux disease	Review author suggested appropriate variants
Access to educational materials
Patient instructions for watchful waiting with hernia	Patient handout provided by author
Inadequate instructions for patient self-Epley procedure	Website and instructions found in a paper referenced in main paper
Name of IBS self help guidebook not given in paper or in references	Author did not reply (three attempts), but a colleague provided details of access to the guidebook via the publisher’s website
Description of equipment
BugBuster kit for eliminating head lice: needed details of combs and combing methods	Website found (via search engine) with treatment details and confirmed by author

Can authors supply the missing information?

When information on any aspect of treatment needed for clinical use was missing we tried to find a solution. We first retrieved references from the paper relevant to the treatment description, searched the internet and other bibliographic databases as required, and emailed study authors (up to three times). We gained some missing information for most study reports; 52 of 59 authors replied. By this simple measure the completeness of treatment descriptions improved from 49% to 76%, and improved similarly across all categories (fig 2).

Fig 2 Percentage of studies with sufficient description of treatment initially (based only on the published paper) and after supplementary information was obtained.

Some authors were extremely helpful in providing additional information. Some volunteered manuals and videos that would form part of the treatment; they were often willing to allow free access to such resources via the internet but had not included such a suggestion in the original paper. For example, one study of “cognitive therapy” had been specifically adapted and tailored for schizophrenia; this required specialised training for which an author was willing to provide considerably more detail than had been published in the journal: “[I] would be delighted for you to include the manual and information booklets in downloadable format . . . it is really important that they are available in the public domain.” Others were less able to help: for a software program to guide nurses in telephone counselling, the authors stated that it was available but not for public use—but this had not been indicated in the publication. Others had concerns about providing sufficient details because of the skill level needed to carry out the treatment. For example, the responding author of a smoking cessation worksheet said: “[The authors of the worksheet] have asked that we do not post this online. The rationale is that people really have to be thoroughly trained in terms of how to use the form.” No details on how to obtain such training had been described in the paper.

Despite our attempts to get further information, elements of the treatment were still missing in 19 studies. Some examples of these are detailed in table 2.

Table 2.

 Problems with reproducing the study’s treatment after attempts to obtain further information

Examples of problems	Reason problem was not resolved
Description of treatment regimen
Telephone care over primary care for smoking cessation	Obtained sufficient description from two referenced papers, but author did not make counselling protocol available because of the training needed to use it correctly
Anticholinergics in acute asthma: review with multiple regimens	Author unwilling to indicate which specific regimen(s) were acceptable
Behavioural treatments for insomnia: review with multiple regimens	Author suggested a book, but not clear if it is based on any of the trials
Cardiac rehabilitation programmes: review with multiple regimens	Author unwilling to indicate which specific regimen(s) were acceptable
Low fat dietary pattern and weight change	Treatment not well described in paper; but a referenced book makes clear that the treatment involves 18 sessions with clear structure
Insufficient description of cognitive behavioural treatment by mental health nurses in schizophrenia	Author helpfully supplied details of training and manuals used, but we felt we required more training to use these
Telephone counselling by a pharmacist for patients receiving polypharmacy	Author promised details but never followed through
Language barrier
Patients with chronic heart failure: educational booklet unavailable	Educational booklet is in Spanish only (language not mentioned in paper)
Pamphlets and field training materials for handwashing to prevent diarrhoea and acute respiratory infections.	Pamphlets and training materials are in Urdu (not mentioned in paper); and no translation available
Equipment
Software for recording data from nurses’ consultations	Software is proprietary (not mentioned in paper)

For systematic reviews, several authors were helpful in providing details and appropriate choices, but many were reluctant to suggest which version(s) of the treatment might be appropriate to use in practice. Many believed that the review already had sufficient information: “All details and recommendations can be extracted from the meta-analysis” and “The references for these studies are found in the article”—but they did not say which treatment should be implemented. One author narrowed the range later, saying “cognitive behavioral therapy was found to have uniform benefits” but did not indicate the type and quantity needed. One author thought it was simply not possible to specify a treatment: “No regimen can or should be based on a single trial, even a systematic review.” Authors understood the need for a specific regimen(s) to be selected but were reluctant to choose one. For example, one suggested a regimen but did not want it publicised: “My hospital’s regimen . . . is definitely not something I would recommend in writing, as it is a local protocol.” Others believed selecting a regimen was not appropriate: one author stated: “It would not be valid to propose one type of planned management over another based solely on our review.”

Improving reporting

While some of the missing detail is intrinsic to the complexity of certain treatments, providing some additional and readily available information would allow a greater use of published research in clinical practice. Many authors were willing to provide extra details and materials so that their treatment could be used, but clearly this had not been required by most journals.

So how might reporting be improved? For trials, the CONSORT statement asks for “Precise details of the treatments intended for each group and how and when they were actually administered,”¹⁰ but further guidance would be helpful. A more detailed checklist about the “who, what, when, and where” of the treatment is desirable but may need to be tailored to different types of interventions. For non-pharmacological treatments the details are often complex, and a recent extension to CONSORT (available at www.consort-statement.org/index.aspx?o=1068) requests extra details on the components, the procedure, including tailoring to individuals, standardisation, and adherence.

Full descriptions of treatment should include any procedures used; the timing of treatment, including duration and intervals of dosing or sessions; any materials needed (such as patient handouts or devices); and accessibility of any materials or instructions, including overcoming language barriers. This may need to be supplemented with graphical methods for depicting the flow and timing of sessions of treatment¹¹ and copies of materials or handouts used. Electronic publishing provides the ideal format for distributingsuch extra material.¹² For systematic reviews, statistical and clinical approaches need to be integrated to help reviewers select the most appropriate treatment(s) from among those included in the review.

In systematic reviews, the high level of abstraction used in selecting “similar” treatments causes a problem. Even when reviews and trials include enough details of the treatment, the clinical reader may find it difficult to select the version to adopt in practice, especially if there is heterogeneity of effect among the treatments. Several formal and informal options can be used to decide the appropriate treatment in practice. For example, selecting the treatment with the largest apparent benefit seems sensible but has several dangers. In particular, a small trial may have a large effect by chance. The treatment used in the largest trial is not necessarily the most effective one. The QUOROM statement asks reviewers to report “details of intervention” of trials,¹³ and the Cochrane Handbook provides more on how to describe complex interventions (Section 7.3.4) and their fidelity,¹⁴ but neither currently gives guidance on how to select and report the appropriate versions of the treatments reviewed.

Access to such complete descriptions should not be restricted. Development of a repository of treatment descriptions, particularly for non-drug treatments, would help clinicians and be supported by many authors. An example of such a process is the Centers for Disease Control and Prevention’s Replicating Effective Programs (www.cdc.gov/hiv/projects/rep), which aims to provide detailed packages and resources in HIV prevention. Without such descriptions and programmes, tens of millions of pounds of research effort could be wasted each year because effective treatments can’t be implemented or will lack fidelity when applied.

Summary points

• To use treatments tested in trials, clinicians need sufficient details of the “how to”

• Many current trials and reviews often omit crucial details of treatments

• Providing some additional treatment details could improve the uptake of trial results in clinical practice