Which outcome measures are reported by clinical trials investigating OME treatment? A case for standardised reporting

Abstract

Introduction

Many different OME treatment trials have been published using different outcomes measures to evaluate the success of particular interventions. We set out to identify the variation in reporting of outcome measures in OME trials that exists at present. This has been achieved by reviewing published trials to determine which outcome measures have been reported.

Method

The literature review was carried out using PUBMED database (1980 to 2013). Data was collected on the treatment outcomes reported, with particular focus on the methods of assessment and the number of treatment outcomes used in each study.

Results

The 171 studies identified used 12 broad treatment outcome measures. The most common outcome measure was OME resolution (48%) followed by hearing level (36%). Only 95 studies used a single outcome measure, with 76 studies using between 2 and 4 outcome measures. The method of assessment varied between studies that used the same treatment outcome measures.

Conclusion

OME treatment trials report a wide range of measures and comparison across studies is thus difficult. Establishing a core set of outcome measures to be reported by all trials in the future could be useful, and would allow comprehensive comparison of different studies and minimise potential for reporting bias.

Background

Otitis media with effusion (OME) is one of the commonest reasons why children visit their doctor and undergo surgery in the developed world.¹ In addition to its effects on the patients and their families, there is a large economic cost, associated both with treatment as well as lost days at school and work. In the USA alone, OME has been estimated to cost in the region of US$3-5 billion annually^2,3. It is therefore important that future management of the condition is informed by high-quality research. A key component of such research is the choice of the outcomes measures that are studied and reported. This is especially important in OME which has a high rate of natural resolution⁵, and thus the effect of any intervention needs to be clearly shown to be treatment-related and not just spontaneous improvement.

An intervention should have a particular goal or predicted outcome, which is operationalised in terms of a number of outcome measures. These measures can therefore be used to evaluate the effectiveness of a particular intervention. The measures vary in their power to show an effect (effect size), and power calculations are performed to determine how many participants are required to be able to reliably show an effect. When designing a clinical trial, the study team makes a choice of which outcome measures will be collected, to assess whether the treatment has reliably resulted in the desired outcome. In appropriately performed clinical trials, the study design, including the predicted outcomes, the applied outcome measures, and the numbers of participants required for sufficient power to show an effect, are established in advance, and often published at the outset. This is to avoid the temptation of reporting just those outcomes that show the authors’ own opinions or results in the most favourable light^6,8.

Clearly, the outcomes chosen and the measures used to capture them need to be relevant to the question asked. However when assessing OME treatments one can look at benefits from different perspectives, whether this is hearing level, resolution of effusion, educational outcome or improvement in quality of life, to name just a few. Many different OME treatment trials have been published focusing on different forms of treatment. However because of the wide range of different outcome measures reported for studies carried out over different time frames, it is difficult for a practicing clinician to determine the most effective form of treatment. It would be ideal if all trials reported the same outcomes at same time points; any subsequent meta-analysis would also be facilitated by this, as combining studies reporting different outcomes may not be possible^6,7. Knowing that all relevant outcome measures are obtained and reported would also assure the reader that reporting bias is not taking place. Minimising variation in the reporting of outcome measures, or establishing a core set of outcome measures which should be reported as a minimum by all studies, could help to achieve consistency and reliability of conclusions between clinical trials⁶.

The aim of this study was to review what outcome measures were used in published studies assessing treatment benefit in OME. Identifying variation in reporting of outcome measures would support the development of a core set of outcome measures for OME treatment in the future.

Methods

A literature review was carried out using PUBMED database (1980 to 2013). All original human studies with participants of any age, written in English, which reported on either surgical or non-surgical treatments for OME were identified for inclusion, focusing on the outcome measures obtained to assess the relative success of an intervention. Studies examining patients with major comorbidities, cleft palate, or Trisomy 21 were excluded, as were reviews and meta-analyses.

The broad search term of ‘otitis media’ was used to initially screen titles. Suitable abstracts were then reviewed, and subsequently full publications were analysed if appropriate for inclusion.

Data were collected on treatments given, minimum follow up period, and the treatment outcome measures reported.

Results

The initial search returned 17,158 titles, with 171 studies eventually determined to be relevant and included (Figure 1). The studies originated from 31 different countries: USA (55 studies), UK (44), Netherlands (9), Denmark (8), Japan (6), 5 each from Italy and Sweden, 4 each from Canada and Turkey, 3 each from Belgium, New Zealand and Norway, 2 each from Australia, Israel and Poland, and 1 study each from Bulgaria, Brazil, China, Finland, Germany, Greece, India, Iran, Jordan,, Malaysia, Pakistan, Slovakia, South Africa, South Korea, Spain, and Thailand. The ages of the participants included in the studies ranged from 0 to 60 years.

Figure 1.

Selection for inclusion in literature review

The 171 trials studied 31 different treatment combinations (Table 1). Among the studies, 94 (55%) reported that the treatment was successful, with 51 (30%) studies noting no effect, and 25 (15%) not clearly stating either way. The length of follow up time varied between 10 days to 14 years and was not reported in 49 (29%) studies; 48 (28%) studies documented a follow up time of between 1 and 3 months.

Table 1.

Treatment combinations used in studies. VT: Ventilation tube

Treatment category	Treatment	Number of studies
Surgical	VT insertion	74
	VT + adenoidectomy/tonsillectomy	17
	Adenoidectomy/tonsillectomy	7
	Myringotomy	5
	Myringotomy + adenoidectomy/tonsillectomy	2
Drug treatment	Antibiotics	9
	Nasal steroids	8
	Antibiotic + oral steroid	6
	Antibiotic + decongestant	5
	Mucodyne	4
	Antihistamines	3
	Antibiotic + nasal steroid	2
	Antibiotic + steroid + antifungal	2
	Antihistamine + decongestant + anti-inflammatory drug	2
	Montelukast	2
	Anti-reflux therapy	1
	Decongestant	1
	Halothane + Nitrous oxide	1
	Naproxen	1
	Nasal glutathione	1
	Intratympanic steroid	1
Surgical + drug therapy	VT insertion + antibiotics	1
Other	Autoinflation	4
	Alternative medicine	2
	Aerosol therapy	1
	Conservative treatment	1
	Elimination diet	1
	Eustachian tube rehabilitation	1
	Hearing aids	1
	Nose blowing	1

Broadly, the individual outcomes were split into outcome categories, as shown in Table 2; the table also shows details of which studies included that outcome, and the method of assessment (if known). The use of multiple outcome measures in one study was common, with 77 (45%) studies reporting more than one; 60 studies (35%) used 2 measures, 13 studies (8%) used 3, and 3 studies (2%) reported 4 measures.

Table 2.

Outcome categories. PROMs: Patient-reported outcome measures. QoL: Quality of life

Outcome categories	Number of studies	Study references (see Appendix 1)	Method of assessment (number of studies)
Hearing level	62	1,4,6,7,9,13,17,19,23,24,25,26,27,29,33,34,43,45,52,54,55,60,61,62,63,68,77,81,82,86,87,89,95,96,102,106,109,115,117,118,120,122,123, 124,127,129,133,135,140,141,143,144,149, 150,153,155,161,165,166,168,170,171	Tuning forks (1), Audiometry (absolute level, improvement or not stated) (61)
OME resolution	83	2,5,7,8,11,14,16,18,23,24,25,26,27,29,34,37, 39,43,45,46,48,49,50,51,59,60,64,71,72,73,78,81,83,86,89,93,94,95,96,98,102,103,105,109, 110,112,114,115,117,118,120,121,122,124, 125,127,128,129,130,131,132,133,134,135, 136,139,141,143,145,146,150,155,157,158, 159,160,161,162,163,166,167,169,171	Microscopy (2), Otoscopy (53), Tympanometry (53)
OME recurrence	14	13,18,41,42,78,79,83,85,88,101,103,153,154, 156	Otoscopy (3), tympanometry (1), not stated (10)
Re-treatment rates	7	6,7,22,56,122,132,151	Re-treatment (7)
Ventilation tube extrusion	11	19,40,66,84,88,92,99,101,116,119,138	Otoscopy (2), microscopy (1), not stated (8)
Tympanic membrane sequelae	37	4,6,18,19,24,28,33,35,38,39,40,41,42,71,74, 77,80,84,88,90,97,101,103,104,108,116,119, 123,129,131,140,143,148,152,162,164,165	Otoscopy (11), Not stated (26)
Adverse effects	8	6,13,15,18,24,35,38,52	Otoscopy (2), not stated (6)
Hospital stay length	1	22	Length of stay (1)
PROMs/QoL questionnaires	20	8,9,10,20,23,25,30,31,34,36,47,50,58,65,67, 70,76,107,135,147	Otitis media- 6 questionnaire (3), study-specific questionnaire (17)
Educational achievement	15	9,21,32,44,50,52,57,60,61,68,69,75,91,100, 142	Woodcock Reading Mastery Tests (1), Reynell, Schlichting, Lexi tests (1), Richman Behaviour Checklist (1), not stated (13)
Speech perception	4	10,21,44,52	Speech awareness threshold (1), not stated (3)
Presence of biofilms	1	3	Middle ear effusion sample (1)

Discussion

A total of 171 studies were included in this study aiming to determine what outcome measures are reported in trials of OME treatment. They varied in study design and size, however the data collected for this review was focused on the outcomes that they had used to evaluate the success of the specific treatment. A wide range of different OME treatment outcome measures were reported, with 12 broad treatment outcome categories identified. In addition to the wide range of outcome measures used, even when reporting each measure the methods of assessment were often varied or not explicitly stated. And in studies using quality of life questionnaires as an outcome measure, some used more recognised tools to evaluate their results than others. This emphasises the difficulty of accurately comparing studies, especially if a measure is developed in-house.

It was often not clear why certain trials had selected one particular outcome measure over another. Furthermore, some trials only documented one treatment outcome measure, whereas others (45%) used multiple. This may reflect a resource issue, weaknesses in the development of the study design or possible reporting bias where only the outcome measures indicating a successful outcome from the trial are incorporated into the published work.

The reported outcomes

Of the outcomes reported, OME resolution was used most frequently as an outcome measure. This might not be unexpected, as this would be the aim of most treatments. However there was variation in how it was assessed. The most common methods used were otoscopy and tympanometry, with all but two studies (studies 2 and 109) using them. Ten studies used just one of them to assess OME resolution; however 43 studies used both methods to assess OME resolution. It could be argued that using both methods would minimise the risk of missing an effect and therefore improve reliability of the outcome measure, especially as otoscopy can be unreliable when diagnosing OME.

OME recurrence as an outcome measure was used in 14 studies and as expected, 4 of these used otoscopy or tympanometry to assess recurrence. However the other 10 studies did not state the method of assessment. Ventilation extrusion was similar with only 3 out of the 8 studies documenting a method of assessment. It is likely that these were assessed using otoscopy, although this was not explicitly stated. Retreatment rates were used by 7 studies to establish treatment success. Although this is a composite outcome with many different factors influencing decision for repeat surgery, it is an outcome that is clinically very relevant as it directly translates to real-life practice and matters to patients.

Hearing level was the second most common outcome measurement assessed, with 36% studies using it. Audiometry was consistently used as the method of assessment in all except one study. However some studies used an absolute level and others looked for an improvement in hearing levels. One study used tuning forks to evaluate hearing level, which could be considered a much less reliable method of assessment (study 106).

Tympanic membrane sequelae was the third most common treatment outcome measurement with 37 studies using it. The conditions that were noted included tympanosclerosis and membrane retraction. Together with other complications, information on tympanic membrane sequelae is useful when assessing risks and benefits of surgery.

Many different patient-reported outcome measures and quality of life questionnaires were used in the studies evaluated. Some reported well-defined questionnaires, for example the Otitis Media-6 Questionnaire. However, others used questionnaires specifically devised for the study, which may or may not have been rigorously developed and validated; whilst specific questionnaires may be useful to assess a specific outcome, any lack of rigorous validation (or external scrutiny by means of prior publication) could risk finding an effect which might not be present⁹. A similar pattern was identified with educational outcomes, with some using well-known tests such as the Woodcock Reading Mastery Tests and others not stating exactly how the outcome was measured.

Educational outcomes, quality of life questionnaires and patient-reported outcomes measures appeared to be more likely to be used in the most recent studies, with only 3 out of the 55 studies reviewed between 1980 and 1990 using these outcome measures.

Core outcomes set

Using a core set of outcome measures when comparing different trials evaluating the effectiveness of a particular intervention has obvious benefits. It would help to achieve consistency and reliability of conclusions between clinical trials by minimising variation and reporting bias⁶. It would also enable subsequent meta-analysis to take place^6,7. Different methods have been documented for agreeing a standardised set of outcome measures including a combination of structured group discussions and the Delphi technique, with involvement of both clinical experts and patients¹⁰. The importance of developing a standardised set of outcomes measures has been widely appreciated and they are now being used more frequently in a wide range of specialties. A recent publication found that, as of December 2013, there were 51 documented on-going projects relating to the development of a core outcome sets¹⁰, while the COMET (Core Outcome Measures in Effectiveness Trials) website, accessed in November 2015, lists 705 planned, on-going and completed works¹¹.

Conclusion

OME treatment trials report a wide range of measures, with different studies choosing different outcomes often with no clear rationale for doing so or summary of how the assessment carried out. Comparison across studies is thus difficult. Establishing a core set of outcome measures to be reported by all trials in the future could be useful, and would allow comprehensive comparison of different studies and minimise potential for reporting bias.