Quantity and quality assessment of randomized controlled trials on orthodontic practice in PubMed

Abstract

Objective:

To find current high-quality evidence for orthodontic practice within a reasonable time, we tested the performance of a PubMed search.

Materials and Methods:

PubMed was searched using publication type randomized controlled trial and medical subject heading term “orthodontics” for articles published between 2003 and 2007. The PubMed search results were compared with those from a hand search of four orthodontic journals to determine the sensitivity of PubMed search. We evaluated the precision of the PubMed search result and assessed the quality of individual randomized controlled trials using the Jadad scale.

Results:

Sensitivity and precision were 97.46% and 58.12%, respectively. In PubMed, of the 277 articles retrieved, 161 (58.12%) were randomized controlled trials on orthodontic practice, and 115 of the 161 articles (71.42%) were published in four orthodontic journals: American Journal of Orthodontics and Dentofacial Orthopedics, The Angle Orthodontist, the European Journal of Orthodontics, and the Journal of Orthodontics. Assessment by the Jadad scale revealed 60 high-quality randomized controlled trials on orthodontic practice, of which 45 (75%) were published in these four journals.

Conclusion:

PubMed is a highly desirable search engine for evidence-based orthodontic practice. To stay current and get high-quality evidence, it is reasonable to look through four orthodontic journals: American Journal of Orthodontics and Dentofacial Orthopedics, The Angle Orthodontist, the European Journal of Orthodontics, and the Journal of Orthodontics.

INTRODUCTION

Evidence-based medicine is defined as the conscientious, explicit, and judicious use of the best current evidence in making decisions about the care of individual patients.1 Since the phrase “evidence-based medicine” appeared in the ACP Journal Club for the first time in 1991,2 evidence-based medicine has been widely incorporated in clinical practice all over the world.3–5 In this context, the demand for evidence of orthodontic practice has recently risen. Peck6 stated “We orthodontists have an obligation to be clinical scientists providing the best evidence-based service.” Ackerman7 also stated, “The challenge facing orthodontists in the 21st century is the need to integrate the accrued scientific evidence into clinical orthodontic practice.”

Results of high-quality randomized controlled trials (RCTs) are considered to be strong evidence in evidence-based orthodontic practice8–11 and are recognized as the gold standard for providing clinical research evidence.8 To retrieve medical and dental articles, the PubMed version of the Medline database is widely used, because it is the largest single database for biomedical references, indexing abstracts from about 4,000 journals worldwide free of charge.12–14 Therefore, it is convenient to utilize PubMed to search for RCTs on orthodontic practice. Sjögren and Halling15 stated that the medical subject headings (MeSH terms) and publication type RCT were valid in PubMed searches for RCTs in dental research. However, some researchers pointed out problems with the Medline search, for example, inherent software and operator limitations,12 difficulty in indexing articles and misclassification of articles,13 and change in database content and indexing practices over time.16 At the same time, to offer the best evidence-based service, it is important not only to find the articles but also to assess the quality of the articles. However, most of the previous studies searching for dental RCTs on Medline focused on the quantity of articles.17–19

We consider that the most important goal for performing evidence-based orthodontic practice is to find current high-quality evidence within a short time. We investigated whether highly sensitive PubMed search strategies15,16,20 were appropriate for a search of RCTs on orthodontic practice. We also assessed all retrieved articles using the Jadad scale.21

MATERIALS AND METHODS

Medline Searches

To identify RCTs on orthodontic practice with high probability, we searched PubMed based on approaches used in earlier studies.15,16,20 The Medline database (Entrez PubMed, www.ncbi.nlm.nih.gov) was used to search for RCTs on orthodontic practice (April 2008) using the MeSH term “orthodontics” and limiting publication type to “randomized controlled trial,” publication date to “from 2003/1/1 to 2007/12/31,” language “English,” and subjects to “humans.”

Comparison of the Results from PubMed Search with Those from a Hand Search of Four Orthodontic Journals

Because 70% of the RCTs on orthodontic practice in the PubMed search were found in only four orthodontic journals, we defined the result of the hand search for RCTs on orthodontic practice in these four journals as the gold standard. This result was used for calculating sensitivity.

Discrimination Between RCT or non-RCT

We chose articles that had the words, “randomly,” “random.” or “randomization” in the title and abstract.21 Initial screening of articles was performed independently by two of the authors. When disagreement in the classification of articles about RCTs occurred, the article was reread and discussed until a consensus was obtained.

Discrimination of Orthodontic Articles

The retrieved articles were scrutinized for their relevance to orthodontic practice. Articles covering other dental areas were excluded. We appraised all the retrieved articles and discriminated the orthodontic articles as follows.

Orthodontic article

• Including a related word; for example, orthodontics, orthopedics, fixed appliance, bracket, or orthognathic surgery.

• Judged as an orthodontic clinical article in terms of article content.

Nonorthodontic article

• Unable to judge as an orthodontic clinical article in terms of article content.

• Relating only to temporomandibular disorder, sleep apnea syndrome, and bruxism and not to malocclusion.

• Not using an orthodontic appliance.

Quality Assessment of RCTs Using the Jadad Scale

The Jadad scale includes three items related to the validity of RCTs. Perfect scores for each item are 2, 2, and 1 for random allocation, double-blinding, and withdrawals or dropouts, respectively. The total score is 0 to 5 points. A score of less than or equal to 2 points is evaluated as poor quality and a score of 3 to 5 points is evaluated as high quality.22 When there was disagreement regarding the scores of the three items, we referred to the results of earlier studies23,24 to make a final decision.

Sensitivity and Precision of PubMed Search

Sensitivity was evaluated in comparison with the results of a hand search (gold standard) of four orthodontic journals. Sensitivity was defined as the number of RCTs on orthodontic practice in the four orthodontic journals on PubMed divided by the number of RCTs on orthodontic practice in the four orthodontic journals by hand search. Precision was defined as the number of true RCTs on orthodontic practice divided by the number of records retrieved.16

RESULTS

PubMed Searches

As shown in Table 1, 277 hits for RCTs on orthodontic practice were obtained in the PubMed search. There were 230 (83.03%) orthodontic articles, and of these, 201 (72.56%) were RCTs. Upon further examination, 161 (58.12%) of these were true RCTs on orthodontic practice. Thus, the precision was 58.12%.

Table 1.

Results of PubMed Search

Number of RCTs on Orthodontic Practice According to the Journal

As shown in Table 2, the 161 RCTs on orthodontic practice were listed in 30 journals. Of these, 115 articles (71.43%) were listed in four orthodontic journals. About half (n  =  57) of these articles (49.57%) were found in the American Journal of Orthodontics and Dentofacial Orthopedics (AJO-DO), 23 (20.00%) in The Angle Orthodontist (AO), 22 (19.13%) in the European Journal of Orthodontics (EJO), and 13 (11.30%) in the Journal of Orthodontics (JO). The remaining 46 articles (28.57%) were published in 26 other journals.

Table 2.

Number and Percentage of Randomized Controlled Trials on Orthodontic Practice According to Journal

Hand Search of the Top Four Orthodontic Journals

We hand searched all of the articles in the four journals, each of which had more than 10 articles. Besides the articles searched in PubMed, only three additional RCTs were found in the hand search; two were from AJO-DO and one was from JO.

We set the results from the hand search of the top four orthodontic journals as the gold standard. The sensitivity for PubMed search was 97.46%.

Jadad Scale of RCTs of Orthodontic Practice

As shown in Table 3, the mean Jadad scale score was 2.17, which was between poor quality (0–2 points) and high quality (3–5 points). The mean score of randomization was 1.41. Randomization with adequate methods was described in 72 (44.72%) articles. Eighty-three (51.55%) articles did not mention any randomization methods. In six (3.73%) articles, the methods described were inadequate.

Table 3.

Number and Percentage (within parentheses) of Randomized Controlled Trials (n  =  161) on Orthodontic Practice Searched by PubMed that Reported Randomization, Double-blinding, Withdrawals or Dropouts and Jadad Scale Score

The mean score of double-blinding was 0.16. Only 12 (7.45%) articles showed adequate methods. Two articles (1.24%) contained description of the double-blinding but no description of the methods, and 147 articles (91.30%) contained no description of the blinding technique.

The mean score of withdrawals was 0.60. The adequately reported number of and reason for withdrawals were shown in 97 (60.25%) articles. The other 64 (39.75%) articles reported only the number of withdrawals and contained no description of number or reason.

The numbers of publications of high- and poor-quality RCTs according to journal are indicated in Tables 4 and 5. The high-quality RCTs were not limited to any specific journal. The total number of high-quality RCTs was 60; 45 (75%) of these were published in the top four journals. There were 101 poor-quality RCTs; 70 (69.31%) of these were published in the top four journals.

Table 4.

Number of High-Quality Randomized Controlled Trials on Orthodontic Practice According to Journal

Table 5.

Number of Poor-Quality Randomized Controlled Trials on Orthodontic Practice According to Journal

DISCUSSION

For evidence-based orthodontic practice, the latest high-quality evidence is essential. To find a simple and convenient method to retrieve RCTs for orthodontic practice, we tested the performance of PubMed search with appropriate strategies. We performed the PubMed search with the MeSH term “orthodontics,” using the limit criterion to confine the results to RCTs. According to a past report,17 we also used the terms “humans” and “English” in the PubMed search to limit the results. Sensitivity and precision were assessed as an index of validity. For evidence-based decision making, two points are important. The first is to get the latest high-quality evidence, and the second is to get such evidence in a matter of minutes. For the former, high sensitivity is needed. If the sensitivity is low, it will be difficult to identify evidence that was not retrieved. This is a great concern when making evidence-based decisions in orthodontic practice. Because the sensitivity was 97.46%, extremely few RCTs were excluded. For the latter, high precision is needed. If the precision is low, one more step will be needed to appraise the retrieved information manually, therefore prolonging the time needed to get appropriate information. Because the precision was 58.12%, this search technique might retrieve many unrelated articles. From the aforementioned validity, this PubMed search strategy is valid for retrieving RCTs on orthodontic practice if one is careful about definitely appraising the retrieved information.

Past investigations of this strategy showed a high probability in medical and orthodontic research. In the medical field, Glanville et al.16 reported a high level of sensitivity (82.78%) and precision (100%) for the Medline search using the MeSH term and “randomized controlled trial.” Our result showed that it was highly sensitive (97.46%), while retaining reasonable levels of precision (58.12%). Sjögren and Halling15 showed that PubMed search had high validity in endodontics and orthodontics and low validity in pediatric dentistry and oral surgery. The reason for difference between our results and a past investigation16 may be the variation of validity for MeSH terms in different disciplines.

In addition to sensitivity, evaluation of quality is also important. We evaluated the retrieved articles using the Jadad scale. The results indicated that 75% of the high-quality articles were published in four journals: AJO-DO, AO, EJO, and JO. We identified these journals as the four major orthodontic journals for evidence-based orthodontic practice.

Our results showed a low percentage (7.45%) for double-blind description compared with withdrawals or dropouts (60.25%). All articles were described as being randomized in our investigation, because we investigated only RCTs. In the past, almost the same result was reported in a study that evaluated RCTs and controlled clinical trials of orthodontic practice using the Jadad scale.25,26 The low percentage of double-blind description may indicate inherent characteristics of the study for comparison of different orthodontic treatments. The Jadad scale is the only scale for evaluation of RCTs, and its validity has been proved. Jadad and colleagues developed this scale for the evaluation of pain treatment and indicated its capability of universal application for any medical area.21

Because removing systematic error (bias) is important in clinical trials, a strong study design is desired. Therefore, randomization, double-blinding, and withdrawals or dropouts are emphasized in general. In this investigation only eight articles were double-blinded with two points of perfect scores in the top four journals.27–34 All studies evaluated the effect of drugs, and no study was concerned about the effect of orthodontic appliances. This fact may indicate that RCT is a supreme study design for evaluating the effect of drugs that can be performed double-blinded by active drug and placebo. However, it may be inappropriate to use RCTs for estimating the orthodontic treatment outcome because it is impossible to perform double-blinding in such a clinical trial. Rinchuse et al.8 stated that results of research studies without placebo or sham procedures might be questionable because of the influence of various psychological factors. However, he did not suggest any alternative methods if blinding was impossible. Double-blinded orthodontic clinical trials are yet to be performed.

To conclude, RCTs on orthodontic practice are easily and reliably accessible in a few minutes free of charge. To gather orthodontics-related, high-quality, evidence-based information, we should hand search the four orthodontic journals AJO-DO, AO, EJO, and JO, on a daily basis.

CONCLUSIONS

• The PubMed search strategy showed nearly 100% sensitivity and 60% precision. This would be useful for evidence-based orthodontic practice.

• We showed that more than 70% of the RCTs retrieved by PubMed were localized in four journals, and 75% of RCTs were also localized even when restricted to high-quality RCTs. Therefore, we defined AJO-DO, AO, EJO, and JO as the four major journals for evidence-based orthodontic practice. If one looks through these four journals on a daily basis, one can gather plenty of high-quality orthodontic information for evidence-based decision making.

• Only eight articles showed a perfect score for double-blinding among the 115 RCTs on orthodontic practice published in the top four journals. Double-blinding is impossible when evaluating the effect of an orthodontic appliance in principle. Quality evaluation of an orthodontic clinical trial, especially double-blinding, will be an issue in the future.