Levels of Evidence in Cosmetic Surgery: Analysis and Recommendations Using a New CLEAR Classification

Supplemental Digital Content is available in the text.

Abstract

Background:

The Level of Evidence rating was introduced in 2011 to grade the quality of publications. This system evaluates study design but does not assess several other quality indicators. This study introduces a new “Cosmetic Level of Evidence And Recommendation” (CLEAR) classification that includes additional methodological criteria and compares this new classification with the existing system.

Methods:

All rated publications in the Cosmetic Section of Plastic and Reconstructive Surgery, July 2011 through June 2013, were evaluated. The published Level of Evidence rating (1–5) and criteria relevant to study design and methodology for each study were tabulated. A new CLEAR rating was assigned to each article, including a recommendation grade (A–D). The published Level of Evidence rating (1–5) was compared with the recommendation grade determined using the CLEAR classification.

Results:

Among the 87 cosmetic articles, 48 studies (55%) were designated as level 4. Three articles were assigned a level 1, but they contained deficiencies sufficient to undermine the conclusions. The correlation between the published Level of Evidence classification (1–5) and CLEAR Grade (A–D) was weak (ρ = 0.11, not significant). Only 41 studies (48%) evaluated consecutive patients or consecutive patients meeting inclusion criteria.

Conclusions:

The CLEAR classification considers methodological factors in evaluating study reliability. A prospective study among consecutive patients meeting eligibility criteria, with a reported inclusion rate, the use of contemporaneous controls when indicated, and consideration of confounders is a realistic goal. Such measures are likely to improve study quality.

The Level of Evidence pyramid was introduced to Plastic and Reconstructive Surgery in July 2011 as a means of grading articles and encouraging a higher level of evidence in plastic surgery publications.¹ On the occasion of its second anniversary, it is timely to assess its validity.

The lack of science in plastic surgery is well recognized.^1–5 Efforts to incorporate evidence-based medicine^6,7 in plastic surgery are justified. Both the Level of Evidence⁸ and Grade² concepts originated in a seminal Canadian Task Force Report published in 1979.⁹ Evidence-based medicine challenges traditional clinical practice based on unsystematic clinical observations, basic principles, common sense, experience, and expert opinions.^7,10–12 Ironically, the Level of Evidence classification⁸ itself is a product of experience and expert opinion. Evidence-based medicine is not intended to be static but rather a dynamic, lifelong process^12,13 that recognizes the need to evolve.¹⁰ There is no grandfather clause that shields it from scientific scrutiny.¹⁴ When analyzed, medical practice guidelines often fall short in meeting methodological standards.¹⁴ About half the guidelines are outdated in 6 years.¹⁵ This study endeavors to use the components of evidence-based medicine,^4,12 including “tracking down the best evidence” and “critically appraising that evidence,” to investigate evidence-based medicine. Such a study has not been reported in the plastic surgery literature.

METHODS

A 2-year period of publications in Plastic and Reconstructive Surgery, July 2011 through June 2013, was retrospectively evaluated. All articles with a Level of Evidence rating published in the Cosmetic Section were included. Each article was designated a quality rating by the author using a new Cosmetic Level of Evidence And Recommendation (CLEAR) scale (Table 1). This classification modifies the traditional Level of Evidence ranking¹ and grade of recommendation (Table 2).^2–5 Table 3 and Figure 1 compare the classifications. Table 4 provides the study design and methodology characteristics for the first 10 articles. To conserve article space, the complete data for all 87 articles are given in Supplemental Table (Supplemental Digital Content 1, which shows the publications in cosmetic surgery, 2011–2013, and quality of evidence criteria, http://links.lww.com/PRSGO/A12). Table 5 summarizes the findings. Correlation between the assigned Level of Evidence and CLEAR Grade was tested using a Spearman rank correlation coefficient.

Table 1.

Cosmetic Level of Evidence and Recommendation

Table 2.

Grade of Recommendation

Table 3.

Comparison of Level of Evidence and CLEAR Criteria

Fig 1.

Comparison of the assigned Level of Evidence (LOE) and CLEAR Grade for 87 consecutive studies published in the Cosmetic Section of Plastic and Reconstructive Surgery from July 2011 to June 2013. Two studies were unratable by the CLEAR classification because of study error.

Table 4.

Publications and Quality of Evidence Criteria

Table 5.

Study Characteristics by CLEAR Rating

RESULTS

Forty-eight studies (55%) were designated as level 4 by Plastic and Reconstructive Surgery using its Level of Evidence rating. Three articles were assigned a level 1. Forty-one articles (48%) evaluated consecutive patients or consecutive patients subject to inclusion criteria. Thirty-five studies (40%) consisted of chart reviews and a recording of complication and reoperation rates. Twenty-five studies (29%) reported physical measurements on patients or images. An equal number of studies (29%) featured subjective evaluations of the result by the investigators. Patient-derived data were collected in 18 studies (21%). The correlation between the published Level of Evidence classification (1–5) and CLEAR Grade (A–D) was weak (ρ = 0.11, not significant).

DISCUSSION

Levels of Evidence Hierarchy

A level 1 study is often considered the “gold standard” of evidence.^10,11,16,17 A grade A recommendation is usually assigned to such studies.^4,13 A level 5 study, on the other hand, constitutes expert opinion that is often open to question. A level 2 study is a prospective comparison of treatment cohorts, a level 3 study is a retrospective case-control study, and a level 4 study is a case series.⁴ Level of Evidence categories are qualitative and nonlinear. The numbers of studies designated to each group are not normally distributed (Fig. 1).^18–21 Consequently, the numerical scores cannot be compared using common statistical techniques that assume normality.^18–20

Grade (A–D) Recommendation

The present grade classification used by Plastic and Reconstructive Surgery⁴ provides recommendations based on current knowledge irrespective of the study. A deficient study could receive an “A” grade if existing high-level studies support its conclusion. The CLEAR Grade rates the overall quality of the study itself, regardless of conventional wisdom. A low-quality study that concludes, for example, that smoking increases the complication rate may receive a low grade of recommendation, despite support in the literature. Because methodology is considered in the CLEAR numerical rating (1–5), the grade tends to be closely linked. In this study, the CLEAR level and grade always matched (2A, 3B, 4C, and 5D). The traditional Level of Evidence rating does not correlate well with the recommendation grade (ρ = 0.11, not significant) because it does not consider several important quality parameters (Table 3).

Level 1 Studies

Only 3 studies were designated level 1. The first study to be assigned a level 1 in the Cosmetic Section of Plastic and Reconstructive Surgery was titled: “A multicenter, prospective, randomized, single-blind, controlled clinical trial comparing VASER-assisted lipoplasty and suction-assisted lipoplasty.”²² This corporate-funded study’s title promises a high level of quality. Unfortunately, however, this article’s methodological deficiencies, including error in calculations,²³ make it unratable.

In their level 1 study, Costa-Ferreira et al²⁴ did not control for an important confounding variable—electrodissection.²⁵ Their study points to the value of medical experience and physiological understanding on the part of the reviewer.⁷ The third level 1 article concludes that 35% of patients undergoing cosmetic rhinoplasty suffer from body dysmorphic disorder, using an expanded and inaccurate definition of this syndrome²⁶ in addition to other methodological deficiencies.²⁷

Paradoxically, all 3 level 1 studies arrive at unreliable conclusions that encourage the reader to needlessly (1) purchase a 6-figure instrument,²² (2) compromise the esthetic result of an abdominoplasty,²⁴ and (3) deny surgery to one third of prospective cosmetic rhinoplasty patients.²⁶ These 3 level 1 studies represent just 3% of the total, equal to the percentage of level 1 studies published in 3 major plastic surgery journals from 1998 to 2007.¹¹ Their frequency does not seem to be increasing as hoped.^11,17 It is reasonable to ask whether a randomized trial (the additional descriptors, “controlled” and “prospective” are redundant) is the ideal model.

Randomized Trials and Cosmetic Surgery

Randomized trials balance both known and unknown confounders and avoid selection bias^3,18,28—at least theoretically.²⁹ In drug testing, the need to identify a true benefit from a medication, without the influence of other factors, is well-known. However, surgery is a much different discipline.^{11,18,30–33}

Unlike a pill, a procedure is not identical from patient to patient,^11,18 placebos and blinding are usually not possible, and randomization is not well accepted by patients,^11,16,28 surgeons,^16,28,33 or referral sources.³¹ Patients are particularly averse to randomization when the choice involves an operation with irreversible consequences.^16,17 Solomon and McLeod³⁴ report that most (60%) surgical questions would not be suitable for randomized trials, citing patient resistance, uncommon conditions, and lack of clinical equipoise as the most common reasons. Other shortcomings include a lack of external validity (generalizability),^3,18,28,35 the fact that surgeons are rarely equally proficient in and enthusiastic about 2 different techniques³² and cost.^18,28,35 Funding is an issue for cosmetic surgeons in practice.¹⁶ Such studies need to be cost-effective.⁴⁶ Lack of funding can lead to methodological compromises.⁴⁷ Randomized trials suffer from low inclusion rates and recruitment biases and may be underpowered.³⁵ In surgery, by the time a randomized trial is conducted, the novel procedure has often been improved.³¹ Techniques evolve quickly, particularly in plastic surgery.³² Fortunately, well-performed nonrandomized studies can still provide accurate and clinically useful information.³²

Two rigorous reviews published in the same issue of The New England Journal of Medicine in 2000 reveal that randomized trials and observational studies usually produce similar results.^48,49 Observational studies may be more consistent and less prone to reporting contradictory results.⁴⁸ Concato et al⁴⁸ conclude that research design should not be considered a rigid hierarchy. They attribute the greater homogeneity of observational studies to their broader representation of the general population.

Randomized trials are inflexible and disallow modifications that might better suit individual patients. Inadequate concealment of randomization and treatment assignments can cause serious bias that may exceed the magnitude of the treatment effect.^50–52 Bhandari et al⁴⁷ report that two thirds of randomized orthopedic trials did not use proper techniques of randomization or concealment. Reviews of randomized trials in plastic surgery uniformly report low quality.^{17,21,53–56}

Equipoise

Ethical considerations prohibit randomization of patients into 2 groups, one of which constitutes a known inferior treatment.²⁸ Different operations on contralateral sides of the face or body may produce asymmetry. Predictably, such studies tend to find no difference in treatment effects.^57,58 Cognitive dissonance may inhibit a surgeon from finding that one half of his or her randomized patients received an inferior treatment.²⁹

This discussion leads to a catch-22. If investigators compare one operation with another, they already believe one treatment is superior or they would not be conducting the study. If the difference is so slight that there is no consistent evidence one way or the other, the study is probably pointless. Fortunately, most clinical questions in plastic surgery do not concern whether a procedure is superior to nothing.⁴⁶ Therefore, shams are usually not needed.⁴⁶

Most randomized controlled trials in plastic surgery evaluate nonsurgical interventions.^59–69 Surgical trials may compare adjunctive techniques or products.^36,70–72 Such issues (eg, the use of drains) do not substantially affect the long-term result and are therefore more appropriate for a randomized model.

Limitations of Historical Controls

Studies using historical controls are predisposed to find that the newer therapy is superior to its predecessor.^51,73 Similar to randomized trials, the conclusions are usually more dependent on the method of selection of control groups than on the therapy, and the majority differ from the results of randomized trials of the same therapy.⁷³ Methodological standards are commonly violated in case-control studies.⁷⁴ Chronology bias is difficult to avoid.⁷⁵ Matched cohort groups are notoriously difficult in plastic surgery, especially cosmetic surgery.⁷⁶ Recent guidelines assign a level 4 to such studies, no better than a case series.⁷⁷ Contemporaneous controls are preferred.

If the treatment effect is dramatic (eg, breast self-consciousness after augmentation), a control group is unnecessary (eg, a control group of women not electing to have a breast augmentation). A prospective study with a dramatic effect, but no control group, can qualify as a CLEAR level 2 study if other requirements are met (Table 1).⁷⁷

Prospective vs Retrospective Study Design

The literature consistently supports the superiority of a prospective study. A prospective study is always preferred over a retrospective study if it is feasible.⁷⁸ Some investigators may question this distinction because data are always collected prospectively. The difference is the vantage point—literally looking forward vs looking backward. The outcome of a prospective study is unknown when it is undertaken, making the investigator less prejudiced. A review of a “prospective” database does not qualify because the investigator is looking back to interpret data. By definition, in a prospective study, the study is conceived before the data are collected.

Selection bias and confounders are reduced by specifying eligibility criteria, encouraging follow-up appointments, standardizing and calibrating photographs and measurements, and administering contemporaneous surveys (rather than years later). An example would be a study to determine whether patient gender affects seroma rates after body contouring surgery. A prospective study would take care to record patient weights on the same scales, preoperative weight loss, intraoperative use of electrodissection, and tissue resection weights. Some of these important details might be missing in a retrospective study. Prospective studies usually disclose more realistic complication rates than retrospective studies. Unavoidable confounders (eg, a difference in mean body mass indices) may be managed using an analysis of covariance or other statistical adjustment.⁷⁹

Markers of Success in Cosmetic Surgery

Patient satisfaction and improved quality of life,^80,81 assessed using patient-derived outcome measures, are the hallmarks of successful plastic surgery. Morbidity and mortality measures are less relevant to plastic surgery than other surgical disciplines.^28,81 Reoperation rates are unreliable markers of quality in cosmetic surgery.⁸²

Consecutive Patients

Over 2 decades ago, Goldwyn⁸³ cautioned that selectively reporting better results does nothing to advance the specialty. Nevertheless, a requirement for consecutive patients is conspicuously absent from the existing Level of Evidence rating (likely because of its nonsurgical origins). This scale does not penalize the investigator for “cherry picking” patients; nor does it reward the investigator for reporting both good and bad results. Both series receive the same catchall level 4 designation. Insisting on consecutive patients (1) sends a message to investigators to report all results and (2) prevents studies of selected patients that include higher level design characteristics from receiving undeserved higher rankings. Like a framework built on a weak foundation, no other study attribute can compensate for an unrepresentative patient sample.⁸⁴

When discussing consecutive patients, it is important to be precise. A study that reports 1-year postoperative photographic findings in 100 “consecutive patients” would be unlikely because not all patients are likely to return for photographs in 1 year; the authors more likely mean “consecutive patients returning for 1-year follow-up” and the inclusion rate should be provided. Many studies would improve from a CLEAR 5 to a CLEAR 4 ranking, or higher, simply by including consecutive patients (eg, clinical studies) or consecutive patients subject to reasonable inclusion criteria that usually include sufficient time for resolution of swelling (eg, measurement and outcome studies). A nonconsecutive case series is just a plural form of a case report and is therefore no more deserving of a higher rank. It is not difficult to report consecutive patients. Goldwyn⁸⁵ observes that “it is amazing how easy it is to be truthful if one wants to be.” Correction of this bad habit represents the single most important change to increase the overall level of evidence in plastic surgery publications. Although level 1 studies will continue to be rare, it is realistic to expect a more balanced distribution of articles between levels 2 and 5.

Statistical Power and α Level

Sample size calculation is an important part of any prospective study, whether randomized or not,^28,56,86 but is infrequently performed (3.5% of studies).^56,86 Small sample sizes predispose to type II false-negative statistical errors. Although an α level of 0.05 is the standard (ie, 5% false positives), most investigators prefer an α level of 0.01 or a Bonferroni correction to reduce the risk of type I error when multiple comparisons are made.

Eligibility Criteria

Eligibility criteria are necessary to preserve the integrity of the data, avoid confounders, and respect patient privacy (eg, for face-lift studies, a minimum follow-up time, no makeup, no additional surgery or injections, and patient consent for photographs).⁸⁴

Inclusion Rate

Every effort should be made to avoid losing patients to follow-up (ideally, <20%).⁸⁷ If the outcome of nonresponders is missing (eg, dissatisfied patients may seek follow-up elsewhere, or alternatively, satisfied patients may see no reason to return), the reliability of the conclusion is jeopardized.⁸⁷

Confounders

Most of the studies (76.5%) included extraneous factors that might correlate with the study variables. If a confounder was judged important enough to undermine the conclusion, a study was given a CLEAR level of 4, provided it still met the requirement for consecutive patients. Plastic surgeons need to take part in evaluating levels of evidence and not delegate this task.⁸⁸ There is no substitute for clinical experience and judgment in assessing a study’s validity.²

Measuring Device

The missing link in the application of the scientific method to plastic surgery is frequently a reliable measuring device.⁸⁹ Most studies feature subjective assessments or arbitrary metrics.¹¹ Direct measurements on standardized, calibrated photographs are preferred. Photographs should include at least one view accompanied by a ruler or measuring tape for calibration, avoiding the need for less intuitive devices such as ratios or pixel counts (eg, rhinoplasty). Computer-assisted photographic standardization and calibration greatly facilitate such measurements.⁸⁹

Discussion of Limitations

All studies had limitations (Supplemental Table, Supplemental Digital Content 1, which shows the publications in cosmetic surgery, 2011–2013, and quality of evidence criteria, http://links.lww.com/PRSGO/A12). However, over half (52%) did not discuss limitations. Such discussions reflect well on the investigators and improve credibility.

Commercial Bias

Corporate sponsorship affects conclusions.⁹⁰ Hall-Findlay⁹¹ expresses a concern familiar to many experienced plastic surgeons: “We listen to the manufacturer’s claims and then years later we find that we have been misled—both by the manufacturers themselves and by those surgeons who are burdened by a conflict of interest.” The willingness to resist marketing pressures and prioritize science over marketing is a sign of professionalism.⁸⁹

Systematic Reviews

A limitation of systematic reviews is that their validity depends on the quality of the reviewed material.⁹² As overall study quality improves, systematic reviews become feasible.

Clinical Relevance to Plastic Surgeon Investigators

How might these principles be put to use? A timely example might be an investigation of the effectiveness of buttock fat injection. An investigator may set out to assess the results using a measuring device (standardized photographic measurements) on consecutive patients meeting appropriate eligibility criteria (a minimum follow-up period to allow resolution of swelling), with a power analysis supporting sample size, and a comparative cohort of patients to serve as controls (eg, breast augmentation patients who agree to have their lower body photographed) with simultaneous measurement of body weights to rule out a possible confounder. These principles are highly practical and are likely to meet with a high level of patient compliance. Such a level 2 study would serve to answer an important clinical question.

Recommendations

The CLEAR classification preserves the common language of the original 5-level scale. A level 1 study remains a randomized controlled trial. The CLEAR system differs in adding important methodological considerations (Table 3). Levels 1 and 2 are considered equals, so that the top 2 levels of the pyramid may be colored just one color. Such modifications are permissible and encouraged to keep up with the latest knowledge.⁷⁷ Classifying articles as “therapeutic,” “risk,” or “diagnostic” does not affect quality assessment. One set of guidelines is simpler than 3.

Limitations of the Study

This review consists only of articles appearing in the Cosmetic Surgery section of Plastic and Reconstructive Surgery. All grade assignments were made by the author, although the opinions of the discussants were considered when available. One might consider whether a committee would make a more valid determination. The fact that all articles passed peer review would suggest that a consensus opinion is not always reliable or objective either. It is impossible to fully objectify this process.

Strengths of the Study

This analysis uses the concepts of evidence-based medicine to evaluate its own guidelines as applied to cosmetic surgery. Using the same reviewer (E.S.) for each study eliminates interobserver variation. Recommendations are made based on analysis of the data, a review of the literature, and the particular needs of this subspecialty.

CONCLUSIONS

The vestiges of an artistic perspective are evident in plastic surgery publications. Plastic surgeons need to recommit to scientific scrutiny of their results.⁸⁹ Practical improvements in study design and methodology are possible. A randomized controlled trial is unlikely to be feasible or even desirable. A prospective study among consecutive patients meeting eligibility criteria, with a reported inclusion rate, and the use of contemporaneous controls when indicated, is a realistic goal. Objective measurements and consideration of patient-derived data are most useful. With attention to such basic steps, an improvement in study quality is inevitable.

Supplemental Table. See Supplemental Table, Supplemental Digital Content 1, which shows publications in cosmetic surgery, 2011–2013, and quality of evidence criteria, http://links.lww.com/PRSGO/A12.

ACKNOWLEDGMENT

The author thanks Jane Zagorski, PhD, for statistical analyses.

Supplementary Material

gox-1-e66-s001.pdf ^{(85.6KB, pdf)}