Antimicrobial Agents for Complicated Skin and Skin-Structure Infections: Justification of Noninferiority Margins in the Absence of Placebo-Controlled Trials

Abstract

Background

The United States Food and Drug Administration requires clinical trial noninferiority margins to preserve a fraction (eg, 50%) of the established comparator drug's efficacy versus placebo. Lack of placebo-controlled trials for many infections complicates noninferiority margin justification for and, hence, regulatory review of new antimicrobial agents. Noninferiority margin clarification is critical to enable new antimicrobial development. In the absence of placebo-controlled trials, we sought to define the magnitude of efficacy of antimicrobial agents and resulting noninferiority margins for studies of complicated skin and skin-structure infection (SSSI).

Methods

We systematically reviewed literature on complicated SSSI published during 1900–1950 (before widespread penicillin resistance) to define treatment outcomes and confidence intervals (CIs). Antimicrobial efficacy was calculated as the lower limit CI of the cure rate with antimicrobials minus the upper limit CI of the cure rate without antimicrobials.

Results

We identified 90 articles describing >28,000 patients with complicated SSSI. For cellulitis/erysipelas, cure rates were 66% (95% CI, 64%–68%) without antibiotics and 98% (95% CI, 96%–99%) for penicillin-treated patients, and penicillin reduced mortality by 10%. Cure rates for wound/ulcer infections were 36% (95% CI, 32%–39%) without antibiotics and 83% (95% CI, 81%–85%) for penicillin-treated patients. For major abscesses, cure rates were 76% (95% CI, 71%–80%) without antibiotics and 96% (95% CI, 94%–98%) for penicillin-treated patients; penicillin reduced mortality by 6%.

Conclusion

Systematic review of historical literature enables rational noninferiority margin justification in the absence of placebo-controlled trials and may facilitate regulatory review of noninferiority trials. Noninferiority margins of 14% for cellulitis/erysipelas, 21% for wound/ulcer infections, and 7% for major abscesses would preserve ≥50% of antibiotic efficacy versus placebo for these complicated SSSI subsets.

Over the past several decades, increasing antimicrobial resistance has driven a critical need to develop new antimicrobial agents [1]. However, recent controversy over acceptable margins of noninferiority for registrational clinical trials has served as a major impediment to successful development of new antibiotics [2, 3]. As for many infectious diseases, selection of an appropriate noninferiority margin is problematic for clinical trials of complicated skin and skin-structure infection (SSSI), because antimicrobials became available in an era prior to randomized, placebo-controlled trials. Complicated SSSIs are among the most common medical conditions in the United States (US) and throughout the world and impart substantial patient morbidity and cost to the US and global health care systems [4–11]. Furthermore, the spread of community-acquired methicillin-resistant Staphylococcus aureus has provided a major impetus to develop new antimicrobial agents for these infections [9, 11–20].

As an example of rational noninferiority margin justification in a manner compliant with US Food and Drug Administration (FDA) [21] and International Congress on Harmonization (ICH) E9 and E10 guidances [22, 23], we sought to define appropriate noninferiority margins for clinical trials of antimicrobial agents in the treatment of complicated SSSI. As a basis for margin justification, we conducted a systematic review of historical literature to determine the effect size of a “gold standard” antimicrobial agent relative to no active therapy against these infections.

Methods

Systematic review

We conducted a systematic review of the peer-reviewed literature on skin infection in the preantibiotic and immediate postantibiotic era (1900–1950). To increase the relevance of our search to modern clinical studies (which exclude enrollment of patients infected with bacteria resistant to the comparator drug), we focused the literature search on a period prior to the dissemination of β-lactamase–mediated resistance to penicillin in both hospital inpatient and community settings in the 1950s [24–29].

PubMed, ScienceDirect, and Google were searched for English language articles published during 1900–1950 using the search combinations “cellulitis OR erysipelas,” “wound AND infection,” or “abscess OR carbuncle.” In addition, references to other SSSI studies found in identified articles were reviewed.

Definitions and statistics

Definitions of clinical cure or failure were adapted from each manuscript on the basis of the criteria available. Objective criteria used to indicate failure included death, septic complications, worsening of infection after initiation of therapy, persistence of lesions after completion of therapy or for ≥28 days while receiving therapy, relapse or recurrence of infection after termination of therapy, failure to heal wounds or wound dehiscence, failure of skin grafts, or amputation.

Skin infections were divided into 1 of 3 major complicated SSSI categories: (1) cellulitis/erysipelas; (2) infection of trauma, surgical, or combat wounds or ulcers; and (3) major abscesses. Patients with a furuncle (ie, an uncomplicated abscess [30]) were excluded from the analysis whenever they were described separately from patients with a major abscess or carbuncle.

Treatments were divided into 3 categories: (1) no active antimicrobial therapy, (2) sulfonamide therapy, and (3) penicillin therapy. Weighted averages of successful treatment were calculated across studies. Confidence intervals (CIs) were calculated using standard linear combination variance formulas [31]. This method allowed inclusion of 1-armed studies and non-randomized 2-armed studies, which is not possible with meta-analytic techniques [32]. Antimicrobial efficacy was conservatively defined as the lower limit of the 95% CI of the cure rate with antimicrobial therapy minus the upper limit of the 95% CI of the cure rate with no antimicrobial therapy.

Egger's test for publication bias and the χ² Q test for heterogeneity were calculated using Comprehensive Meta-Analysis Software (Biostat). A χ² or Fisher's Exact test was used to compare proportions of cure or mortality. A 2-tailed P value ≤.05 was considered to be significant.

Results

Literature summary

Ninety peer-reviewed publications from 1900 through 1950 were included in the analysis, describing cure or mortality rates for >28,000 patients with a complicated SSSI. Additional studies were excluded because specific cure rates could not be determined [33–36]. Studies of chlortetracycline or streptomycin treatment of complicated SSSIs were also excluded [37–39], as were studies of erysipeloid (ie, cellulitis caused by Erysipelothrix rhusiopathiae) [40, 41].

Two additional studies reported population-based mortality rates from erysipelas spanning the pre- and postantibiotic eras (figure 1). In the first study, Hoyne et al. [42] described the mortality among patients with erysipelas treated at Cook County Hospital during 1929–1938. In another study, Madsen [43] reported the population-based mortality rates for erysipelas over a 90-year period with use of a national database in Norway. The magnitude of mortality reduction reported immediately after the availability of first sulfonamides and then penicillin was dramatic (figure 1).

Figure 1.

Mortality rates associated with erysipelas before and after the introduction of antimicrobial agents. A, Mortality rates of erysipelas at Cook County Hospital in Chicago, Illinois, from 1929–1938. Sulfonamides (sulfa) became generally available in 1936. Adapted from Hoyne et al. [42] with permission. B, Mortality of erysipelas from a national registry in Norway. Sulfonamides (sulfa) became generally available between 1936 and 1937. Penicillin (PCN) was first used in patients in 1941 but did not become generally available for civilian use until after World War II, between 1946 and 1947. Adapted from Madsen [43] with permission.

Cure rates for cellulitis/erysipelas

Thirty-seven studies reported cure rates for cellulitis/erysipelas for children and adults (table 1). Concordant with modern experience, β-hemolytic streptococci were the predominant organisms identified, with S. aureus being the second most common (table 1) [44].

Table 1. Clinical cure rates for cellulitis/erysipelas.

	Treatment
Outcome	Othera	Sulfonamide	Penicillin
Overall, proportion of patients cured	1520/2294	1423/1573	196/200
Percentage of patients cured (95% CI)	66 (64–68)	91 (89–92)	98 (96–99)
Effect size, % (95% CI)	…	24 (21–28)	32 (28–36)
Lower limit of effect size,b %	…	…	28

Note. CI, confidence interval. This table is available in its entirety in the online version of the journal.

^a“Other” refers to nonantimicrobial therapies, including topical creams (eg, magnesium sulfate, glycerin, etc.), blood transfusion, injection of anti-streptococcal serum into lesions, X-ray or ultraviolet therapy, or bacteriophage therapy.

^bLower limit of effect size is calculated by subtracting the upper bound of the 95% CI of cure with no antibiotic therapy (68%) from the lower bound of the 95% CI of cure with penicillin (96%).

No difference in cure rates was found for topical creams or ointments, ultraviolet radiation therapy, X-ray therapy, active vaccination, antitoxin serum, bacteriophage therapy, or injection of autologous blood (data not shown). Therefore, all non-antimicrobial treatments were grouped as “other” treatments for analysis (table 1). The average cure rate for cellulitis/erysipelas was 66% for non–antimicrobial-treated patients, 91% for systemic sulfonamide-treated patients, and 98% for systemic penicillin-treated patients. The lower limit of efficacy for systemic penicillin was 28% (table 1).

Two additional studies reported the efficacy of topical or local penicillin treatment for a combined total of 37 patients with cellulitis/erysipelas [45, 46]. Topical or local penicillin treatment was less effective than was systemic penicillin, resulting in a cure rate of 89% (95% CI, 80%–98%).

Significant heterogeneity was detected in studies reporting cure rates for no antimicrobial therapy or a sulfonamide treatment (P < .001 for both, by Q test). However, no heterogeneity was detected in studies reporting penicillin cure rates (P = .9). Heterogeneity was largely accounted for by differences in the factors used by the studies to define treatment failure. Specifically, studies tended to report higher cure rates when they considered fewer factors in the definition of treatment failure. In contrast, studies reported lower cure rates when they considered more factors in the definition of treatment failure. Publication bias was not detected by Egger's test for any of the groups (P = .6, P = .1, and P = .4 for no antimicrobial, sulfonamide, and penicillin treatment, respectively).

Cure rates for wound/ulcer infections

Twenty-three studies reported outcomes of traumatic, surgical, or combat wound or ulcer cutaneous infections (table 2). Again, consistent with modern experience, S. aureus was the most common pathogen isolated, with streptococci (predominantly β-hemolytic streptococci) being the second most common (table 2). Several studies also reported culturing Enterococcus species, gram-negative rods, and/or anaerobes from polymicrobial wound infections.

Table 2. Clinical cure rates for trauma, surgery, or combat wounds or ulcer infections.

	Treatment
Outcome	Othera	Sulfonamide	Penicillin
Overall, proportion of patients cured	160/449	385/531	1476/1775
Percentage of patients cured (95% CI)	36 (32–39)	73 (70–76)	83 (81–85)
Effect size, % (95% CI)	…	37 (30–43)	48 (42–53)
Lower limit of effect size,b %	…	…	42

Note. CI, confidence interval. This table is available in its entirety in the online version of the journal.

^a“Other” refers to nonantimicrobial therapies, including topical creams (eg, magnesium sulfate, glycerin, etc.), blood transfusion, injection of anti-streptococcal serum into lesions, X-ray or ultraviolet therapy, or bacteriophage therapy.

^bLower limit of effect size is calculated by subtracting the upper bound of the 95% CI of cure with no antibiotic therapy (68%) from the lower bound of the 95% CI of cure with penicillin (96%).

The cure rate was 36%, 73%, or 83% for patients treated with no antimicrobial, a sulfonamide, or penicillin, respectively. The lower CI limit of penicillin efficacy compared with no antimicrobial therapy was 42% (table 2). Significant heterogeneity in cure rates was detected for all 3 treatment groups (P < .001, by Q test) because of the diverse types of wounds and the mixture of studies using topical or local versus systemic routes of antimicrobial administration. Publication bias was not detected by Egger's test for any of the groups (P = .1, P = .9, and P = .1 for no antimicrobial, sulfonamide, and penicillin treatment, respectively).

Seven studies were identified in which 109 patients were clearly described to have received only systemic, and not topical or local, penicillin treatment of wound or ulcer infections [47–53]. The cure rate in these studies of systemic penicillin therapy was 89% (95% CI, 83%–95%). There was no significant heterogeneity in these studies (P = .5) and no evidence of publication bias (P = .3).

Cure rates for major abscesses

Thirty-four studies reported cure rates for children and adults with a major abscess (table 3). S. aureus was by far the most common etiologic organism identified, with streptococci identified much less often (table 3). Abscesses (mostly carbuncles) in the historical literature were typically complicated and accompanied by fever and other systemic signs of illness. Furuncles constituted <10% of the analyzed cases.

Table 3. Clinical cure rates for major abscesses.

	Treatment
Outcome	Othera	Sulfonamide	Penicillin
Overall, proportion of patients cured	254/336	60/69	282/293
Percentage of patients cured (95% CI)	76 (71–80)	87 (80–94)	96% (94–98)
Effect size, % (95% CI)	…	11 (0–23)	21 (14–27)
Lower limit of effect size,b %	…	…	14

Note. CI, confidence interval. This table is available in its entirety in the online version of the journal.

^a“Other” refers to nonantimicrobial therapies, including topical creams (eg, magnesium sulfate, glycerin, etc.), blood transfusion, injection of anti-streptococcal serum into lesions, X-ray or ultraviolet therapy, or bacteriophage therapy.

^bLower limit of effect size is calculated by subtracting the upper bound of the 95% CI of cure with no antibiotic therapy (68%) from the lower bound of the 95% CI of cure with penicillin (96%).

The cure rate for major abscesses was 76% with no antimicrobial treatment, 87% with sulfonamide treatment, and 96% with systemic penicillin treatment. The lower limit of systemic penicillin efficacy compared with no active antimicrobial was 14% (table 3). When studies that included any furuncles were excluded from the analysis, the cure rates were changed marginally (77%; [95% CI, 72%–81%] for no antimicrobial treatment vs. 97% [95% CI, 95%–100%] for penicillin treatment). In contrast, in 3 studies including 69 patients primarily treated with local or topical penicillin rather than systemic therapy [45–47], the cure rate (84% [95% CI, 76%–93%]) was not significantly different from placebo.

Excluding studies in which surgical management was not clearly described [47, 52, 54–58], surgical intervention was significantly more common among patients treated without antimicrobial therapy than among those treated with penicillin (98 [37%] of 268 patients vs. 18 [16%] of 112 patients; P < .001). Furthermore, cure rates were not significantly different with or without surgery among patients treated without an antimicrobial agent (75 [77%] of 98 patients vs. 131 [77%] of 180 patients; P > .99) or among those treated with penicillin (18 [100%] of 18 patients vs. 93 [99%] of 94 patients; P = .8).

Significant heterogeneity in cure rates was detected in studies of no antimicrobial therapy (P < .001, by Q test) but not in studies of penicillin treatment (P = .8). Publication bias was not detected (P = .3 and P = .4 for no antimicrobial or penicillin treatment, respectively).

Mortality rates for skin infections

Ultraviolet therapy, sulfonamides, and penicillin each significantly reduced the mortality rate of cellulitis/erysipelas compared with other nonantimicrobial treatments (table 4, P < .001 for all 3 comparisons). Sulfonamide and penicillin treatment were also more effective than ultraviolet therapy (P < .001 for both comparisons), and penicillin was more effective than sulfonamide treatment (P = .02, by Fisher's exact test). Penicillin mediated a 10% absolute reduction in the mortality rate attributable to cellulitis/erysipelas compared with no antimicrobial therapy.

Table 4. Mortality rates for cellulitis/erysipelas.

	Treatment
Outcome	Othera	UV	Sulfonamide	Penicillin
Overall, proportion of patients cured	2528/23657	409/4936	35/1593	1/325
Percentage of patients cured (95% CI)	10.7 (10.3–11.1)	8.3 (7.5–9.1)	2.2 (1.5–2.9)	0.3 (0–0.8)
Effect size, % (95% CI)	…	−2.4 (−1.3 to −3.6)	−8.5 (−7.4 to −9.6)	−10.4 (−9.5 to −11.1)

Note. CI, confidence interval. This table is available in its entirety in the online version of the journal.

^a“Other” refers to nonantimicrobial therapies, including topical creams (eg, magnesium sulfate, glycerin, etc.), blood transfusion, injection of anti-streptococcal serum into lesions, X-ray therapy (but not ultraviolet therapy), or bacteriophage therapy.

Mortality data were available from 33 studies of major abscesses (table 5). In 4 studies, sulfonamides did not significantly reduce mortality versus no antimicrobial therapy. In contrast, penicillin mediated a 6% absolute reduction in the mortality rate attributable to major abscesses relative to no antimicrobial therapy (P < .001).

Table 5. Mortality rates for carbuncles and furuncles.

	Treatment
Outcome	Other	Penicillin
Overall, proportion of patients cured	64/1016	0/337
Percentage of patients cured (95% CI)	6 (5–8)	0 (0–0)
Effect size, % (95% CI)	…	−6 (−5 to −8)

Note. CI, confidence interval. This table is available in its entirety in the online version of the journal.

^a“Other” refers to nonantimicrobial therapies, including topical creams (eg, magnesium sulfate, glycerin, etc.), blood transfusion, injection of anti-streptococcal serum into lesions, X-ray therapy (but not ultraviolet therapy), or bacteriophage therapy.

Modern dose-escalation data

A recent, phase II dose-escalation study of dalbavancin for complicated SSSI provided evidence of the minimal efficacy of active antimicrobial therapy for this disease. Seltzer et al. [59] randomized patients with complicated SSSI to receive a single infusion of 1100 mg of dalbavancin or a single 1000 mg infusion followed by a 500 mg infusion 1 week later. The clinical cure rate at the test-of-cure visit for the clinically evaluable population was 94% for patients treated with 2 doses of dalbavancin and 62% for patients treated with 1 dose of dalbavancin. For the intention-to-treat population, the cure rate was 91% for patients who received 2 doses of dalbavancin versus 60% for patients who received a single dose of dalbavancin. Compared with the single dose dalbavancin regimen, the 2-dose regimen was 32% more effective in the clinically evaluable population and 31% more effective in the intention-to-treat population.

Discussion

A comprehensive review of complicated SSSI from the preantibotic era and the immediately postantibiotic era revealed a substantial treatment effect of antimicrobial therapy. Penicillin was more effective than sulfonamides, consistent with both extensive historical experience and the fact that sulfonamide monotherapy has not been used since the advent of combination sulfonamide-dihydrofolate reductase inhibitors 40 years ago. Therefore, systemic penicillin was the clear historical gold standard antimicrobial therapy on which the basis for noninferiority margins should be justified.

ICH E9 and E10 guidances indicate that noninferiority margins should preserve a clinically meaningful fraction of the lower limit of the established efficacy of the comparator drug [22, 23]. Preservation of 50% of the comparator drug's efficacy relative to placebo or no therapy has been suggested as reasonable when setting noninferiority margins [60–64], but others have argued that this method is highly conservative [65], and this issue has not been clarified [64]. To preserve one-half of the lower limit of the treatment effects we found, the non-inferiority margin should be 14% for cellulitis/erysipelas, 21% for wound or ulcer infection, and 7% for major abscess. In practice, the noninferiority margin for a specific complicated SSSI trial should be weighted for the proportion of enrolled patients with cellulitis/erysipelas, wound or ulcer infections, and major abscesses. For example, if a 1:1:1 ratio of patients with cellulitis, wound or ulcer infection, or major abscess were enrolled, the noninferiority margin should be 14% (average of 14%, 21%, and 7%). We emphasize that, because of the low mortality of complicated SSSI treated with antibiotics, it may be reasonable to preserve <50% of the gold standard comparator's efficacy, resulting in wider noninferiority margins, especially if the new agent offered other clinical benefits [64], such as enhanced activity against antimicrobial-resistant bacteria and an enhanced safety profile, compared with currently available agents.

Our calculations for antibiotic efficacy were also conservative in that they were generated by subtracting the upper limit of the 95% CI of no antimicrobial therapy efficacy from the lower limit of the 95% CI of penicillin efficacy. Therefore, our calculations likely resulted in underestimates of the actual efficacy of penicillin relative to no antimicrobial therapy.

FDA [21] and ICH guidances [22, 23] also emphasize that the data used to justify a noninferiority margin must be relevant to modern studies (the “Constancy Assumption” standard). Several elements in our analysis support fulfillment of the Constancy Assumption standard for the efficacy of antimicrobial agents for complicated SSSI. First, the efficacy of 2 doses versus 1 dose of dalbavancin in a modern study of complicated SSSI provided an estimate of antibiotic efficacy that was strikingly similar to the historical datasets, and this conclusion was conservative, because 1 dose of dalbavancin was likely more effective than placebo. Second, the microbiology of the historical studies of cellulitis/erysipelas, wound or ulcer infection, and major abscess was similar to the microbiology of such infections in the modern era. Finally, the mortality rates associated with complicated SSSI treated with penicillin were similar to modern mortality rates of complicated SSSI when effective antimicrobial therapy is used.

We emphasize that previous studies evaluating antibiotic efficacy compared with placebo or in the setting of discordant therapy have focused on uncomplicated SSSI rather than complicated SSSI [11, 15, 17–19, 66–68]. In contrast, we specifically excluded analysis of furuncles (ie, uncomplicated abscesses) to focus on complicated SSSI. The 6% mortality rate among patients with a major abscess receiving no antimicrobial therapy reinforces the fact that these infections represented complicated SSSI rather than uncomplicated SSSI.

Because of the low mortality associated with antimicrobial-treated complicated SSSI, mortality is not viable as the only outcome measure for modern complicated SSSI clinical trials. Nevertheless, the historical mortality data underscore the efficacy of antimicrobial agents for treatment of complicated SSSI. Indeed, the fact that cellulitis/erysipelas caused an 11% mortality rate in the preantibiotic era has been largely forgotten in the antibiotic era. The magnitude of that mortality rate is emphasized by its similarity to the 12% mortality rate of myocardial infarction in the placebo arm of a modern, randomized, double-blinded study [69].

It is highly unlikely that changes in background medical care affected the mortality rates during the period of our survey, or indeed subsequent to it. For example, during >50 years leading up to 1936, there was no change in the mortality rate of erysipelas (figure 1) [43]. Immediately after the availability of sulfonamides, the mortality rate decreased by >3 fold [42, 43], at a time when no other new medical technology was being introduced. When penicillin became available there was another immediate 10-fold decrease in mortality, to rates comparable to those in the modern era, with no further change during the subsequent >20 years. Published testimonials from physicians caring for patients in the 1930s and 1940s affirm that it was the introduction of antibiotics, and not another change in medical practice, that led to the immediate decrease in mortality attributable to infection [70–73].

The primary limitations of our analysis include heterogeneity, the potential for publication bias, and the large proportion of single-armed, observational studies in the analysis. The heterogeneity of outcomes for patients receiving no antimicrobial therapy was largely accounted for by higher cure rates reported when fewer criteria were used to define treatment failure, and thus, results were likely biased toward a lower efficacy of antimicrobial agents rather than a higher efficacy. For wound or ulcer infections, heterogeneity was driven by inclusion of patients who received topical or local antimicrobial therapy in the analysis. Again, this resulted in a more conservative estimate of antibiotic efficacy, because the efficacy of penicillin relative to no antimicrobial therapy was higher (53%; 95% CI, 44%–62%) in the 7 studies that exclusively evaluated systemic therapy.

We found no statistical evidence of publication bias. However, given the limitations of such analyses [74], we cannot exclude the possibility that publication bias existed. If publication bias did exist, it likely affected the publication of results for both antimicrobial and nonantimicrobial therapeutic efficacy. For example, published cases in which topical ointments, dye solutions, ultraviolet therapy, or X-ray therapy were used to treat skin infection are likely selected for those cases in which more favorable results were observed.

Ideally, rigorous establishment of the magnitude of efficacy of antimicrobial versus no antimicrobial therapy for complicated SSSI would rely upon contemporary, double-blinded, placebo-controlled trials. However, the clear establishment of efficacy of sulfonamides and penicillin [60, 70, 72, 75–78] predated, by several decades, the widespread use of randomized, placebo-controlled trials. Nor can placebo-controlled trials of antimicrobial agents be conducted today for most types of infection [60, 79]. The limitations of the current analysis must be considered in the context of the public health imperative to develop new antimicrobial agents for resistant infection [1]. Antimicrobial agents are unique, not just among all drugs, but among all technologies, in that they continually lose efficacy over time in a transmissible manner. Therefore, unlike any other class of drugs, there is a perpetual need to develop new antimicrobial agents to enable treatment of bacteria that are continually becoming resistant to currently available drugs.

The ICH E10 guidance emphasizes that “the determination of the margin in a non-inferiority trial is based on both statistical reasoning and clinical judgment” [23, p. 9]. Therefore, in light of (1) the critical need for new antimicrobial agents, (2) the robustness of the datasets reviewed, (3) the conservative nature of the calculations, (4) the evidence of a large magnitude of antimicrobial efficacy for treatment of complicated SSSI, and (5) our compliance with critical features of the ICH E9 and E10 and FDA guidances, we believe that the current findings are sufficient to enable a rational justification for noninferiority margins for complicated SSSI clinical trials.