OARSI Clinical Trials Recommendations: Design and conduct of clinical trials of surgical interventions for osteoarthritis

summary

To highlight methodological challenges in the design and conduct of randomized trials of surgical interventions and to propose strategies for addressing these challenges. This paper focuses on three broad areas: enrollment; intervention; and assessment including implications for analysis. For each challenge raised in the paper, we propose potential solutions. Enrollment poses challenges in maintaining investigator equipoise, managing conflict of interest and anticipating that patient preferences for specific treatments may reduce enrollment. Intervention design and implementation pose challenges relating to obsolescence, fidelity of intervention delivery, and adherence and crossover. Assessment and analysis raise questions regarding blinding and clustering of observations. This paper describes methodological problems in the design and conduct of surgical randomized trials and proposes strategies for addressing these challenges.

This paper focuses on several methodological issues that are especially important in the context of surgical randomized controlled trials. This is a broad category of trials that includes comparisons of two (or more) distinct surgical procedures (e.g., open vs arthroscopic); comparisons of distinct technical features (e.g., different types of implants or screws); comparisons of surgical vs nonoperative treatments for a given condition (e.g., arthroscopy vs physical therapy) and still others. The authors include two clinician investigators (a rheumatologist and an orthopedic surgeon) and a biostatistician. Each of us has experience in the design and conduct of surgical trials. We planned the content of this paper via email conversations and resolved any differences in opinion through iterative comments on working drafts, emails and occasional face to face discussions. We focus our recommendations around three broad aspects of surgical randomized trials: enrollment; intervention; and assessment including implications for analysis.

Enrollment

This section focuses on three related issues: investigator equipoise; conflict of interest; and patient preference.

Investigator equipoise

As the term suggests, ‘equipoise’ refers to indifference (or equal position) between two alternatives. In the context of an RCT, equipoise refers to acceptance by members of the research team that each of the arms under study offers a reasonable treatment^1–3. This is especially important for research team members engaged in enrolling patients. If the enrolling investigator believes that one of the treatments under study is superior, he or she may subtly steer eligible patients away from enrollment in the trial and toward that treatment^4,5. There is no straightforward way to estimate and remedy biases in enrollment that may arise because investigators lack equipoise^6,7.

There are practical and conceptual consequences of departure from investigator equipoise. If investigators surgeons choose not to present the trial to certain eligible patients, because they feel these patients would benefit more with one treatment than the other, the selective enrollment can introduce bias. This is especially problematic if it turns out that particular subgroups of subjects have a higher likelihood of successful outcome if they receive one treatment than if they receive the comparator⁵. For example, if an investigator selectively enrolls patients with more severe symptoms, and if treatment A turns out to be more efficacious than treatment B in those with more severe symptoms than it is in those with less severe symptoms, then treatment A may have spuriously favorable results.

Several approaches have been developed to address the need for the research team to maintain equipoise. Perhaps the most important is that the trial should be presented to the potential subject by a member of the research team who is not involved in the patient’s care. While clinicians may identify potentially eligible patients in their practices, they should refer these patients to members of the research staff who present the trial in a standardized, neutral fashion. These presentations should follow scripts developed by the research team to convey the treatment arms neutrally. Members of the research team who present the study to patients can engage in role-playing prior to launching the trial in order to gain comfort with presenting the treatment arms in an unbiased fashion. Finally, as the trial progresses the research team can assess patterns of enrollment to discern whether particular investigators are referring selectively. The PI and team can try to work with such investigators and, if that is not effective, can resort to replacing the investigators with others better able to approach enrollment with equipoise (Table I).

Table 1.

Challenges arising in surgical randomized trials and suggested approaches

Challenge	Description	Potential approaches
Investigator equipoise	Surgeon investigator may believe one treatment is superior, leading to selective enrollment and potential bias.	-Standardized enrollment scripts delivered by research staff, not the treating surgeon. -Screen potential surgical investigators carefully -Role-playing to help investigators present the trial in an unbiased fashion. -Monitoring to detect biased enrollment.
Patient preference	Patients must assess their preferences for surgical vs nonoperative therapy to ascertain whether they are indifferent to the options under study.	-Standardized enrollment script delivered by unbiased research staff. -Media presentations (e.g., DVDs) providing unbiased, standardized information on risks and benefits in each arm.
Intervention obsolescence	The interventions under study may already be obsolete by the time the study is reported.	-Choose fundamental principles as study questions rather than very specific technologies.
Intervention fidelity	The surgical intervention must be delivered in a standard fashion, despite inherent variability in the procedure depending on surgical findings.	-Meetings among surgical investigators prior to study launch to establish protocol for intraoperative decision points. -Ensure that all surgeons are experienced with the interventions under study to avoid learning curve effects.
Crossover and adherence	Subjects may cross over from nonoperative to surgical therapy and may choose not to undergo the assigned surgery; these phenomena make interpretation of the intention-to-treat analyses challenging.	-Explain clearly to subjects and investigators that subjects should adhere to assigned treatment at least until the first assessment. -Capture reasons for crossover when they occur. -Specify secondary analyses a priori that include crossover as failures.
Blinding and controls	Mounting evidence points to the potency of the surgical placebo effect, creating a rationale for sham interventions. Sham procedures present ethical, logistical and interpretative challenges.	-Research funding bodies must be persuaded that sham procedures are worth paying for, as many insurers will refuse to do so. -Investigators should consider an additional ‘no treatment’ group to quantify the sham effect. -Research is needed to harness sham effects at lowest possible cost and risk.

Conflicts of interest

Commercial bias and conflict of interest are additional reasons for the erosion of equipoise in RCTs⁸. Investigators who stand to benefit financially if the treatment under study is successful may be especially likely to steer toward the trial those patients whom they suspect will benefit from that treatment and less likely to refer to the trial those patients who they suspect may not benefit from the treatment^9–13. In these circumstances, it is particularly critical for the trial to be presented to potential subjects by dispassionate research staff using standardized scripts. Clinician-investigators with a financial interest in one of the treatments under study should not participate as investigators in trials that assess the efficacy of that treatment.

Patient preference

Patients with a strong preference for one treatment typically decline enrollment in a trial of that treatment, knowing that they have just a 50% chance of receiving their treatment of choice (assuming a two-arm trial). If such patients enroll in trials they may be less likely to adhere if they are assigned to the comparator arm¹⁴. Thus, it is important to provide patients with detailed, comprehensible information on short- and long-term benefits of each treatment under study so that they can determine whether they have a preference for one of the treatments. Several groups of investigators have used videotaped presentations of the options, presented in a neutral fashion that highlights the benefits and drawbacks of each intervention^15–19. These programs permit potential subjects to learn about risks, benefits and alternatives in their own homes, providing them with an opportunity to develop preferences that are informed by the best evidence in the field. Investigators should be careful not to describe the treatment arms in a way that subtly fosters a preference for one treatment over the other. For example, the term ‘watchful waiting’ may suggest to patients a weak treatment, whereas the term ‘active, individualized monitoring,’ used to describe the same regimen, may seem more appealing.

Enrollment in surgical trials is often slower than the investigators anticipate. This observation underscores the importance of performing a pilot recruitment so that the research team can appreciate the number of patients that will need to be approached to yield the desired sample. Recruitment in multiple centers adds complexity but may permit enrollment goals to be reached faster, thereby providing a more prompt answer to the study question.

Interventions

This section discusses two key aspects of interventions in surgical trials, obsolescence and intervention fidelity.

Obsolescence

Surgery is characterized by rapid innovation and adoption of technology^20–22. This is one of the compelling reasons to perform surgical RCTs: new procedures and devices are introduced frequently. However, this phenomenon creates the risk that the intervention is no longer used by the time the trial is reported. This is a subtle problem to prevent as it is difficult to see the future. Investigators should endeavor to study interventions that are likely to endure, or to study broad approaches (e.g., arthroscopic vs open, surgery vs physical therapy) that remain relevant even when the particular surgical approaches and devices have been modified or supplanted. In an effort to avoid studying interventions that are likely to become obsolete, some investigators may choose to study technical details that may remain relevant for many years but are unlikely to change practice or result in improved quality of life for patients. As trials are resource intensive, we suggest that investigators choose questions that could change practice and/or alter the quality of patients’ lives.

Intervention fidelity

Intervention fidelity refers to the extent that the intervention is delivered in an identical fashion to each subject²³. This is seldom a problem in pharmacologic trials as the active agents and the comparators (e.g., placebos) can be taken at well-defined times of day and are given in the form of standardized tablets or capsules. But fidelity presents challenges in trials involving surgical procedures (e.g., surgery A vs surgery B or surgery vs nonoperative therapy)²⁴. Each patient’s anatomy is slightly different, so dissection planes, extent of lavage, control of blood loss, soft tissue modifications and many other aspects of the intervention require real-time, individualized decisions and actions. Surgeon experience and the learning curve phenomenon are related, important concerns²⁵. It takes time for surgeons to become accustomed to each procedure and to different sets of equipment²⁶.

To address these issues of intervention fidelity and learning curve, the surgical investigators should meet prior to the study launch in order to develop consensus on the precise surgical protocol, including the decisions to be made in the face of particular intraoperative findings. These discussions should be summarized in a study document and the principles should be revisited periodically at investigator meetings over the course of the study. The surgical details for each subject should be captured with a standardized data collection form. Postoperative care including rehabilitation protocols should be standardized similarly. If resources permit, sharing of intraoperative DVDs may also help to detect and remedy departures from protocol. While it may be difficult to define and measure the learning curve, nonetheless in trials that involve novel procedures and materials (e.g., implants, instruments), surgeons who have minimal experience with the procedure or equipment should not be involved.

Crossover and adherence

The issue of crossover is particularly salient in trials of operative vs nonoperative therapy. In these settings, subjects in the nonoperative arm may discuss with their physicians the option of crossing over to surgery if nonoperative therapy has not resulted in benefit. Also, subjects in the surgical arm may elect to decline surgery post-randomization^15,27. Both of these phenomena may make results of the intention-to-treat analysis of the trial (in which subjects are analyzed according to the treatment they were assigned, not the treatment they ultimately received) difficult to interpret²⁸. If many subjects in the nonoperative group were actually treated surgically by the time of the primary outcome assessment, then the intention-to-treat analysis will have surgically-treated subjects in each arm. Further, if a substantial number of subjects randomized to receive surgery do not undergo the procedure by the time of the outcome assessment, as occurred in the Spine Patient Outcomes Research Trial (SPORT)¹⁵, the problem is heightened further. The question of whether an initial strategy of surgery leads to better outcomes than an initial strategy of nonoperative therapy can be measured incisively in the intention-to-treat analysis. But more general inferences about whether surgery is associated with better outcomes than nonoperative therapy are conflated in the intention-to-treat analysis by the subjects who crossed over.

Given the likelihood of crossovers, all efforts should be made to retain subjects in their assigned treatment arms, at least until the first outcome assessment. Investigators should state this goal specifically in enrollment scripts and in discussions with surgeon investigators at the outset. When subjects do cross over, the investigators should ascertain the reason for crossover and assess the patient’s status with respect to the primary outcome measure just prior to crossover. Finally, the research team should consider an a priori secondary analysis plan that regards crossover from one arm to another as a failure of the assigned treatment, and include this in their study protocol¹⁶. This approach retains the intention-to-treat principle but acknowledges that the patient who crossed over was in fact a failure of the initially assigned treatment arm. Cross-over from surgery to nonoperative therapy (e.g., the patient decides post-randomization not to undergo surgery) would similarly have to be considered a failure in this type of analysis, underscoring the importance of ensuring that surgery is delivered promptly to those randomized to the operative arm.

Assessment and implications for analysis

This section discusses two important topics, blinding of subjects and assessors and accounting for clustering.

Blinding and control groups

A small number of surgical trials that compared surgery vs a nonoperative regimen have used sham procedures to ensure blinding^29,30. In the absence of a sham, blinding the subject to the whether they received surgery or a nonoperative intervention is difficult. A sham procedure controls for the experience of undergoing a surgical procedure, which may be potent. In fact, invasive procedures (surgery, injections) appear to have considerably stronger placebo effects than medications^31–33. One pragmatic drawback to sham treatment is that payers may not reimburse for it as they would for an accepted therapy. While insurance payers often bear the cost of treatment in research studies when the treatments are well established, the cost of sham generally must come from the research funds. Sham controls also raise the question – unresolved at present – of whether it is ethical to include a treatment designed to have no therapeutic effects. This issue pertains to surgical sham procedures and more generally to placebos of all sorts.

Interpretation of the results of sham studies also presents challenges. The recent sham controlled trial of arthroscopic partial meniscectomy for degenerative meniscal tear is a good example. Both groups underwent a structured exercise program as well as arthroscopic surgery and lavage. While both groups had arthroscopic surgery, only the ‘active’ group had partial meniscal resection. The arthroscope was simply withdrawn in the sham group. Over 80% of subjects in each group reported improvement and over 90% in each group indicated they would choose the procedure again²⁵. Clinicians, patients and policy makers are left to ask whether to abandon arthroscopic partial meniscectomy. We suggest when a procedure appears to be no more efficacious than a sham control, patients contemplating the procedure should be informed that it has been shown to be no more effective than sham. Further, policy makers and payers will need to think carefully about whether to support procedures that are no more efficacious than sham procedures. Finally, given the power of sham surgery and of placebo effects in general, further research is warranted that attempts to harness sham or placebo effects at the lowest possible cost and risk to patients.

Clustering of observations

In a randomized controlled trial of two treatments, we are primarily interested in the effect of treatment on outcome. Naturally, other features may also affect outcomes, including (in surgical trials) the surgeon. Subjects who are treated by the same clinician may have similar outcomes and the effect of the clinician on outcome may be independent of the intervention and of patient characteristics. If the surgeon influences outcomes then subjects cared for by a particular surgeon are not entirely independent, from a statistical standpoint. This clustering effect (of subjects within surgeons) can be handled analytically using one of several techniques. For example the investigators could include a random effect for surgeon, or use generalized estimating equations to adjust standard errors for non-independence of patients treated by the same surgeons. These methods account for clustering of outcomes within clinician or center. In addition, dummy variables designating particular surgeons or centers can also be added as explanatory variables.

Summary

Surgery may offer patients dramatic benefits, but generally at some risk of complications and at substantial cost. The cost and quality of life consequences of withholding effective surgery may be substantial as well. The evidence base available to assess the efficacy of surgery is limited. To address these and related problems it is critical to evaluate outcomes of surgical interventions rigorously, in randomized controlled trials. This paper highlights distinctive methodologic problems arising in surgical trials and offers strategies for addressing these challenges.