Four “Lessons Learned” While Implementing a Multi-Site Caries Prevention Trial

Abstract

As the number of dental-related randomized clinical trials (RCTs) increases, there is a need for literature to help investigators inexperienced in conducting RCTs design and implement studies. This commentary describes four “lessons learned,” or considerations important in the planning and initial implementation of RCTs in dentistry that to our knowledge have not been discussed in the general dental literature describing trial techniques. These considerations are 1) preparing or securing a thorough systematic review, 2) developing a comprehensive set of study documents, 3) designing and testing multiple recruitment strategies, and 4) employing a run-in period prior to enrollment. Attention to these considerations in the planning phases of a dental RCT can help ensure that the trial is clinically relevant while also maximizing the likelihood that its implementation will be successful.

The demand for “evidence” continues to escalate as the dental profession increases its reliance on the precepts of evidence-based dentistry when making both clinical decisions and policy recommendations.¹ This escalation has underscored the relative paucity of well-conducted randomized clinical trials (RCTs) that test the effectiveness of caries preventive agents in a variety of populations.² There are some signs that this need for evidence is being recognized. The number of RCTs published annually that examine effectiveness of methods for the prevention and management of dental caries has approximately doubled every five years since 1993, from 6 in that year, to 11 in 1998, 16 in 2003, and 32 in 2008. Additionally, the National Institute of Dental and Craniofacial Research (NIDCR) now supports Phase I, II, and III clinical trials designed to identify effective preventive, diagnostic, and treatment approaches for oral and craniofacial diseases and disorders.³

To sustain future growth of RCTs testing caries prevention and treatment methods, additional investigators who have not had the benefit of previous clinical trial experience will be needed to plan and conduct the studies. While most clinical dental studies will afford investigators a variety of opportunities to learn clinical research techniques, some aspects of the planning and conduct of dental RCTs are unique to the genre,⁴ and becoming familiar with them requires either experience with RCTs or directed didactic learning. Reading reports of clinical trials, becoming familiar with the CONSORT statement,⁵ and studying texts are all useful methods of directed learning, but all tend to emphasize statistical and design issues while offering the reader less in the way of techniques useful in planning and implementing the trial.

We are not aware of other reports written for dental investigators that discuss such techniques. Thus, the purpose of this brief commentary is to describe four “lessons learned,” or techniques that we found to be critically important in planning and conducting a multi-site RCT that evaluates the effectiveness of a caries preventive agent (the Xylitol for Adult Caries Trial, or X-ACT). The four techniques are 1) preparing, or securing a thorough systematic review, 2) developing a comprehensive set of study documents, 3) designing and testing multiple recruitment strategies, and 4) including a run-in period in the recruitment phase of the trial.

Thorough Systematic Review

Most trials are preceded by a literature review to justify the study hypothesis, determine sample size for the proposed trial, and demonstrate the need for the new information to be generated by the trial. However, all literature reviews are not equal, and will not necessarily offer the same degree of guidance to the investigators. Performing a systematic review, or reviewing available pertinent recent systematic reviews prior to preparing a research proposal, can substantially strengthen that proposal and subsequent clinical trial.

If important studies are missed or disregarded by the reviewer, pertinent information about the proposed intervention may be missed. As a result, the basis for the trial may be misstated, or information that would be useful in designing the intervention may be overlooked. By design, systematic reviews require exhaustive searches of the periodic and grey literature, with identification of pertinent studies accomplished independently by at least two investigators.⁶ Thus it is unlikely that pertinent studies will be missed.

Systematic reviews also require assessments of the quality of the included studies. Performing these assessments can afford investigators a detailed understanding of the design features and performance standards considered necessary for high-quality trials. Just as importantly, systematic reviews are expected to address the strengths and weaknesses of the available evidence.⁶ Completing the review provides investigators an opportunity to reflect on necessary steps to improve on, rather than simply replicate, any existing trials addressing the research question.

One other feature of systematic reviews useful in planning a clinical trial is the expectation that a systematic review address one or more focused clinical questions, usually stated in the “PICO” (Problem, Intervention, Comparison, Outcome) format. Formulating these questions, which will closely parallel the statement of specific aims of a research proposal, should help investigators design a trial that provides information that is immediately applicable to clinical care. The ability to cite a systematic review and summarize its findings and research recommendations concisely will also pay dividends under the new NIH brevity requirements.⁷

Our clinical trial evaluates the effectiveness of xylitol lozenges in reducing the incidence of caries in caries-active adults. We benefited both from preparing a systematic review, and from an existing systematic review. Our interest in evaluating the effectiveness of xylitol in a high-risk adult population arose from a systematic review⁸ prepared for the Consensus Development Conference on the Diagnosis and Management of Dental Caries Throughout Life,² which later appeared in the periodic scientific literature.⁹ Preparing this review alerted us both to the potential of xylitol for caries prevention and to the dearth of studies examining caries prevention in high-caries-risk adults. The existing systematic review of xylitol’s effectiveness¹⁰ clearly identified crucial design weaknesses of existing trials that we were able to address in our design.

Comprehensive Set of Study Documents

A successful clinical trial requires a lengthy planning period before it is launched, and that planning will be embodied in several study documents, including a study protocol, a manual of procedures (MOP) or manual of operations (MOO), and possibly documents for Food and Drug Administration (FDA) and Office of Human Research Protections (OHRP) requirements. The process of preparing study documents for a clinical trial provides investigators an opportunity for thorough and unhurried deliberation to design and test details of various trial elements. In contrast to an observational study, clinical trial investigators make an educated assumption about the possible effects of an intervention, agree to a design before study launch and test the approved design in a rigorous manner for the entire study period (typically two to four years) with very little modification. Therefore, careful planning is needed before the study is launched.

While many clinical trial elements, such as data management, training and monitoring procedures may be relatively simple applications of existing “off the shelf” systems used by data coordinating centers, other elements may demand more complex adaptation, if not de novo development. Such development could seriously delay initiation of the trial, or if rushed, compromise its internal validity. As an example, in the X-ACT trial, an adaptation of the International Caries Detection and Assessment System (ICDAS) caries recording method¹¹ was to be used. This required the design of an examiner and recorder training curriculum, development of recording forms, and specification of weights for all possible transitions in the transition matrix. The development of these items required an unexpectedly long time, and could have delayed the start of enrollment and baseline examinations had they not been developed prior to funding of the final clinical trial.

The study protocol is the final design document wherein the final primary outcome measure must be articulated together with secondary measures and the statistical plan and sample size justification.¹² All procedures and materials related to human subjects, such as advertising materials, recruitment procedures, and consent are included and submitted to an Institutional Review Board (IRB) for approval. If the investigators have taken advantage of pre-review advice offered by some IRBs during protocol development, prompt approval is made more likely. Details that may change throughout the trial that do not impact study design or human subjects concerns (such as location of screening visit or dates for lozenge resupply) should not be included in the protocol, as any changes must be approved by the IRB. These are typically included in the Manual of Procedures (MOP) or Manual of Operations (MOO)

The MOP contains complete descriptions of all roles and responsibilities, which facilitates recruitment and training of trial personnel once trial funding has been obtained. In addition, because clinical trials typically run several years, staff turnover is a reality. With a detailed, site-specific MOP, the learning curve for new personnel is reduced. Other advantages of a pretrial MOP are related to the establishment of clear policies and procedures for a variety of potentially divisive decisions to be taken later in the operation of the trial. For example, a MOP may contain policies on publication, including authorship, or it may identify a publication committee charged with developing those policies. Because sites may have different operating policies, development of the MOP should involve members of the investigative team from all participating sites. Procedures and forms developed by a smaller subset of investigators may not be transferable to other participating organizations. Also, developing the MOP offers the opportunity to form working relationships necessary for successful collaboration at an early stage in the trial. The MOP is a dynamic document that changes as needed to refine procedures. Any changes to the MOP or to the study protocol must be tracked, so the MOP format should be one that allows easy updating and tracking.

Multiple Recruitment Strategies

Recruitment is a basic component of trial planning, but the extent to which recruitment strategies need to be developed and refined prior to the initiation of the trial can easily be underestimated. Sources of potentially eligible participants should be identified, and for each source, methods to approach potential participants should be fully developed and, ideally, should be tested prior to initiation of the trial. The recruitment strategy designated as primary, that is the strategy that will be employed first in the recruitment phase of the trial, will usually represent the most cost-effective means of offering opportunities for participation. This designation will be based on assumptions about the rate of availability of potential participants and the likelihood of both their interest and eligibility for enrollment. It is critical that these assumptions be tested during the planning phase of the trial. It is equally important to have multiple secondary, or “back-up” recruitment strategies developed to the point of implementation, in the event that the primary strategy fails to perform as well as expected. While increasing the compensation for participation may be one of the back-up strategy, caution should be exercised with this approach. The line between compensation and coercion is not easy to identify, and paying more may be frowned upon by both the IRB and other investigators who find the expectations of potential participants raised.

In the X-ACT study, the methods actually used for recruiting participants varied greatly across the three clinical sites, despite a uniform planned primary strategy involving recruitment from dental school clinics. Pre-testing at one site indicated that this strategy would be efficient and effective. However, once recruitment began, another site found that the flow of patients through the clinic was inadequate. A back-up strategy was successfully initiated that involved recruiting from community dental clinics using posters and flyers distributed with the help of clinic staff. A third site found that when the school clinic yield was less than expected, it had to employ mass media advertisement to the local population at large through newspaper and radio ads. This unplanned change in strategy caused IRB delays, increased the budget, and required more staff time to handle inquiries and screenings. The site where the original strategy was tested found the patient flow from the dental school clinic adequate, but that a “prescreening approach” to target recruitment efforts was necessary for greater efficiency, as staff time required was greater than anticipated. Thus, despite a reasonably well-planned and tested primary recruitment strategy, two of three sites needed to employ secondary strategies, and the third site had to modify the strategy to operate within available staff time. More complete pilot-testing of the strategy in all three sites may have provided more accurate estimates of potential participant interest and flow per week, yielding more realistic staffing, facility and budgeting projections.

Run-In Period

A design feature that is seldom used in dental trials is the “run-in” period.¹³ The run-in permits potential participants to experience the actual trial procedures prior to entering the trial and being randomized to a treatment arm. Run-in periods are of value in identifying ineligible participants and in establishing that participants are capable of accomplishing trial procedures. If participants’ current treatments are discontinued, the feature is sometimes termed a washout period. Perhaps the greatest benefit of a run-in period is the opportunity it affords potential participants to decide if they can adhere to the trial regimen for the projected duration of the trial. Allowing participants to make this decision after experiencing the regimen should reduce subsequent dropout, but at the potential cost of weakening generalizability.¹⁴

Run-in periods can offer a means of reducing dropouts and poor adherence particularly for dental trials where participant adherence involves oral care routines. Certainly, in the X-ACT trial wherein participants were expected to let a lozenge dissolve in their mouths five times a day for three years, adherence to the study regimen was an important consideration. We asked potential participants to begin the trial with a four-week run-in period during which they would consume placebo lozenges according to the study regimen. We stressed that either they or the study coordinator could decide to discontinue further participation based on this experience. We found that the run-in period was effective in making potential participants aware of their long-term adherence responsibilities, with the result that of 945 individuals initially admitted into the trial’s four-week run-in period, 81 (8.6%) terminated prior to the end of the run-in period. Of the 864 individuals completing the run-in, 173 (20.0%) were not randomized and therefore did not formally enter the trial. Overall, 26.8% of potential participants did not enter the trial after experiencing the regimen during the run-in period. The majority of this group decided not to enroll because of the burden that long-term adherence represented.

These then are four important “lessons learned” from the X-ACT trial that we believe may prove useful to other investigators. Design the study based on learnings from a systematic review, develop a protocol and comprehensive MOP in advance of the grant application that would fund the full trial, plan and test multiple recruitment strategies before recruitment begins, and when indicated, use a run-in period to help minimize dropouts and poor adherence during the trial. Of course, attention to these four lessons alone will not ensure success. Problems can arise with virtually any aspect of a trial. Thus, seeking the advice of persons experienced in trial design and management will always be a method to benefit from others’ lessons learned.