Lessons Learned During the Conduct of Clinical Studies in The Dental PBRN

Abstract

Effectively addressing challenges of conducting research in nonacademic settings is crucial to its success. A dental practice-based research network called The Dental Practice-Based Research Network (DPBRN) is comprised of practitioner-investigators in two health maintenance organizations, several universities, many U.S. states, and three Scandinavian countries. Our objective in this article is to describe lessons learned from conducting studies in this research context; the studies are conducted by clinicians in community settings who may be doing their first research study. To date, twenty-one studies have been completed or are in implementation. These include a broad range of topic areas, enrollment sizes, and study designs. A total of 1,126 practitioner-investigators have participated in at least one study. After excluding one study because it involved electronic records queries only, these studies included more than 70,000 patient/participant units. Because the DPBRN is committed to being both practitioner- and patient-driven, all studies must be approved by its Executive Committee and a formal study section of academic clinical scientists. As a result of interacting with a diverse range of institutional and regulatory entities, funding agencies, practitioners, clinic staff, patients, academic scientists, and geographic areas, twenty-three key lessons have been learned. Patients’ acceptance of these studies has been very high, judging from high participation rates and their completion of data forms. Early studies substantially informed later studies with regard to study design, practicality, forms design, informed consent process, and training and monitoring methods. Although time-intensive and complex, these solutions improved acceptability of practice-based research to patients, practitioners, and university researchers.

Practice-based research networks (PBRNs) have continued to grow in number because of the unique advantages they offer to both research and quality improvement.^1-8 PBRNs generally aim to conduct research that will have immediate impact on the delivery of routine clinical care. To that end, ideas for studies typically come from clinicians in regular practice, called practitioner-investigators (p-i's), not from clinical scientists in academic institutions. These interactions between p-i's and academic scientists should involve close collaboration and shared decision making. This often results in a healthy tension between scientists’ desire to include substantial data detail, with a desire by the p-i's to limit data collection to what is feasible in busy clinical practice. The process is necessarily reciprocal and iterative, with both parties contributing and both deriving benefit. As each party becomes familiar with the other's priorities, pressures, interaction styles, and values, successful collaborations establish common ground within which studies are implemented.

One such PBRN is The Dental Practice-Based Research Network (DPBRN).^9,10 The DPBRN is comprised of p-i's, staff, and patients in outpatient dental practices, which constitute a wide variety of practice types, treatment philosophies, and patient populations, including substantial racial, ethnic, geographic, and rural/urban area of residence diversity among both its p-i's and their patients. Analyses confirm that DPBRN dentists have much in common with dentists at large, while at the same time offering substantial diversity.^11,12

The DPBRN emphasizes doing “practical science” about, in, and for the benefit of “real-world” clinical practice. This means that the p-i's themselves participate not only in developing study ideas, but also in designing, conducting, and communicating this research—all with the intent of having a direct impact on clinical practice in non-academic settings. We have discussed in detail previously the development and basic operations of the DPBRN.^9,11-13 Although p-i's from many U.S. states and Scandinavia participate in certain DPBRN studies, studies that require extensive training and interaction via face-to-face contact with p-i's have been focused in five regions to date: Alabama/Mississippi, Florida/ Georgia, Minnesota, Permanente Dental Associates in the states of Oregon and Washington, and the Scandinavian countries of Denmark, Norway, and Sweden.

The PBRN research context can present both scientific and operational challenges. Some literature does exist on these challenges.^4,6,13-17 However, to our knowledge, none have addressed the broad and diverse range of scientific and operational challenges faced by the DPBRN—challenges that have derived from multiple U.S. geographic regions and Scandinavian countries, multiple regulatory, scientific, and academic entities, and multiple types of public and private clinical delivery systems and patient populations. Such information and lessons learned from conducting research in this context should be helpful to those who plan to conduct research in comparable settings; this report aims to summarize these lessons. We describe these lessons learned by organizing them into three overarching themes: 1) determining how best to serve practitioner-investigators and to integrate their practical clinical expertise; 2) determining how best to engage dental patients as research participants; and 3) determining how best to develop and implement studies in diverse practice settings.

Methods

Key operating principles for the DPBRN are that the research questions mainly originate from p-i's and that their answers have the potential to improve clinical practice quickly. Furthermore, the research itself is done within the practices of the network members. Importantly, p-i's provide input at each step of the process. Ideas for studies are obtained from responses provided on the DPBRN enrollment questionnaire (publicly available¹⁸), in face-to-face meetings (e.g., orientation sessions, at DPBRN annual meetings, or in visits to the practice), or at the DPBRN website. Ideas for studies are discussed and prioritized by the Executive Committee (EC), which must approve all studies. This committee is comprised of six p-i's in regular clinical practice, the network chair, the principal investigator of the Data Coordinating Center, and a representative of the National Institute of Dental and Craniofacial Research (NIDCR), part of the U.S. National Institutes of Health (NIH). Each member has one vote, so the majority voting authority resides with the p-i's.

If the EC approves a study concept, it is forwarded to the NIDCR for approval based on lack of overlap with other NIDCR-supported studies and consistency with agency priorities. After approval, the DPBRN forms a Protocol Working Group comprised of the study's principal investigator, at least one p-i (who could also be the study's principal investigator), one or two other content experts, and a Data Coordinating Center statistician. This group develops the study concept into a full grant application. Typically, preliminary versions are reviewed by the EC to maximize the study's scientific merit, applicability to daily clinical practice, and feasibility. Once the final version is approved by the EC, it is forwarded to the Protocol Review Committee (PRC) for final scientific review. Members of the PRC are selected by the NIH and are unaffiliated with the DPBRN so as to provide an objective, independent scientific review. This committee consists of academic clinical scientists only. If approved by the PRC, the study is considered “approved” and can next be submitted to each DPBRN region's Institutional Review Board (IRB) for human participants protection review. Once IRB approval is obtained, data collection forms are pilot tested with p-i's on the EC and selected p-i's across the network. Once final versions of all forms have been completed and received approval as amendments by the respective IRBs, a study is implemented in all regions.

Network enrollment consists of completing an online questionnaire (DPBRN enrollment questionnaire¹⁸). To participate in clinical studies, p-i's are required to receive DPBRN orientation and human participants research training. The most commonly used course is available online,¹⁹ although training processes vary some by region. P-i's can participate in studies only after an IRB has approved their participation for a specific study; annual recertification is typically required. Thus, each p-i is added to each study separately, on a study-by-study basis. Additional requirements vary by region.¹³

Table 1 lists the twenty-one studies implemented by the DPBRN as of March 2011; a broad range of topic areas, enrollment sizes, and study designs is included. A total of 1,126 individual dentists or dental hygienists have participated as p-i's in one or more of these studies. After excluding Study 7 (because it involved electronic dental records queries only), these studies included more than 70,000 patient/participant units. This situation has provided the DPBRN many opportunities from which to glean lessons. Although the aforementioned literature about PBRNs did provide some guidance, the methods by which we learned these lessons were in some instances deliberative and intentionally iterative; in other instances, they were done by trial and error or without deliberate planning. Some learning opportunities occurred because dental PBRNs were still relatively new during this learning period; others occurred because of the wide range and size of studies that the DPBRN developed and implemented; and others occurred because the DPBRN has such a broad diversity of practice types and patient populations.

Table 1.

DPBRN studies field tested as of March 2011

Study Number	Study Title	Study Design	Number of Practitioners or Practices	Number of Patients, Procedures, or Other Entities
1	Dental tobacco control randomized clinical trial	Randomized clinical trial	190	11,898 patients
2	Practice-based root canal treatment effectiveness	Retrospective cohort study	13	84 patients; 174 treated teeth
3	Assessment of caries diagnosis and caries treatment	Paper questionnaire to dentists	565	n/a
4	Reasons for placing the first restoration on permanent tooth surfaces	Cross-sectional; consecutive patients	227	5,810 patients; 9,902 restorations
5	Reasons for replacement or repair of dental restorations	Cross-sectional; consecutive patients	197	7,463 patients; 9,828 restorations
6	CONDOR case-control study of osteonecrosis of the jaws	Case-control study	81	764 patients
7	Retrospective cohort study of osteonecrosis of the jaws	Retrospective cohort study	n/a	572,606 patients
8	Longitudinal study of dental restorations placed on previously unrestored surfaces	Prospective cohort study	227	5,810 patients; 9,902 restorations
9	Longitudinal study of repaired or replaced dental restorations	Prospective cohort study	197	7,463 patients; 9,828 restorations
10	Development of a patient-based provider intervention for early caries management	Cross-sectional; consecutive patients, clinical data collection, interviews, and questionnaires with dentists and patients	10	336 patients
11	Patient satisfaction with dental restorations	Cross-sectional; consecutive patients	196	7,384 patients
12	Prevalence of questionable occlusal caries lesions	Cross-sectional; consecutive patients	82	7,677 patients; 2,301 caries lesions
13	Longitudinal study of questionable occlusal caries lesions	Prospective cohort study	82	7,677 patients; 2,301 caries lesions
14	Hygienists’ Internet tobacco cessation randomized clinical trial	Randomized clinical trial	100	1,865 patients so far [study in progress]
15	Blood glucose testing in dental practice	Cross-sectional; consecutive patients	28	498 patients
16	Assessing the impact of participation in practice-based research on clinical practice and patient care	Cross-sectional paper questionnaire with dentists and dental hygienists	687	n/a
17	Perioperative pain and root canal therapy	Short-term prospective cohort study	63	759 patients
18	Persistent pain and root canal therapy	Prospective cohort study with 6-month follow-up	63	759 patients
19	Primary care management of temporomandibular joint disorders	Electronic questionnaire with dentists	503	n/a
20	Infrastructure update survey	Electronic questionnaire with practitioners who had completed one or more studies to that point	685 so far [study in progress]	n/a
21	Diagnoses for persistent dentoalveolar pain following root canal therapy	Nested case-control study	[study in progress]	[study in progress]

n/a: not applicable

Results

We organized the key lessons learned—discussed in this section—into three overarching themes (listed in Table 2).

Table 2.

Lessons learned, organized by three overarching themes

Overarching Theme	Lesson Number	Description
Theme 1. Determining how best to serve practitioner-investigators and to integrate their practical clinical expertise into studies.	1	Practitioners will readily contribute ideas for projects, but these require close collaboration with academic researchers to refine them into researchable questions.
	2	The practical expertise of practitioner-investigators on the Executive Committee will become evident when they 1) reject study ideas that they consider impractical, not of widespread interest, or not likely to be quickly translated into daily clinical practice and 2) provide feedback about drafts of the grant application and data forms.
	3	Many practitioners feel isolated and enjoy interaction with other practitioners during and after network meetings; these meetings offer opportunities to translate research into practice because some p-i's serve as change agents.
	4	Practitioner-investigators respond more positively to findings presented by other p-i's than by academic researchers.
	5	Many practitioners welcome opportunities to present findings in a broad range of venues.
	6	Practitioner-investigators are very curious about how results from their practice compare to other practices.
	7	Remuneration must convey to practitioners that the network respects and values their time.
	8	Practitioners will use the remuneration amount in a variety of ways.
	9	Recent graduates can effectively add to the diversity of p-i's in the network.
Theme 2. Determining how best to engage dental patients as research participants.	10	PBRN patients will be responsive to a broad range of informed consent processes, and their reaction can help influence IRBs to simplify these processes.
	11	Information and remuneration must convey that patients’ participation is valued; patients will return questionnaires at a high rate even if required to return them after they leave the office.
Theme 3. Determining how best to develop and implement studies.	12	Academic researchers who are new to the PBRN research context have to become educated regarding how to adapt their studies to the PBRN context and to learn that clinicians drive the process, not them. This may cause delays in the grant application process, field testing, and study implementation.
	13	Practitioner-investigators who are new to the PBRN research context have to become educated about the necessity to standardize the data collection process.
	14	It is helpful to have latitude in how a practice can implement the informed consent and data collection processes.
	15	Answering questions to improve clinical practice requires careful consideration of the best study design; randomized clinical trials are not always the best such design.
	16	Even though they have completed all the steps required for certification to do studies, not all p-i's and their staff are going to effectively complete studies that they begin.
	17	It is important to design data collection forms that can be easily understood. Clear graphics and color coding help make data collection forms more readily understood. If possible, data forms are best limited to two or three pages per clinical encounter. Research staff and investigators must be held to deadlines when asked to provide suggestions for forms design or this will unduly delay implementation.
	18	Practices may not be willing to be responsible for electronic data entry unless it can be done readily as part of the routine dental care process.
	19	Even when data forms are customized to electronic health records, delays can occur due to data extraction and data transfer.
	20	Although many practitioners are open to doing studies in quick succession, few will want to do more than one new study at any one time.
	21	Even with recent training on a particular protocol, close monitoring is always advisable. Although all practitioners and staff can be trained on a particular protocol, it is best to identify a key contact person who will take responsibility for following proper study protocol.
	22	The Data Coordinating Center must be prepared to provide multiple methods of data collection even within single studies.
	23	Poor economic conditions can substantially affect patient recruitment rates for some studies, so enrolling more practitioners than originally planned will be necessary to end data collection by the planned date.

Theme 1. Determining how best to serve practitioner-investigators and to integrate their practical clinical expertise into studies

Lesson 1: Practitioners will readily contribute ideas for projects, but these require close collaboration with academic researchers to refine them into researchable questions

Numerous ideas for research studies were received from practitioners. However, these ideas were seldom ready for use in actual studies. Instead, an iterative process was required to better define the clinical problem and the appropriate research question. Additionally, it was not uncommon to conclude that identifying enough eligible patients with the required condition(s) in a timely manner would be unlikely, or that the required study design would be too complex for busy clinical practices, or that the length of the necessary observation period would exceed the length of the grant funding period.

Lesson 2: The practical expertise of p-i's on the EC will become evident when they 1) reject study ideas that they consider impractical, not of widespread interest, or not likely to be quickly translated into daily clinical practice and 2) provide feedback about drafts of the grant application and data forms

Although a total of twenty studies had been approved as of June 2010, many others never got past the concept phase with the EC. As of June 2010, forty-four formal study ideas had been considered, all of which required writing a one-to-three page summary of the proposed project. Of these, twenty became approved studies, twelve were rejected at the concept stage, and an additional twelve did not progress to full-scale applications. The twelve rejected by the EC were due to these reasons: 1) the topic area would not be of sufficiently broad interest; 2) recruitment could not be completed in a reasonable time; 3) the data collection protocol could not readily be incorporated into clinical routines; or 4) the anticipated budget was too high for the priority given the study. The p-i's on the EC were involved with all of the main drafts of the grant applications because every project had to be approved by this committee. Although one p-i on the EC had received formal research training, none of the others had. Nonetheless, each p-i consistently provided valuable insight and recommendations for improving the proposed studies.

Lesson 3: Many practitioners feel isolated and enjoy interaction with other practitioners during and after network meetings; these meetings offer opportunities to translate research into practice because some p-i's serve as change agents

The DPBRN has annual face-to-face meetings of its p-i's in each region. These meetings include the usual lecture-style presentations, but also small-group discussions and other small-group interaction venues, such as poster discussions. It was clear early on that p-i's enjoy these interactions and typically rated the interactive sessions the highest. Topics ranged from discussion of study results to discussion of evidence summaries to discussion of proposed studies, data forms, and proposed data collection processes. These venues provide opportunities for equals to interact collegially and often lead p-i's to discuss and question the treatment approaches that they use, offering a unique venue for rapid translation of study findings into clinical practice. These treatment approaches often differ substantially among p-i's even within a single DPBRN region and especially across the network. This creates an opportunity in which fellow p-i's can act as change agents, as we have reported previously.²⁰

Lesson 4: P-i's respond more positively to findings presented by other p-i's than by academic researchers

A common means of disseminating clinical evidence is to have full-time academic researchers present a lecture to an audience of practitioners. The academician is thereby cast as the expert who is providing knowledge in the researcher-to-practitioner direction. However, because the DPBRN is committed to the notion that information flow should proceed in the practitioner-to-academician direction as well, study results are presented by p-i's themselves. After several years of experience in many different venues, it is clear that practitioners often grant greater credibility to the practitioner presenter than the academic researcher. We believe that this is because the practitioner is “one of them” and is perceived as being more understanding of the realities of regular clinical practice and what the study results should mean to fellow practitioners. This approach also boosts the p-i's credibility and contribution to new learning.

Lesson 5: Many practitioners welcome opportunities to present findings in a broad range of venues

P-i's have successfully presented findings in a broad range of venues, including meetings of DPBRN practitioners, dental association meetings at the local, state, national, and international levels, and national and international research meetings. Their level of engagement has varied, but typically p-i's are involved in interpretation of study results, development of presentations (oral, poster, or poster discussion formats), and in-person and/or telephone discussion about the presentation beforehand. The DPBRN considered it a transition point when practitioners presented orally at one international dental research meeting after the DPBRN had been in operation for three years. The audience was comprised of only academic researchers and NIH personnel. The practitioners gave effective scientific presentations and responded well to questions from the audience. At that point, it was clear that practitioners engaged in PBRN research do indeed have valuable practical clinical expertise and can function as true collaborators in the clinical research process. At its funding inception, there had been skepticism among many academic researchers that the dental PBRN concept could be effective at recruitment, proper study conduct, and full engagement of practitioners. We no longer hear this skepticism.

Lesson 6: P-i's are very curious about how results from their practice compare to other practices

The DPBRN provides to practitioners who participate in a particular study the key results from their own practice, as well as overall results from the entire network. This allows them to objectively compare their practices to other practices in their own region and networkwide and not rely only on their own perceptions. Practitioners have widely reported that they find this information very enlightening and that this leads them to question why their practice differs from others, which then can lead to additional research questions and openness to change.

Lesson 7. Remuneration must convey to practitioners that the network respects and values their time

The most common reasons that practitioners report being engaged in the DPBRN are the opportunity to participate in research and the increased stature that it conveys to their practice and patients. However, they also are motivated because the time that it takes to do the research is remunerated, instead of relying entirely on an altruistic spirit. Notably, most of the DPBRN Institutional Review Boards stipulate that practitioner compensation be for the time that it takes to do the research—not for enrolling patients—because this might create a conflict of interest. The amount of this compensation is determined by the EC, typically after it has tested the data forms and made its own assessment of how much time the study will take. For DPBRN questionnaires completed by p-i's, the typical remuneration amount is US$50. A small percentage of p-i's (less than 5 percent) decline this incentive even though they have completed a questionnaire. The DPBRN's first clinical study that involved all five of its regions (DPBRN Study 4) required that practitioners record information about fifty consecutive dental restorations of a particular type, which typically involved about thirty-five patients. The typical remuneration to each practitioner was about US$2200 for the whole study.

Lesson 8: Practitioners will use the remuneration amount in a variety of ways

The DPBRN allows p-i's to use the remuneration as they deem appropriate. Some p-i's view this as regular practice income and therefore do not share it with staff directly, while others use it both to incentivize the staff and to validate that they are an integral part of the research project. Others use it to fund continuing education for themselves or their staff or use it as travel or entertainment bonuses for their staff.

Lesson 9: Recent graduates can effectively add to the diversity of p-i's in the network

Some PBRNs limit enrollment to established clinicians only, but the DPBRN has consciously included clinicians of all types, including recent graduates. The DPBRN's initial concern was that new graduates might be quite mobile, entering a practice upon graduation from dental school but then moving to a different practice a few years later. This has indeed happened on several occasions, but these practitioners usually remain DPBRN p-i's and continue to add to the diversity of experience, training, treatment philosophy, and patient characteristics, so our recommendation is that all PBRNs include recent graduates. Additionally, all of the DPBRN regions with an administrative base associated with a university dental school have also made a point of including content about the PBRN research context in the school's curriculum. As these students graduate, ideally they enter practices in which there already are other DPBRN p-i's, who can then reinforce the notion of including scientific inquiry as part of routine clinical practice. Because some DPBRN p-i's also serve as part-time faculty members at dental schools, the DPBRN also contributes to the research expertise of each school's faculty. The DPBRN's long-term aim is to inculcate in students and residents a desire to engage in the excitement of scientific discovery when they enter practice and to feel empowered to lead evidence-based dentistry efforts in the profession throughout their careers.

Some PBRNs require substantial training before allowing membership. The DPBRN's approach has been to make entry requirements relatively minimal, hoping that as a result of being engaged at a minimal level, practitioners will ultimately become more committed to PBRN research and consequently request further training that allows them to participate in clinical studies. This has indeed happened in many instances, mainly because practitioners become interested in a particular study and are eager to participate in a study having to do with that topic.

Theme 2. Determining how best to engage dental patients as research participants

Lesson 10: PBRN patients will be responsive to a broad range of informed consent processes, and their reaction can help influence IRBs to simplify these processes

DPBRN studies have used a broad range of informed consent processes, owing to the broad range of study designs and topics and to the variation among its IRBs.¹³ These processes have ranged from requiring no informed consent to verbal informed consent following review of an informed consent information sheet to detailed, multipage consent forms that each patient must sign. Patients’ reactions to these processes have helped the DPBRN simplify the informed consent process for low-risk studies.¹³

The PBRN context is not only a means to collaborate as equals with community clinicians, but in keeping with the basic principles of community-based participatory research,^4,14,21 it is important to engage patients as well. The DPBRN began initially by obtaining this input during feasibility and pilot testing of certain studies, then progressed to a study that formally included patient perceptions as part of the formal data set (DPBRN Study 11), and now involves a Patient Advisory Board. Additionally, patient representatives serve on the NIDCR PBRN Monitoring Committee, an advisory committee managed by the NIDCR. As the DPBRN has progressively increased its direct input from patients, it is clear that they also can offer valuable input to the research process as end-users.

Lesson 11: Information and remuneration must convey that patients’ participation is valued; patients will return questionnaires at a high rate even if required to return them after they leave the office

The DPBRN provides a brochure describing general DPBRN operations for practices to place in their waiting room or at their reception desk.¹¹ Study-specific information is provided as part of the informed consent process. These documents convey to patients the value of their participation. Participation by patients has been high. DPBRN Study 1 required that consecutive eligible patients complete a brief questionnaire regarding smoking cessation before they left the office and had an 82 percent completion rate.²² The only incentive was that participants were allowed to keep the pen (which had a study logo) that they used to complete the card. DPBRN Study 4, which required no data collection by patients but instead only required informed consent to allow the p-i's to record data, had a 95 percent participation rate.²³ Another study required participants to complete a “satisfaction with care” questionnaire.¹⁸ Because the study (DPBRN Study 11) had to do with placement of a dental restoration, patients had to complete the questionnaire after the dental anesthetic had worn off and therefore after they had left the office. Patients were mailed a $10 participation incentive after a completed questionnaire had been returned, along with the contact information necessary to mail the incentive. Consideration was given to making the participation incentive quite high so as to encourage a high response rate, but one of the network's IRBs expressed concern that high incentive amounts can be coercive. The DPBRN's concern was that not requiring completion before leaving the office would cause an unacceptably low response rate. This ended up not happening; about 80 percent of the patients returned completed questionnaires.²⁴ A small percentage of patients (less than 5 percent) have refused incentives even though they have completed their participation. The DPBRN Danish IRB has concluded that all patient payments can be coercive and does not allow them; however, it is permissible to provide to eligible patients (regardless of whether they actually participate) a bag of dental items of comparable value.

Because the NIH requires that the race and ethnicity of participants be described in all studies, the DPBRN regularly collects this information by asking patients after informed consent. This information includes age, sex, race, Hispanic/Latino ethnicity, and presence of dental insurance of any type. If the study is about routine care and if this is all that is required of patients, no remuneration is provided to the patients. In studies of this type, more than 95 percent of patients who have consented to the study have agreed to answer these questions. In the DPBRN's first networkwide clinical study, the data collection form asked race, followed by Hispanic/ Latino ethnicity. In one region, some patients were offended by this because they felt that they had already answered the second question once they had answered the first question. In all subsequent DPBRN studies, this order was reversed, and no further complaints were received.

Theme 3. Determining how best to develop and implement studies

Lesson 12: Academic researchers who are new to the PBRN research context have to become educated regarding how to adapt their studies to the PBRN context and to learn that clinicians drive the process, not them. This may cause delays in the grant application process, field testing, and study implementation

Just as practitioners have to become educated about the research process, academic researchers have to become educated about the PBRN context. Many academic clinical scientists have never been in full-time clinical practice and therefore do not have first-hand experience of its practicalities. Most are used to conducting highly detailed studies in academic health centers, studies that are labor-intensive for the investigator-clinicians and time-intensive for patients. Therefore, it has been common for academic researchers to submit an initial application to the DPBRN EC that is deemed too disruptive to regular clinical practice, has data collection forms that are too long or complex, or requires very time-intensive training of p-i's. We have learned to have these researchers attend the EC meeting at which their applications are under consideration in order to answer questions and hear first-hand the reactions from the p-i members. This tends to reduce the number of iterations at which the EC will need to consider the application. We then require the academic researcher to work closely with staff from each region to account for local variations in dental care delivery. The same lesson applies to academic researchers on the application's NIH study section (PRC); it is fortunate if the study section's members have actual experience in doing PBRN research because these members can educate the other study section members about the PBRN research context.

Lesson 13: P-i's who are new to the PBRN research context have to become educated about the necessity to standardize the data collection process

Because p-i's typically have had no formal research training, they need to be educated and trained in the necessity of standardizing the data collection process, following proper research protocol and creating an audit trail or “Good Clinical Practice.”²⁵ In the DPBRN, this process begins with the practitioner's first orientation session, typically delivered in a continuing education classroom-style format. These principles are expanded upon and reinforced when training is done subsequently for a specific study. In some DPBRN regions, training for a specific study occurs in a classroom-style format with small groups of p-i's. However, in most cases, this is done via a session in the practice itself, on a practice-by-practice basis. It is common for the Research Coordinators to remain in the practice for one-half day to a couple of days, depending on the experience of the practice. During this observation period, standardization concepts are reinforced, and compliance is verified. Follow-up support by telephone is also provided as needed.

Lesson 14: It is helpful to have latitude in how a practice can implement the informed consent and data collection processes

The dental office has a mix of personnel that includes dentists, dental hygienists, dental assistants, office managers, and other staff members. P-i's must be the ones to confirm that participants understand the study and are fully informed before they sign the informed consent form. However, other personnel are allowed to explain the study to potential participants if they are certified in human participants research. Office personnel who have not been certified are only allowed to provide the patient an information sheet or a copy of the informed consent form to read as background. Therefore, each DPBRN practice is asked to consider whether it wants to have certain non-dentist staff certified in human participants research. Having other personnel trained increases efficiency because most questions that potential participants ask can be answered by these trained personnel. In most DPBRN studies, data collection occurs in the dental operatory. However, for those protocols in which some aspects can occur at the reception desk or another location in the practice, during protocol training the research coordinators help the practice identify how best to customize that protocol's data collection process to minimize burden on the practice's busy patient flow.

Lesson 15: Answering questions to improve clinical practice requires careful consideration of the best study design; randomized clinical trials (RCTs) are not always the best such design

Although an RCT typically is the most rigorous design for clinical studies, it is not always the best design to help move scientific evidence expeditiously into routine clinical practice. PBRNs may come under pressure to conduct RCTs because RCTs are viewed as a measure of the value of PBRNs as a research strategy. A key advantage of most PBRN studies is that they intentionally do not use highly selected samples, but instead enroll all consecutive patients for whom certain treatment options would be appropriate. In that manner, they maximize the generalizability of conclusions made about treatment effectiveness. They also allow for an analysis of the process of care, such as determining which patients are offered treatment by clinicians and which patients choose to accept it—a possibility precluded in an RCT.

Lesson 16: Even though they have completed all the steps required for certification to do studies, not all p-i's and their staff are going to effectively complete studies that they begin

In DPBRN studies to date, some practices have dropped out after being formally enrolled in a study—as high as 14 percent in one study. This has been due to a wide variety of reasons, such as death or disability of the p-i or a key staff member or some other disruptive event, unrealistic expectation by the practice regarding its ability to enroll patients, a practice's decision that data collection is too burdensome, or that proper protocol was not followed throughout the data collection period such that the practice has to be excluded. Therefore, it is best to train and enroll more practices than the protocol requires so that by the end of the data collection period, the required data volume will have been achieved. For several DPBRN protocols, the enrollment target had to do both with the number of p-i's and the number of treatment procedures (e.g., number of dental restorations). Because the statistical power for these studies was driven more by the number of practitioners than the number of procedures, it has been common to “overenroll” p-i's, with explanation to them that if we succeed in overenrolling (which in fact has occurred in most of the DPBRN studies so far), then we will ask them later not to contribute as many treatment procedures.

Lesson 17: It is important to design data collection forms that can be easily understood. Clear graphics and color coding help make data collection forms more readily understood. If possible, data forms are best limited to two or three pages per clinical encounter. Research staff and investigators must be held to deadlines when asked to provide suggestions for forms design or this will unduly delay implementation

Although all p-i's and applicable staff members are fully trained on each protocol before implementation, the high level of activity in practices can lead to mistakes, memories can fade about details, and staff can experience turnover. Therefore, during the development of each study, data collection forms are fully vetted to a full range of regional coordinators, p-i's, and staff, who represent a broad diversity in dental clinical experience, research experience, practice types, and patient populations served. Early stages of development entail thought experiments, while the later stages test implementation of the data forms in the midst of actual delivery of care. Data collection forms for most studies are publicly available.¹⁸ Because of the DPBRN's commitment to designing forms that are readily understandable and to conducting studies that do not unduly disrupt busy patient flow, early studies allowed revisions in data forms until immediately before networkwide launch. Although some DPBRN reviewers had had several months and/or iterations to review forms, in a few instances these reviewers did not provide serious input until it was clear that the current version of the form was the one that would be implemented across the network. In the first several studies, this led to delays in launch. For subsequent studies, deadlines were mandated, so reviewers knew that for their voice to be heard, they had to provide input by the stated deadline.

Lesson 18: Practices may not be willing to be responsible for electronic data entry unless it can be done readily as part of the routine dental care process

One vision for PBRNs is that practices enter research data directly into a computer, instead of sending paper forms to a Data Coordinating Center. This vision mirrors the circumstance of how some clinical trials and epidemiologic studies are conducted in academic health centers, whereby trained staff directly enter research data via a web-based portal. Two p-i groups in the DPBRN (HealthPartners and Permanente Dental Associates) have customized their electronic dental records software for certain DPBRN studies to facilitate real-time data collection during routine delivery of care. However, the typical p-i in the network does not want to be responsible for this type of activity. The one exception has to do with online questionnaires that query responses directly from the practitioner, a mode that the DPBRN has used for its enrollment questionnaire and questionnaire studies.

Lesson 19: Even when data forms are customized to electronic health records, delays can occur due to data extraction and data transfer

An expected advantage of data collection at the point of care is that it decreases the time between data collection and data analysis. With the exception of the online questionnaires, this has not proven to be the case in the DPBRN. Point-of-care data collection for specific studies has required customized screens. This customization facilitates getting the required data into the system, but it has slowed its extraction at the end of the study because local information technology staff members have to devote focused time to do this outside of their routine work schedule. Electronic dental records currently available were not designed for research purposes. Additionally, once data extraction has been done locally, it has to conform to certain formatting requirements so that these data can be merged into the larger DPBRN data set.

Lesson 20: Although many practitioners are open to doing studies in quick succession, few will want to do more than one new study at any one time

Because dental practices tend to deliver care using very structured, predictable routines, adding a research study into their daily routine requires them to shift gears and redirect attention in the midst of care. This can be disruptive or confusing and therefore potentially burdensome. This is further compounded if more than one research protocol at a time has to be taken into account. The one exception has to do with studies that are already occurring “in the background.” In the DPBRN this means longitudinal follow-up of specific treatments done in previous years that will require intermittent data collection because only occasionally will patients enrolled in these studies appear in the practice, each of whom has a chart flagged to readily identify the patient and for whom the data collection forms are already in the chart.

Lesson 21: Even with recent training on a particular protocol, close monitoring is always advisable. Although all practitioners and staff can be trained on a particular protocol, it is best to identify a key contact person who will take responsibility for following proper study protocol

The usual plan in the DPBRN is for regional coordinators to mail or e-mail documents to the practices and ask the practice to review them in advance of a training session. The training session is then conducted in the practice itself, often at midday. The practice arranges not to schedule any patients during that time, and the regional coordinator typically brings lunch for all those who plan to attend. Following the training session, regional coordinators remain on site for the remainder of the day and/or the initial hours of data collection. Practices are then required to fax completed logs, informed consent forms, and data collection forms so that the regional coordinator can verify that proper protocol is being followed. Practices are typically asked to do this on a weekly basis while they are actively collecting data. If problems arise, the regional coordinator may need to revisit the practice for additional training. After data collection has ended, the regional coordinator will visit the practice to close out the study and ensure that a proper audit trail has been laid, informed consent has been properly documented, and each patient's chart is properly prepared for any longitudinal follow-up where applicable. Regional coordinators are given latitude about how frequently to monitor practices, given that they have observed the effectiveness of their training and often have experience with each practice from earlier studies.

DPBRN regional coordinators report that it is best to identify a key staff person who not only will serve as the main point of contact, but who will also serve as the practice's champion to ensure that all staff and p-i's follow proper protocol. In those rare cases when this champion leaves the practice, regional coordinators revisit the practice to reorient any new staff and retrain existing staff as necessary. Although in some practices the dentist p-i handles all the informed consent and data-recording processes, in most practices, staff play key roles; DPBRN regional coordinators typically report that the more the staff are involved in the study, the more successful its implementation in busy clinical practices.

Lesson 22: The Data Coordinating Center must be prepared to provide multiple methods of data collection even within single studies

At the request of the DPBRN EC, data collection for most studies has been recorded on paper forms. These forms have bar codes that embed the p-i identifier and other study-specific items. Two regions do direct data entry for many of their studies within the routine electronic dental record, using a screen customized for that particular study. Some studies (e.g., DPBRN Study 19 and the DPBRN Enrollment Questionnaire) were done via online entry by all p-i's. Nonetheless, even in Study 19, a small percentage of p-i's (less than 5 percent) requested a printed questionnaire because of computer problems or simply a desire to record the information by hand.

Lesson 23: Poor economic conditions can substantially affect patient recruitment rates for some studies, so enrolling more practitioners than originally planned will be necessary to end data collection by the planned date

In the early part of the field phase of Study 17, it became evident that more patients than normal who were scheduled for root canals were choosing for financial reasons to receive dental extractions instead of root canals or never returned to have the root canal finished once they were relieved from their pain. We also began receiving anecdotal reports from p-i's that patient flow was down in their practices overall and patients seemed more likely to opt for lower-cost treatment options. Study 17 began with a goal of recruiting forty-eight dentists. As this lesson was being learned, we decided to increase recruitment to sixty-three dentists. This was fortunate because it allowed us to finish data collection on the scheduled date.

Discussion

The PBRN research context presents unique challenges. Unlike studies conducted in academic health centers, PBRN studies are conducted by clinicians in community settings and thus by persons who may be doing their first research study. Unique to the PBRN context, a healthy tension exists among multiple factors, including the need to conduct research directly relevant to daily clinical practice, protect confidentiality, provide informed consent, minimize the burden on p-i's and their patients, collect important data that will advance the state of scientific evidence, and provide evidence that can be directly and quickly implemented into regular clinical practice, while at the same time making it all work in a single research project across a diverse range of practice types, clinical settings, and patient populations.

As we have described in this article, many types of lessons were learned: those having to do with interacting effectively with p-i's and staff in a very broad range of practice types and leading a scientific process that must operate in a “real-world” context. A limitation of this report is that our experiences derive from only one network and therefore may not generalize to other networks. Because the DPBRN is both large and very diverse, most of these lessons learned should nonetheless be applicable to a broad range of settings.

Early studies substantially informed later studies with regard to study design, practicality, forms design, data collection process, and training and monitoring methods. Although time-intensive and complex, solutions derived from these lessons substantially improved acceptability of practice-based research to patients, practitioners, and university researchers.

Our self-assessment is that a key to the DPBRN's success has been that it has made a point of listening to p-i's, practice staff, and their patients—listening closely and listening often. The DPBRN has consciously developed a customer service orientation, in which the “customers” are the p-i's, practice staff, and their patients. That is, academic researchers do not “use” the network or its p-i's to do research studies; instead, academic researchers collaborate with p-i's to help meet the p-i's needs and clinical priorities and to engage these practitioners as equals who have crucial contributions to make. The research studies are not ends in themselves; instead, they are means to an end: the improvement of daily clinical practice and oral health. Although the overall goal of the network is to improve the level of oral health in the population at large, the key method by which this is accomplished is by conducting research that is relevant to regular clinical practice—relevance as judged by the p-i's themselves—the results from which can be immediately implemented into regular clinical practice.