Revisiting Practice-Based Research Networks as a Platform for Mental Health Services Research

Abstract

Practice-based research networks (PBRNs)—collaborations of practice settings that work together to generate research knowledge—are underused in mental health services research. This article proposes an agenda for mental health services research that uses a variety of PBRN structures and that focuses on what really happens in practice, the effectiveness of practice innovations in real world care, the challenges of implementing evidence supported interventions, modification of clinician behavior, and assessment of the effect of mental health policy changes on practice. The challenges of conducting research within PBRNs are substantial, including difficulties in maintaining positive member relations, securing ongoing funding, sustaining productivity, overcoming IRB entanglements and achieving both scientific excellence in recruitment and measurement validity and utility for practitioner members. However, the awareness of these challenges allows researchers and practitioners to build networks that creatively overcome them and that infuse mental health services research with heavy doses of the realities of everyday clinical practice.

Introduction

Despite being well suited to address some of the most critical issues in mental health services, practice-based research networks (PBRNs)—collaborations of practice settings that decide to work together to generate research knowledge—are underused in mental health services research. Long used in primary care medicine, PBRNs were only introduced to mental health services in the mid-1990s. Since then, much has been learned about their value and the challenges inherent in their operation. In this article, we introduce PBRNs to the uninitiated, examine the contributions they have made thus far to mental health services knowledge, and describe the implementation and methodological challenges they have encountered. We also propose a research agenda for PBRNs that focuses on examination of usual care, treatment effectiveness, implementation of evidence-supported interventions, and the impact of policy on practice.

What are Practice-Based Research Networks?

Practice-based research refers to the conduct of research and generation of knowledge within natural practice settings (Nutting and Stange 1998). Practice-based research networks (PBRNs) are collaborations of practicing providers who commit to using their work settings as laboratories for practice-based knowledge generation. A single mental health clinic would not be considered a PBRN no matter how much research it conducts, but a statewide collaboration of mental health clinics working together on specific research projects can be considered a PBRN. Similarly, a private practice mental health clinician who carefully tracks client outcomes is not a PBRN, but she could potentially join a PBRN of private practice clinicians that collaborate on joint research projects.

Although PBRNs are quite variable, they all share a few characteristics. First, they generate data reflective of community-based care (e.g., private practices, community mental health centers), not of care received in academic centers or clinics set up specifically for research purposes. Second, the network transcends any one research project. PBRNs form to support the execution of a variety of research projects over time, most of which are not specified at the PBRN’s formation. Third, PBRNs typically represent a partnership between practitioners and researchers. Often, an academic center provides the initial investment in a PBRN and houses its data and staff (Tierney et al. 2007). In such cases, the leadership of the PBRN is mutually shared by representatives of practice and research through co-directorships, executive committees or governing boards consisting of practitioner and researcher partners. To provide guidance and oversight to PBRN activities, members may hold annual or bi-annual meetings to handle administrative tasks, review proposed projects, discuss research findings and decide on future directions (Green and Dovey 2001; Green et al. 2005). Fourth, while PBRNs are often affiliated with an academic center that provides the infrastructure to maintain the network and methodological expertise to execute the research, practitioner members generate or vet research ideas. Thus, the research agenda of any one PBRN evolves over time and strives to remain practice-relevant.

Despite these shared characteristics, PBRNs are diverse in genesis, agenda, membership and leadership. The initial creation of most PBRNs followed an interest in generating practice-based knowledge, but the individual or group that took the first step to form a given PBRN varies. Often, academic researchers with an interest in a specific practicerelated question are the genesis for a PBRN. A treatment developer may start a PBRN of the users of their interventions in order to better understand exactly how their treatment is being implemented and what outcomes are achieved. For example, agencies using Multidimensional Treatment Foster Care (MTFC) contribute data for their MTFC clients each day through a web portal to the treatment developers in Oregon. This could serve as the beginning of a fruitful PBRN of MTFC providers. Similarly, the owner of a proprietary outcomes management system may start a PBRN to exploit the richness of the clinical data obtained across practice settings that use identical measures (e.g., Barkham et al. 2001). Outside of academia, a large professional organization such as the National Association of Social Workers (NASW; https://www.socialworkers.org/naswprn) or American Psychological Association (APA; http://apapracticenet.net) may start a PBRN to tap the experiences of their members and promote the profession. Any large organization that oversees a number of service providers could consider starting a PBRN of their partner agencies to better understand service realities or challenges or to advance a specific initiative.

Focus or agenda is a second way PBRNs vary. Each of the above types of PBRNs may possess differing agendas that may include advocacy for certain causes or products, in addition to knowledge generation. A PBRN can be established on the basis of a narrowly defined problem or issue (e.g., smoking cessation for persons with severe mental illness), a broadly defined problem or issue (e.g., mental health), a discipline or profession (e.g., social work), a funding source (e.g., providers that serve Medicaid clients), or a geographic area (e.g., Medicaid mental health providers in Texas). PBRNs also vary in the complexity and depth of the kinds of data that are collected and shared. At one end of this continuum are networks that use survey methods to periodically query their members on practices and opinions. At the other end of the continuum are agencies or practices that agree to use the same electronic client record system and find ways to jointly share all client data collected in these records with a third party, that manages the data and provides research reports back to network members.

A third way PBRNs differ is the unit of membership. First, individual practitioners can form a PBRN. For example, private practice therapists may form a PBRN to examine therapy practices and outcomes. A group of agencies with a similar mission, such as delivering services to adults with severe mental illness, can also form a PBRN. The agencies provide information to the network and allow themselves to be sites for mental health services research. At a broader level, organizations such as state coalitions of residential treatment programs for children and youth with mental health problems may form a PBRN to study how residential treatment issues are addressed nationally. Here, each state coalition might agree to be the contact point for gathering information from agencies or providers within their state (e.g., Zarin et al. 2005).

Finally, there is variability across existing PBRNs in the extent to which the agenda is set by the practitioners, by the researchers, or by truly shared decision-making. As noted above, PBRNs ideally represent a partnership between practitioners and researchers, wherein practitioners as well as researchers generate and vet research ideas. However, in practice, some PBRNs may operate more to serve the agenda of the researcher or larger organization that initiated the PBRN rather than the collective agenda of the practitioner members.

History of PBRNs

Practice-based research networks are well established in primary medical care. In the 1950s, groups of medical practitioners in England, Australia, New Zealand, and South Africa began collecting and sharing data across practices as an effort to learn from one another (e.g., Del Mar and Askew 2004). This work went largely unnoticed in the United States until the late 1970s, when family medicine physicians began to promote PBRNs within their field (e.g., Green and Hickner 2005). Initially, regional PBRNs were developed (Green et al. 1978) and these were followed soon after by national PBRNs sponsored by professional organizations, such as the Pediatric Research in Office Settings network founded by the American Academy of Pediatrics in the 1980s (e.g., Wasserman et al. 1998). These PBRNs have been funded by a variety of sources, particularly federal agencies like National Institutes of Health. For example, since 2000, the Agency for Healthcare Research and Quality (AHRQ) has funded over 52 PBRNs in primary medical care, a national resource center for their operation, and an annual PBRN research conference. Other NIH institutes also fund PBRNs. For instance, in 2005, the National Institute for Dental and Craniofacial Research committed $75 million to develop dental PBRNs (Ship et al. 2006). AHRQ maintains a web page of research articles generated from PBRNS, listing over 900 as of January 2009 (http://pbrn.ahrq.gov).

Primary care PBRN members have included physicians, nurse practitioners and physician assistants who practice within the primary care office (Tierney et al. 2007; Deshefy-Longhi et al. 2002; Pulcini et al. 2008). Topics studied using PBRNs as the field laboratory include the understanding of certain health screening practices in primary care, such as alcohol screening and discussions. For example, Vinson et al. (2000) utilized the Ambulatory Sentinel Practice Network (ASPN) to examine alcohol related discussions and routine screening.

Historically, PBRNs were formed to answer a practice-based question. Patient presentations or symptoms that were either not well understood, inconsistent, or remained a “mystery” could be examined over numerous practices and physicians to get different viewpoints, analyses, thoughts, interpretations and diagnoses (e.g., Allattar et al. 2007). Similarly, practitioners could evaluate different treatment methods such as various pharmaceutical or surgical interventions for an illness (e.g., Herbeck et al. 2004). As PBRNs evolved, other questions were examined, such as patterns of treatment and referral (e.g., Blanco et al. 2006) quality of care (e.g., Cox et al. 2008) and patient compliance (e.g. Compton et al. 2005). Additionally, the flow of knowledge began to go the other way: that is, instead of practice informing research, research began to inform—or attempt to inform—practice by implementing interventions or evaluating practice. In this way, medical practices became ‘labs’ in which researchers could carry out their projects with the goal of supporting evidence based practice.

In September 2004, NIH solicited an Inventory and Evaluation of Clinical Research Networks (IECRN). As part of the IECRN, an attempt was made to identify all existing clinical research networks in the world (whether or not they were funded by NIH), only some of which were PBRNS, and to collect basic information about the nature and scope of these networks to be freely shared on a searchable database to advertise the existence of these networks and to promote interactivity among the networks, interested practitioners and researchers (www.clinicalresearchnetworks.org). The IECRN also conducted a descriptive survey with a sample of these networks in order to document practices that these networks employ and identify barriers and facilitators that networks experience when carrying out their research. They also identified networks that had been particularly successful (e.g., in recruiting and retaining participants, in changing clinical practice, in accomplishing research) and interviewed leaders of these networks in order to pinpoint practices that allow a network to operate effectively and efficiently. The IECRN is currently working on a tool kit to assist clinical, or practice-based, research networks with planning and executing network research, including sample templates for human subjects applications, consent forms, and conflict of interest policies.

PBRNs within Mental Health

Following the success of PBRNs within primary medical care, the development and spread of PBRNs in mental health began with the American Psychiatric Institute for Research and Education’s (APIRE) Practice Research Network formed in 1993 in order to promote more practice-relevant research in the field of psychiatry. APIRE’s PBRN is a national network of over 700 psychiatrists focused on health services and health policy research. By measures of productivity, longevity, and scholarly contributions, it is also the most successful of all current mental health related PBRNs. According to their website (http://www.psych.org/MainMenu/Research/PracticeResearchNetworkandHealthServicesResearch/PublishedArticles.asp), as of January, 2009 this PBRN had generated over 90 published articles on mental health care, services and policy. It has received funding from the American Psychiatric Foundation, the Center for Mental Health Services, the Center for Substance Abuse Treatment, the John D. and Catherine T. MacArthur Foundation, as well as private pharmaceutical companies. We highlight several findings to demonstrate the breadth and importance of research topics addressed.

First, the PBRN has consistently demonstrated the complexity of real life psychiatric patient care. A study of adult patients with schizophrenia, for example, showed that 41% had a comorbid Axis I disorder, 75% were unemployed and 37% experienced treatment adherence problems (West et al. 2005). One study compared the patients seen in typical psychiatric practice with those seen in groundbreaking clinical trials and found that patients seen in usual care by PBRN psychiatrists were more likely to have more comorbid conditions, be female, white and older than patients seen in clinical trials. Thirty-eight percent of PBRN patients with schizophrenia and 55% of patients with bipolar disorder would have been excluded from the clinical trials (Zarin et al. 2005). Another study examined what psychiatrists do with patients with schizophrenia who are non-responsive to an antipsychotic medication. While most psychiatrists switched medications, in one-third of cases they added another antipsychotic medication. Switching was reported by members to have been more effective (Kreyenbuhl et al. 2007).

Numerous APIRE PBRN studies have also unveiled problems with the quality of care. A substantial racial disparity in the use of second generation antipsychotic medications was revealed in one survey (Herbeck et al. 2004). In another, it was shown that 39% of patients with obsessive compulsive disorder were receiving dosages of a serotonin reuptake inhibitor considered to be too low to be effective (Blanco et al. 2006). A third showed problems in the treatment of patients with comorbid depression and substance use disorders, including potential overuse of benzodiazepines and under-treatment of the substance use disorders (Montoya et al. 2000). The PBRN has also been successful at identifying some influences on practice variation. For example, frequent contact with pharmaceutical representatives was associated with more favorable attitudes toward second-generation antipsychotics (Arbuckle et al. 2008) and with the tendency to add antipsychotic medications (as opposed to switching) in nonresponsive patients with schizophrenia (Kreyenbuhl et al. 2007).

Finally, access issues have been highlighted in several studies by this PBRN. For example, a survey of psychiatrists showed that while most psychiatrists (77%) would accept new self-pay patients, only 44% would accept a new Medicaid patient and this rate was even lower for psychiatrists who were white, board certified, or graduates of US medical schools (Wilk et al. 2005). As an example of how a PBRN can respond to emerging policy changes, the PBRN was used to assess psychotropic medication access and continuity in the first 4 months of the implementation of Medicare Part D’s prescription drug benefit (West et al. 2007). Psychiatrists reported that 10% of eligible patients seen by psychiatrists experienced improved medication access, but that 18% of medically stable patients were required to change medications, 22% discontinued or temporarily stopped taking a medication due to coverage or management issues, and that 27% of these patients experienced a significant clinical adverse event as a result.

Since the development of APIRE, the American Association of Marriage and Family Therapists (AAMFT), the NASW, the APA, and the American Counseling Association (ACA) have all formed guild-specific PBRNs. Each of these PBRNs began with funding from the Center for Substance Abuse Treatment to conduct surveys of their members to assess their involvement in substance abuse treatment. The ACA also used this money to learn more about its members’ practice patterns and behaviors and to develop a system for PBRN members to enter client data following sessions, track clients over time and compare their clients’ outcomes with those seen by other therapists (Smith et al. 2005). Key findings from the ACA PBRN have been descriptions of current practicing counselors and their caseloads. For example, counselors largely based their treatments on theory (39%), experience (28%) or other decision making tools (19%), 13% of their clients have substance abuse problems, 76% of their clients were self-paying, and a pre-post analysis of symptom change in counseling sessions revealed a statistically significant difference (Smith et al. 2005).

According to the NASW website, its PBRN was used for two surveys to characterize fields of practice and compensation and professional encounters with substance abusing clients (https://www.socialworkers.org/naswprn). Results suggested that about 30% of NASW members practiced clinical social work in a private practice setting. About 30% of clients seen in agency settings and 20% of clients seen in private practice had substance abuse problems (Smith et al. 2006).

American Psychological Association’s PBRN has posted results from five surveys on its website, http://apapracticenet.net/ with the most recent survey results from 2003 (Lynn Bufka, of the Association, indicates plans for the PBRN to be reactivated in 2009, December 22, 2008, personal communication via email). Similar to that reported by the ACA PBRN, 12% of clients seen by psychologists had an active substance abuse problem. The most common treatment strategies that psychologists employed with substance abusing clients were cognitive behavioral therapy, motivational interviewing, relapse prevention and harm reduction. Overall, 17% of clinical psychologists said they practiced from a psychodynamic perspective, 28% from a cognitive behavioral perspective, and 45% from an eclectic combination of perspectives. Psychologists reported that 37% of their clinical sessions involved a discussion of spirituality. About half of psychologists’ therapy clients were also receiving psychotropic medications. To our knowledge, none of these results have been published in peer reviewed journals but they are used to inform APA’s Practice Directorate in their advocacy efforts and they are available on their website.

American Association of Marriage and Family Therapists has surveyed its members at least twice using its PBRN. Northey (2002) found that 74% of its members worked in a private practice at least part time, that couples problems were the most common client presenting issue, and that 10.5% of private practice and 20% of organizational practice clients had substance abuse problems, but they were rarely seen as the primary clinical issue. Morris (2007) characterized the practice of a small number of rural AAMFT members. Finally, Northey (2005) reported on the rates of AAMFT members who reported using specific interventions.

We know of at least two additional practice research network in mental health. The Pennsylvania Practice Research Network (PPRN; Borkovec et al. 2001) was started by a clinical researcher and a clinical practitioner in the 1990s and was initially supported by funding from the APA and the Pennsylvania Psychological Association. To date, this PBRN has conducted two studies examining therapist and client characteristics, client progress during outpatient therapy, and specific events within therapy that are helpful or hindering to client progress (Borkovec et al.2001; Ruiz et al. 2004; Castonguay et al. 2007) and is preparing to launch a third examining the impact of providing session-by-session feedback to therapists about the techniques their clients find most helpful (Parry et al.2009). Therapists and Researchers: Advancing Collaboration (TRAC) is a practice research network that began in 2003 as a therapist advisory group to help guide the development and execution of a specific practice-based research study of usual care practices for children with disruptive behavior problems (Garland et al. 2006). It has since evolved into an ongoing collaborative network of therapists (marriage and family therapists, psychologists, social workers) and researchers with a shared interest in conducting feasible and clinically relevant research within the local practice context (see http://www.casrc.org/pro jects/PRAC/expo08/trac.html). TRAC has completed their initial study on therapy processes for children with behavior problems (Garland et al. 2008, 2006) and is preparing another examining clinical supervision practices (Ann Garland, April 2009, personal communication). In addition to the declared PBRNs described above, research generated by cooperating practitioners in connection with an academic researcher may have been conducted within a PBRN framework, although publications resulting from the partnership have not identified the partnership as such. As examples, both Barkham et al. (2001) and Lambert et al. (2001) have described outcome data from thousands of outpatient psychotherapy clients from many service sites using uniform outcome measurement packages.

What are the Advantages of Practice-Based Research Networks?

A number of scholars have enumerated the advantages of using PBRNs. First, as illustrated by the various practice questions they can answer and the methods they can use, PBRNs are flexible; they can address a variety of issues in a variety of ways (e.g., Nutting and Stange 1998). Second, PBRNs are efficient; studies can often be put together quickly and cheaply because a ready pool of participants has agreed to be contacted and to submit data through efficient means, such as fax and email (e.g., Nutting and Stange 1998). Third, PBRNs provide access to practical experience; via PBRN meetings, practitioners can inform researchers about what will and will not work in their settings, thereby saving time and effort in building and adapting interventions and study protocols (Nutting et al.1999). Fourth, PBRNs help forge a practice-informed research agenda; practitioners can tell researchers about the things that matter most to them, nominate topics for study, help adjust study design, contribute data, help interpret results and in some cases, help analyze data. Finally, PBRNs can help make mental health services a true learning discipline providing practitioners with the capacity to conduct research in addition to providing direct services (e.g., Nutting and Stange 1998; Nutting et al. 1999) and fulfill the scientist-practitioner goal of the Boulder model in psychology (Kendall and Hudson 2001).

What Challenges do Practice-Based Research Networks Face?

Although PBRNs may ultimately yield improvements in both the practice-relevance of clinical research and the research-basis of everyday clinical practice, they also face several challenges in their operation. We detail six primary challenges to realizing the potential of PBRNs below. Potential solution to these challenges are shown in Table 1.

Table 1.

PBRN challenges

Challenge	Potential solutions
Managing relationships  with PBRN members	Established communication infrastructure, budget for  communications, interactive web site, newsletters, open meetings
Ongoing financial support	Institutional support; study budgets to include ongoing management  and infrastructure support
PBRN productivity	Include team members who write for publication; review potential  projects for committed authors and scientific interest beyond the  PBRN members
Responsible conduct  of research issues	Budget money and time to account for IRB complexities and the  training of PBRN members in the responsible conduct of research
Recruitment and  generalizability	Incentivize participation; don’t restrict recruitment to atypical  members
Measurement validity	Recognize potential for measurement problems, pilot measurement  protocols, conduct mini-validity studies

Managing Relationships with PBRN Members

Practitioners who choose to join PBRNs as members are investing some of their time and resources into a joint research enterprise. As such, they are more than simply research subjects—they are partners in the research. This requires specific efforts to manage the relationships between PBRN practitioner-members and researcher-members (Baxley and Stanek 2007; Green and Dovey 2001; Love et al. 2005) and necessitates frequent opportunities for communication and exchange. There are needs for practitioner-members to inform the PBRN’s research agenda (Lindbloom et al. 2004), to provide input into study design and results interpretation (Genel and Dobs 2003), and to receive and discuss the importance of research findings. There will likely be a need to periodically address and manage differing expectations from academic researchers and clinicians (Pulcini et al. 2008). This may involve clinicians teaching researchers about the realities of their practice lives or researchers training clinicians on some aspects of research design and measurement (Cardarelli et al. 2007; Genel and Dobs 2003). PBRNs likely involve requests for action from busy practitioners (e.g., to complete a survey or submit client data) that may require timely messages delivered in multiple formats and through multiple means (Deshefy-Longhi et al. 2002). Some clinicians may need personalized attention to complete their tasks or to become fully informed members (Ornstein 2001). Such regular communication and engagement between members requires a well-designed and implemented infrastructure. Needed resources might include ample budgeted time from a project manager, web sites with interactive features, automated email notification systems, annual open meetings, and newsletters detailing findings from previous PBRN studies and describing upcoming or underway studies.

Ongoing Financial Support

By design, PBRNs are expected to endure over time and to involve multiple research studies. Studies, however, are typically funded one at a time and slowly at that. PBRNs often report difficulty in maintaining ongoing financial support that allows them to be functioning enterprises between study funding (Green and Hickner 2005; Han 2000; IECRN Executive Summary 2006; Lindbloom et al. 2004; Pulcini et al.2008; Tierney et al. 2007). To survive, PBRNs may require a financial source beyond funded studies, such as a minimal institutional commitment (Clothier 2005). PBRN’s budget needs for ongoing project management, membership communication, data management, practitioner incentives and research assistants are often underestimated (Graham et al. 2007). Borkovec et al. (2001) recommend reimbursing clinicians for lost clinical time and paying clients for completing any necessary assessments, costs often overlooked in PBRN budgets. Graham et al. (2007) emphasize a need to select among all solicited study ideas only those that are most fundable, accentingagain the need to manage expectations between PBRN practitioner-member and researcher-members.

PBRN Productivity

Not all PBRNs are productive, in terms of completing research studies or publishing their results. A recent survey of primary care PBRNs revealed that 27% had not yet completed a research study, while about 40% had completed between four or more research studies (Tierney et al. 2007). The number of PBRNs that consistently contribute to the professional literature has remained small. An informal review of the 900? articles posted on the AHRQ’s PBRN resource website reveals that a few PBRNs are responsible for the majority of publications. These PBRNs were not necessarily the older, more established networks. The IECRN report notes lack of time for data analysis and writing for publication as frequently encountered challenges (IECRN Executive Summary 2006). Additional challenges may include a lack of experience among network members in writing for publication, a focus on local questions that are not generalizable outside the network, and potentially a lack of understanding among editors and reviewers about practice-based research. APIRE’s leadership attributes their high productivity to a team of professional researchers who have worked well together over time and the network’s ability to generate rich clinical databases through relatively efficient means (personal communications via email with Joyce West and Eve Miscicki, April 6, 2009).

We offer three modest suggestions for ensuring PBRN productivity. (1) PBRN developers should ensure that some PBRN team members are knowledgeable about and experienced in publishing research findings in peer reviewed journals. (2) In determining whether a specific study should be initiated, PBRNs should consider as decision-making criteria whether there is a (a) PBRN team member willing to lead the study scientifically and (b) one who is committed to lead authorship of papers from the data to be collected. Practitioner members may assume that an academic partner will be eager to publish from study data, while the academic partner may think the study is not closely enough aligned with his or her interests or career trajectory. (3) Similarly, PBRN teams should consider the degree to which proposals for new research offer the potential for generalizable knowledge that will be of utility to an audience wider than the PBRN. It may be easy for some PBRNs to sanction new studies based on current enthusiasm levels without a careful consideration of the scientific knowledge that can be gained.

Responsible Conduct of Research Issues

Unique ethical and human subjects issues can arise with PBRN-based research (Graham et al. 2007; Neale and Schwartz 2004; Wolf et al. 2005; Wolf et al. 2002). If a PBRN simply surveys network members, the PBRN may pose few ethical challenges beyond what is familiar to experienced researchers and Institutional Review Boards (IRBs). If data collection is via the web or fax, extra precautions are needed to protect the security of the data. PBRNs that use practitioner members to collect information from organizations or clients will likely encounter more thorny issues. For a moderately complex example that illustrates several points, let’s say an academic researcher works with a public mental health department to help start and administer a PBRN of 50 community mental health organizations operating in a large state, with the academic partner serving as PBRN Research Director. It is agreed early on that the clinical directors of these 50 agencies will be the primary contacts for the PBRN. Let’s imagine that as prelude to a new state effort to improve care for clients dually diagnosed with substance use and bipolar disorders, the PBRN wants to collect information on usual care services and clinician and client preferences at the 50 facilities. The member agency clinical directors are asked to (1) provide information on the agency’s clinical training, policies, and service structure for dually diagnosed adult clients, (2) to distribute a short pencil and paper questionnaire to all psychiatrists who work at their facilities, (3) provide electronic deidentified information on past year service data for all patients who meet inclusion criteria, and (4) to coordinate the distribution of client surveys via agency case managers to agency clients that meet inclusion criteria.

Multiple issues arise in this scenario, including the use of multiple IRBs, determination of who are the researchers and researcher participants and the collection of private health information. Some of these organizations may have their own IRBs. Some may not. The use of multiple IRBs will likely lead to inconsistent requirements and delays (IECRN Executive Summary 2006; Wolf et al. 2005). A number of agreements may need to be reached, where one IRB oversees work at an agency without an IRB and/or where some IRBs agree to accept another IRB’s oversight (Wolf et al. 2002). Most IRBs would agree there are three categories of research subjects active in this scenario: the clinical directors, the psychiatrists, and the clients. There are also several categories of people serving in a research role, including the PBRN leaders, the clinical directors, and the case managers who distribute and collect questionnaires. IRBs will likely want assurances that the people serving in research roles have been trained in the ethical conduct of research. Who will provide this training and how much is required? Different IRBs will have different opinions. Finally, the example involves the collection of private health information. Will the IRBs grant waivers of patient authorization to disclose private health information in this circumstance? What data will the IRBs allow in the deidentified data set? IRBs may also differ on answers to these complex issues. One set of researchers offered what may be the most practical advice given these complex IRB issues: budget money and time to account for IRB complexities and the training of PBRN members in the responsible conduct of research (Wolf et al. 2002).

Recruitment and Generalizability

Findings from PBRN research have the potential for high generalizability, but only if the PBRN is able to motivate practitioners to participate to minimize any differences between volunteers and decliners (Baxley and Stanek 2007; Clothier 2005; Norquist 2001; West et al. 1998). Incentives that are relevant to the practitioner may need to be developed, such as continuing education credits and reimbursement from lost clinical time (Borkovec 2002). PBRN leaders are cautioned not to restrict recruitment to atypical practices, such as local leaders or colleagues of the PBRN leaders (LeBailey et al. 2003). In addition, PBRN leaders are encouraged to recognize that a sample that is representative when a study is designed may be less so by the time results are reported (LeBailey et al. 2003). Finally, the activities of the PBRN may, over time, make the participant practitioners increasingly unique from their non-participating peers, especially if PBRN membership provides ample opportunity for training or other practice improvements. This is especially important in clinical trial research, where PBRN members’ behavior in one trial may be influenced by their participation in previous trials.

Measurement Validity

Practice-based research networks often rely heavily on self-report from practitioners, typically with no validation of key variables (Fagnan et al. 2007; Montoya et al. 2005). For example, clinicians may be asked to report on the interventions they provided to a recent client with a specific diagnosis, but there is no independent validation of that client’s diagnosis or the services delivered (Herbeck et al. 2004). In AAMFT’s first PBRN survey, for example, its members reported exceptionally high rates of using interventions for which few practitioners had been trained. For example, 69% said that they used Multisystemic treatment. In the second survey, they associated a treatment developer’s name with the intervention in order to improve validity due to misunderstanding generic sounding terms; 12% said that they used “Scott Hengeller’s Multisystemic Treatment,” a number that is still high enough to create validity concerns. Networks that attempt to capture client data face additional challenges. At times, busy practitioners with no or little research training are asked to report data collected from their clients, introducing additional validity concerns (Zarin et al. 1997). PBRNs may ask clinicians to institute specific measurement systems into their practice, but should recognize that the literature is full of warnings that these systems need to be clinician friendly, clinically relevant, and compatible with practice preferences to get clinicians to adhere to their use and yield valid data (Clothier 2005; Genel and Dobs 2003; Kho et al. 2007; LeBailey et al. 2003; Lindbloom et al. 2004; Van Weel et al. 2000). Efforts to mitigate potential measurement validity problems include (1) piloting measurement protocols to work through problems before the larger study is launched, and (2) conducting small validity pilots to assess whether self-report measures correlates with client report or case record data.

A PBRN Research Agenda for Mental Health Services

Despite these challenges, the fact that PBRNs focus on settings where services actually take place make them ideal platforms for addressing some of the most important and vexing issues in mental health services research. Below, we outline a proposed PBRN research agenda based on six research questions.

What really happens in practice?

Mental health services possess an extraordinary number of “black boxes”, situations where we know that something is happening, but it is unclear what is happening. What happens when a person with mental illness meets with a case manager? What happens in the therapy room? Without knowing what clinicians do, it will be difficult to know what to ask them to continue doing and what to ask them to change to become more effective practitioners. PBRNs can help describe the patterns and practices of direct care (Nutting et al. 1999) and what client, clinician and contextual characteristics influence these practices (Kolko 2006), therefore opening the black boxes that constitute much of mental health practice.

Studies utilizing PBRNs have examined the existence and consistency of screening of certain illnesses or behaviors, such as alcohol use (e.g., Vinson et al. 2000). Garland and colleagues examined the black box of usual therapy practice for children with disruptive behavior problems and the extent to which usual practices are consistent with evidence-based practices. Their work is ongoing but preliminary findings indicate that parent attendance and involvement in treatment, critical precursors to evidence-based therapy for child disruptive behavior problems, are a challenge in community mental health services (see http://www.casrc.org/projects/PRAC/expo08/sessions.html). Kelleher et al. (1999) examined the recognition and treatment of behavior problems in children in primary care settings; specifically, whether or not clinicians were biased with regard to patient ethnicity and race when treating psychosocial problems in minority children. Utilizing the ASPN and Pediatric Research in Office Settings (PROS) PBRNs, they found that in fact race/ethnicity was not related to mental health services utilization in primary care settings. Similarly, Wilk et al. (2005) examined the patterns of Axis I and comorbid diagnoses and detection among psychiatrists in the APIRE PBRN. They compared data from the APIRE sample to a clinical subset of patients in the National Comorbidity Study who were treated by other mental health specialists. Results indicated potential differences in diagnostic assessment between psychiatrists in the APIRE sample and other mental health clinicians (Wilk et al. 2005).

As an example of a potential PBRN study that asks a “what really happens in practice?” question, a PBRN of psychiatrists, with proper permissions and protections, could use webcam technology to record 15 min medication check-ups with mental health consumers. A database of such encounters could be mined and coded by research teams to assess a number of critical issues. They include interviewing style and effect on patient disclosure, the amount of time spent discussing certain topics (side-effects, medication effectiveness, barriers to medication use), racial disparities in topics discussed, and communication between consumers and US born and foreign born clinicians.

Do specified evidence-supported interventions or practice guidelines work when implemented in front-line care?

PBRNs can demonstrate whether treatments with proven efficacy are effective and sustainable when applied in the real world (Westfall et al. 2007). Effectiveness research by definition requires real-world practice settings. PBRN members, who have already identified themselves as willing to participate in research can provide these settings. PBRN members with an interest in learning a new mental health intervention can be randomized to receive training (say now, or a year later) and deliver the intervention. Clients recruited from practices in the treatment or delayed training condition can be recruited into a clinical trial of the interventions effectiveness as delivered by the PBRN provider.

What are the real life challenges to implementation?

Every innovation, policy change, practice guideline, and evidence-supported treatment requires implementation in real world practice settings in order to bring about the desired public health benefit. Proper implementation of evidence-supported interventions in mental health poses extra-ordinary challenges (Proctor et al. 2009), yet may be one of the most important issues in our field. It is estimated that new effective interventions languish on the shelf for 15–20 years before incorporation into usual care and once they do reach usual care are often poorly and inequitably delivered (e.g., US Department of Health and Human Services 1999, 2001; Institute of Medicine 2006). Mental health practitioners are ideally suited to inform researchers about the real-life challenges to implementation. They can be queried prior to implementation to discover potential pitfalls so that implementers can mediate the risk of problems, and they can be queried during or after implementation to ask what went right and what did not to improve future implementation efforts. PBRNs can identify the problems that arise in daily practice that create the gap between recommended care and actual care (Westfall et al. 2007). As an example, a network of community mental health centers operating in one large state may be interested in bringing the JOBS program (Vinokur et al. 1991), a group-based psychoeducation and reemployment intervention, into their centers. Administrators and clinicians from across the network could be brought together for an on-line and phone conference call focus groups, where they are presented with aspects of the JOBS curriculum on their computer screens and asked to talk about potential barriers to implementation of the program in their centers and communities. During the initial implementation of JOBS, provider clinicians could be brought together again to discuss the lived experience of delivering the intervention and to make suggestions for future JOBS implementation efforts.

How can practitioner behavior best be modified and maintained?

Most practitioners attempt to improve their practice through attending workshops and independent reading, but these may possess limited efficacy. For example, studies of physician behavior change have typically shown that the mere provision of information has little impact on practice (Davis et al. 1995; Goldberg et al. 1998; Horowitz et al. 1996). Mental health practitioners involved in a PBRN may be ideally placed for the development and testing of strategies to change clinician behavior and improve client outcomes. As one example, prominent mental health services researchers have advocated for and developed measurement-based clinical feedback systems within a quality improvement framework as a method of practitioner behavior change (e.g., Bickman 2008; Lambert et al. 2005). In such systems, clinicians receive regular feedback about client outcomes and warnings of failure to see treatment response in an effort to prompt an adjustment of clinical strategies. With such systems, clinicians can also receive feedback that compares their performance with the outcomes achieved from other clinicians (Bickman et al. 2006). A PBRN could implement a feedback system for and with providers, provide summative comparison data to clinician members seeing similar clients, and test whether the implementation of such a feedback system results in improved client outcomes.

Are PBRNs effective dissemination vehicles for evidence-supported interventions?

Another way clinician behavior is changed is through the adoption of evidence-supported interventions. PBRNs can serve as the dissemination vehicle for evidence-supported treatments. In other words, PBRNs can be the laboratory for studying system improvements related to the implementation of evidence-supported interventions (Westfall et al. 2007). As an example, PBRN of mental health clinicians who work in rape crisis centers could be trained in delivering cognitive processing therapy for trauma victims (Resick and Schnicke 1993). These clinicians could be assigned to receive differential supervision supports, or differential incentives to participate in training. Implementation outcomes could serve as the dependent variables. These outcomes might include acceptability of the intervention to the clinicians and consumers, the clinicians’ decisions to adopt cognitive processing therapy in practice, the degree of penetration into the clinician’s everyday practice, and the degree of fidelity to cognitive processing therapy.

How do policy changes affect everyday practice?

Mental health agencies and practices are subject to frequent policy changes from funders, regulators and accreditors, who frequently demand more and offer less. PBRNs can be positioned to study how policy changes affect practice in real world settings, following the example of the American Psychiatric Association’s study of Medicare Part D implementation (Westfall et al. 2007). PBRNs can respond more quickly to changing policy environments because they have already recruited a willing participant pool. For example, if a state Medicaid agency decides to implement a new on-line prior authorization procedure for mental health services for children with Medicaid, a PBRN of state Medicaid mental health practitioners could study member clinicians’ experience with the on-line system, perceived increases or decreases in clinician burden, and self-reported changes in willingness to serve Medicaid clients as a result of the new procedure.

What Types of Research Methods Can PBRNs Use?

Given the creativity of its members and leaders, there is no limit to the types of different research studies that can be conducted through a PBRN. We mention six.

Surveys

Once a PBRN is established, surveying its members can be cost effective, because its members may require little convincing to participate and can often be surveyed using the least expensive means, such as web or faxed paper and pencil surveys. To enhance the reliability and validity of clinician self-report, PBRN members can also be recruited to engage in real-time behavior sampling (RTBS) or ecological momentary assessment (EMA) which allows practitioners to report on activities or experiences close in time to the actual event and provide these snapshots at various point during the day or during the course of treatment with a given client.

Qualitative Interviews

Practice-based research network members can be solicited to provide more indepth information and opinions or to respond to specific stimuli via focus groups or individual qualitative interviews. Group interviews could take place in person if the practitioners are local, or via a conference telephone call or with web technology if they live in a wider geographic area.

Observation of Actual Practice with Consumers

Given the potential validity problems of self-reported treatment delivery, PBRNs could become more creative in their methods to explore the black boxes of clinical care. With proper human subject protections, researchers could observe PBRN member practitioners in interaction with consumers to study the realities of everyday practice in naturally occurring settings. This observation could be inperson, via audio recordings, video recordings or recorded via web technology.

Chart Review Information

Given the permission of consumers, practitioners could submit data from client records to be shared with the research arm of the PBRN. As mentioned, groups of clinicians may agree to use identical electronic records or outcomes systems from which information can be gathered.

Clinical Trials

As mentioned, PBRNs can also be the vehicle for mounting effectiveness clinical trials of promising interventions (Nutting et al. 1999). This provides a mechanism to move clinical trials from academic centers into real world care and provides the advantage of training community practitioners in cutting edge practice.

System Improvement Trials

A number of system improvements can be studied in randomized trials with PBRN members recruited and randomized into the different conditions. The proposed improvements could be simple, such as the introduction of screening tool into clinical practice. Or, they could be more complex, like the integration of measurement feedback systems into clinical care (Bickman 2008).

The Future of PBRNs in Mental Health Services Research

We can imagine a number of creative PBRNs addressing important mental health service research agendas in novel ways. This will require that a variety of entities start and fund PBRNs with a full understanding of their potential, the common pitfalls and needs for infrastructure and scientific support.

There are several potential candidates for starting PBRNs, some of which we have already mentioned: professional guilds, advocacy coalitions, treatment developers, outcomes systems managers, state departments of mental health, research academics, and currently funded research centers that could add a PBRN to their research mix. Each of these candidates must ask themselves hard questions. What can be learned? What is the value of this potential knowledge? Will the knowledge gained be important enough to the PBRN members to sustain their involvement? Do we have the research expertise and capacity to conduct these studies? Do we have the institutional commitment and the financial flexibility to fund the PBRN in the absence of external resources?

To date, the National Institute of Mental Health has not invested in PBRN research to the extent of some of its sister institutes. We would like to encourage more NIMH funding for PBRNs to examine this proposed research agenda, but the challenges of PBRN research and continuity suggest that this investment be measured. Current NIMH funding mechanisms can support PBRNs. Most notably, the R24 Intervention and Practice Research Infrastructure Program (IP-RISP) mechanism (Program Announcement Number 06-441) is structured to support research partnerships between community-based clinical settings and research institutions and could be ideal for funding the initial years of a PBRN. The P30 Advanced Centers for Interventions and/or Services Research mechanism (Program Announcement 08-088) could be used to fund PBRNs that serve the research agenda of the larger center. A variety of mechanisms could be used to fund specific studies of existing PBRNs. Regardless of the mechanism, these PBRNs must propose work that is scientifically rigourous enough to pass a demanding peer review process, meaning they must address issues of generalizability and measurement validity that can plague some PBRNs.

State departments of mental health with dedicated funding for research could consider self-funding PBRNs. State legislatures may consider funding PBRNs if they can be convinced that they will lead to service improvements that will eventually re-coup their investments. Private foundations and associations may consider funding PBRNs if the PBRNs squarely fit their missions.

The past 15 years have proven that PBRNs can be important mechanisms for investigating crucial issues in mental health services research. But these years have also shown that PBRN research is not easy to do well and that PBRNs require substantial care and feeding. The entitites that may consider starting or funding PBRNs must be aware of the difficulties in keeping PBRNs scientifically productive and their research scientifically valid and be sure that the PBRNs have sufficient support and scientific expertise. However, we believe that in many cases the scientific and practice benefits that can be realized by using mental health PBRNs justify the investment.