AccrualNet represents a unique, centralized comprehensive-solution platform to systematically capture accrual knowledge for all stages of a clinical trial.
Abstract
Purpose:
Present the design and initial evaluation of a unique, Web-enabled platform for the development of a community of practice around issues of oncology clinical trial accrual.
Methods:
The National Cancer Institute (NCI) conducted research with oncology professionals to identify unmet clinical trial accrual needs in the field. In response, a comprehensive platform for accrual resources, AccrualNet, was created by using an agile development process, storyboarding, and user testing. Literature and resource searches identified relevant content to populate the site. Descriptive statistics were tracked for resource and site usage. Use cases were defined to support implementation.
Results:
AccrualNet has five levels: (1) clinical trial macrostages (prestudy, active study, and poststudy); (2) substages (developing a protocol, selecting a trial, preparing to open, enrolling patients, managing the trial, retaining participants, and lessons learned); (3) strategies for each substage; (4) multiple activities for each strategy; and (5) multiple resources for each activity. Since its launch, AccrualNet has had more than 45,000 page views, with the Tools & Resources, Conversations, and Training sections being the most viewed. Total resources have increased 69%, to 496 items. Analysis of articles in the site reveals that 22% are from two journals and 46% of the journals supplied a single article. To date, there are 29 conversations with 43 posts. Four use cases are discussed.
Conclusion:
AccrualNet represents a unique, centralized comprehensive-solution platform to systematically capture accrual knowledge for all stages of a clinical trial. It is designed to foster a community of practice by encouraging users to share additional strategies, resources, and ideas.
Introduction
The problem of low accrual to oncology clinical trials has been known for decades.1–2 Yet low accrual to trials persists,3–5 with an unacceptable percentage of National Cancer Institute (NCI)–sponsored trials failing to achieve minimum accrual goals.6–9
NCI's Office of Communications and Education (OCE) has researched factors related to accrual and recruitment best practices for clinical trials, including structured interviews with health association leaders, in-depth studies of five comprehensive cancer centers, and more than 15 community clinical oncology programs (CCOPs). Four specific themes arose from this research: (1) despite numerous published articles and strategies developed at the local level, clinicians require assistance identifying best practices and strategies to improve accruals to clinical trials; (2) existing tools and resources are scattered across multiple media and require extensive work to retrieve and render them useful for local conditions; (3) efforts to improve enrollment to trials are generally point solutions, that is, they are developed to solve a particular problem at a single site; and (4) there are limited opportunities to share best practices and collaborate among the community of oncology trialists.
To address these four themes, OCE linked two theoretical concepts: diffusion of innovations and community of practice. With respect to diffusion of innovations,10 literature searches and published reports showed that innovative solutions to address low accrual did exist. Yet OCE's field research indicated that such solutions were difficult to identify and poorly disseminated; in effect, the ideas failed to reach the clinicians who most needed them. According to Rogers (2003),10 an innovation fails to diffuse if it (1) has limited relative advantage (its value does not surpass that which it is to replace), (2) is incompatible with the adopters' workflow, (3) is too complex, (4) cannot be tried or experienced vicariously before adoption, and (5) is not visible (or known) to those most likely to adopt it.
The second concept—building a community of practice—stemmed from clinicians' stated desire to be better connected with their peers and to identify opportunities to share new accrual approaches. A community of practice is a group that evolves, either naturally or intentionally, because of its members' similar interests in gaining and sharing knowledge in their field.11 Critical aspects for success are the identification of a group of individuals who share a common interest, have access to “crucial resources for increasing participation,” and create and share knowledge to “attack common problems.”11–12
OCE was interested in both improving the diffusion of evidence-based accrual practices and accrual innovations, and building a community of practice around clinical trial accrual strategies to better meet the needs of oncology clinical researchers. A viable approach was to develop a user-friendly online platform that could both function as a comprehensive repository of clinical trial accrual strategies and offer clinicians a mechanism to connect and coalesce around best practices.
In this article, we present the design, development, and initial evaluation of an online, interactive platform developed to meet the principles of successful diffusion and community of practice. Entitled AccrualNet (https://accrualnet.cancer.gov),13 its mission is to provide a platform on which a community of oncology clinical trialists can discover, collect, and disseminate best practices toward improving overall participation in oncology clinical trials. This article helps illuminate what is possible with such a platform and discusses the most effective and efficient use of AccrualNet to address accrual problems.
Method
AccrualNet was created by using established marketing and user-centered research practices, described below.14–15 A steering committee composed of NCI and field experts in health communication, clinical research, systems design, and usability was responsible for all aspects of the site's strategy, design, implementation, and piloting.
Agile Platform Development
To avoid the lengthy and costly process of traditional software development, the steering committee opted for an agile platform development approach, which included frequent and rapid delivery of software iterations; openness to change at all points in development; close and daily contact between the steering committee and developers; and constant attention to technical excellence, good design, and simplicity.16
Information Architecture
To ensure that AccrualNet content was organized, structured, and labeled in a manner intuitive to its intended users (ie, its information architecture),17 the steering committee used flowcharts and storyboarding (ie, series of graphic illustrations) to categorize the platform's information and map it onto a complete, hierarchical structure showing the overall flow of information.18–19 The platform's draft information architecture was then shown to five experts in the oncology field for recommended changes and validation.
User Testing
Stakeholder input is critical to develop an innovation that meets the characteristics of successful adoption.10 During AccrualNet's creation, user testing was conducted with 27 potential users, representing cancer centers and community oncology practices and a variety of job positions (eg, principal investigators, clinical research coordinators). Five rounds of testing explored (1) responsiveness toward the platform concept and framework, (2) understanding of the platform's functionality, (3) input on designs and navigation, (4) usability testing of a live prototype, and (5) ways to fine tune the final platform.
Use Cases
Use cases are representative descriptions of possible ways users could, or expect to, interact with a software or platform. Use cases are based on user feedback and describe scenarios to highlight the most likely ways users will use and benefit from the platform.
Selecting Content
We used three methods to identify relevant resources: literature searches, e-mail requests to members of the target audience to solicit samples of tools or strategies, and a review of existing accrual resources at NCI. On selection, materials were classified as a journal article, report, tool, or other resource, and a one-page description was developed to synopsize each resource's highlights, details, and usefulness to clinicians.
Results
AccrualNet was launched on April 30, 2010 at the NCI-ASCO Cancer Trial Accrual Symposium: Science and Solutions in Bethesda, MD. It was presented as a resource and professional community for achieving improved clinical trial accrual.
Description of Platform
Functions.
AccrualNet is a Web-based platform intended to support improving accrual activities for oncology clinical trials. It provides (1) a framework to guide accrual planning throughout the life cycle of a trial; (2) a searchable central repository of hundreds of accrual resources; (3) a training portal supplying resources for professionals and patients to learn more about accrual and promising practices; and (4) a conversation portal where clinicians can ask questions, share observations, and promote best practices among their peers (Appendix Figure A1).
Architecture.
To reflect user research on how clinicians seek accrual support, the site's information architecture centers on the life cycle of a typical clinical trial (ie, users said, “I may not know what I need but I know when I need it”). Figure 1 charts the final outcome of the storyboarding activity that reflects the life cycle hierarchy, and Appendix Figure A1 depicts this same chart as a colorful, simple life-cycle wheel. Atop the hierarchy are the macrostages: prestudy (blue), active study (green), and poststudy (gold). Seven substages exist: three in prestudy—(I) developing a protocol, (II) selecting a trial, (III) preparing to open the trial; three in active study—(IV) recruiting and enrolling participants, (V) managing the trial, (VI) retaining participants; and one in poststudy—(VII) evaluating accrual and lessons learned. Drilling down, each substage has a set of affiliated strategies listed. The next level in the hierarchy is activities, specific action items to consider for each strategy. Finally, for each activity, the site connects the user to resources to help them carry out the activity (eg, annotated journal articles, sample tools).
Figure 1.
Final storyboard for AccrualNet information architecture.
Navigation.
By keeping content focused on accrual issues and having only four portal tabs at the top, the overall design of AccrualNet is uncluttered and easy to navigate. The site uses established health communication techniques14–15 to aid quick grasp and retrieval of key content (eg, white space, bullets, bolded words). The life cycle design and color scheme provide a navigational hierarchy to allow rapid point-to-point movement within the entire platform. The simplicity of the circular design is also maintained as a user moves into the strategies and activities (Appendix Figure A2). Additional visual cues are provided throughout the site to subtly remind users where they are in the trial life cycle.
Search.
The platform offers multiple search strategies to accommodate users' preferences in accessing content. Users can maneuver through the content via the trial life cycle, directly search using the search box in the site's upper right corner, or use the “View all tools and resources” page to sort by categories (eg, demographics, resource type).
Community of Practice
To foster a community of practice, AccrualNet is designed to encourage activity among stakeholders. First, users can access accrual resources within AccrualNet that are ready for deployment (eg, templates). Second, the AccrualNet architecture easily highlights knowledge gaps and prime targets for new research projects. For example, the dearth of information in the poststudy stage, with only 17 resources currently available, makes it a promising choice for future research. Finally, this platform is structured for reinvention, encouraging users to recommend new content, comment on existing content, join or start a conversation with other stakeholders, and pose questions to an expert. Community input also provides fresh content to help bring users back regularly.
Metrics
As of April 30, 2011, AccrualNet had received 54,957 unique page views and 10,208 site visits since its April 2010 launch. The top five non–home page areas of interest have been Tools and Resources (45.4% views), Recruiting/Enrolling (31.1%), Conversations (23.7%), and Training (15.1%). Since its release, the original set of 294 unique tools and resources has expanded 69% (n = 496) with an average of 10 to 15 added monthly. Since the NCI began to actively promote the conversation function, there have been 37 discussion topics and 57 approved comments. Examples of conversations include topics related to insurance denial, ethics of clinical trial participation, and a subforum to address CCOP-specific issues.
Analysis of Content
Among the seven substages depicted on the life cycle wheel (ie, seven circles), each stage has several strategies listed (N = 27; mean = 3.9, range, 2 to 5), and each strategy has approximately four activities (N = 102; mean = 3.8, range, 2 to 7). Each activity has one or more resources (mean 16.1, range, 1 to 76), gleaned from journals and nonjournal sources alike, that are available to assist in performing the activity. The percentages of resources, sorted by volume, are preparing to open the trial, 35%; recruiting and enrolling participants, 24%; developing a protocol, 14%; selecting a trial, managing the trial, and retaining participants, 8% each; and evaluating accrual and lessons learned, 3%.
Inspecting the distribution of the content shows a pronounced bias toward both journal and nonjournal content relevant to the prestudy stages (53%) and the study-activation stages (44%), and a dearth of content relevant to the poststudy stage (3%). In addition, there is a pronounced long tail (subexponential) effect for sources of published literature. Specifically, 21.6% of all published literature came from two journals (Contemporary Clinical Trials, 13.4%; Journal of Clinical Oncology, 8.2%). However, nearly half of the journals (46.3%) supplied only one published article to AccrualNet (Table 1).
Table 1.
Journals Sources for Articles in AccrualNet (as of 1/31/11)
| Journal | No. | % | Cumulative Total (%) |
|---|---|---|---|
| Contemporary Clinical Trials | 44 | 13.4 | 13.4 |
| Journal of Clinical Oncology | 27 | 8.2 | 21.6 |
| Controlled Clinical Trials | 19 | 5.8 | 27.4 |
| Clinical Trials | 14 | 4.3 | 31.7 |
| BMC Medical Research Methodology | 9 | 2.7 | 34.5 |
| Cancer | 8 | 2.4 | 36.9 |
| Trials | 7 | 2.1 | 39.0 |
| Annals of Epidemiology | 5 | 1.5 | 40.5 |
| Cancer Nursing | 5 | 1.5 | 42.1 |
| Journals with four articles | 11 | 3.4 | 45.4 |
| Journals with three articles | 4 | 1.2 | 46.6 |
| Journals with two articles | 23 | 7.0 | 53.7 |
| Journals with one article | 152 | 46.3 | 100.0 |
Use Cases
Four AccrualNet use cases emerged from the research. They are:
1. Decrease training time concerning issues of accrual to clinical trials.
Scenario.
Every year, new fellows arrive at the hem/onc division, with little clinical trial experience or familiarity with the research language. Educating them quickly and effectively is Sheila's job. She sees AccrualNet as an easy way to teach staff the general framework of clinical research, and appreciates all training materials gathered in one place.
In this scenario, AccrualNet is used to help new clinical researchers understand the connections, relationships, and common context for oncology clinical trials. Using a single source, it will be simpler for novice researchers to understand the flow of research from inception to closure and the major issues connected with each stage.
2. Reduce rework and limit reinvention of known solutions to accrual issues.
Scenario.
As a CCOP coordinator, Fred's challenge is figuring out how to work more effectively with the surgeons for their clinical trials. Unaware of any resources for this problem, he typed “surgeons” into the AccrualNet search engine and found two articles. The resource page in AccrualNet highlighted the articles' key information and provided the hyperlink to PubMed.
This use case demonstrates the platform's potential as a knowledge repository for information about various elements of accrual. Its value is as a central source, cross-indexed and available via a search engine. In addition, the resource abstracts makes it easier to understand the resource's highlights.
3. Increase responsiveness to accrual issues that arise during a clinical trial.
Scenario.
This year, Jean wants to try social networks to talk directly to patients. She believes she can reach out to the AccrualNet community and share and learn from others' experiences.
This use case shows the potential of AccrualNet as a primary source for community engagement. Rather than post a request on multiple discussion boards or listservs, it engages the community of practice in tailored accrual assistance and facilitates connections among individuals in similar situations with common problems.
4. Improve the degree of innovation in developing strategies, activities, and resources to solve accrual issues.
Scenario.
Dr Smith identified an accrual area where almost no research exists. She wants to use AccrualNet to identify a group of like-minded oncologists interested in the problem, as she is otherwise unaware of others in the oncology community working on this issue.
In this scenario, AccrualNet has the potential to identify gaps in the knowledge base, and create connections among teams to research the problem, perhaps allowing for more rapid development of solutions.
Discussion
The recently released Institute of Medicine report on reinvigorating the oncology clinical trials system establishes as a major pillar the fostering of expanded participation by both patients and physicians.20 AccrualNet supports this effort by assembling, organizing, and disseminating nationally an otherwise scattered and disparate body of accrual resources and tools. AccrualNet also facilitates dialog and collaboration among the community.
AccrualNet was created with an explicit focus on the life cycle of a clinical trial, providing resources from study inception to trial closure. The graphic design, the use of a common language to describe clinical research, and the central repository of accrual resources should facilitate understanding among novices of the relationships among the trial stages and the identification of accrual solutions.
AccrualNet also offers a just-in-time learning platform, where desired information can be discovered swiftly and at the point most relevant to the end user. Initial market analysis showed that finding specific accrual solutions is difficult, time consuming, and nearly impossible unless one has access to a wide breadth of resources. Among the journals with articles appearing in AccrualNet's resource collection, most supplied only one article, which demonstrates how widely scattered (and difficult to obtain) accrual information can be without a centralized platform.
The information architecture of AccrualNet highlights the multifaceted nature of the oncology clinical research system. Within a trial life cycle, AccrualNet identifies 102 individual activities required to complete 27 specific strategies to promote accrual. Although not all activities or strategies are equally important in their impact on trial accrual, they are all considerations that should be addressed. Similarly, there was a high variance in the number of resources found for AccrualNet across a trial life cycle. For example, the most researched activity is “Understanding participants' perceptions of being part of a clinical trial,” with 76 resources. In contrast, “Ensure buy-in from experts/specialists who are needed to implement the trial” has just two resources. The field of oncology will need to better understand the gaps in knowledge highlighted in AccrualNet to more comprehensively address the problem of low clinical trial accrual.
Beyond presenting useful content in an accessible manner, AccrualNet hopes to benefit clinical trial professionals by serving as a “virtual community of practice”—a place to ask questions, share observations, and promote best practices. The site design promotes engagement by enabling readers to comment on existing resources, submit new ones, and start or join conversations with peers. Recognizing that accrual successes happen frequently but may be published rarely, the site hopes to encourage readers to blog about their experiences and to connect with colleagues on AccrualNet.
The inclusion of features to build a community of practice enables the system to add to the knowledge repository over time, in the form of both published research and lessons learned from clinical trial professionals in the field. AccrualNet allows oncology researchers to identify, address, and share solutions to accrual challenges. It permits such evidence-based knowledge to be easily disseminated and acted on via both a formal and an interpersonal network of a community of practice. As such, this platform is structured for reinvention, by users themselves, as new knowledge becomes available. It encourages users to recommend new content, comment on existing content, join/start a conversation, and pose questions to an expert, thus allowing for the reinvention required for successful technology diffusion10 and building of an active community of practice. It is important that the site be continuously monitored to ensure that the appropriate direct and surrogate metrics of usefulness are maintained.
In considering Rogers' characteristics of successful diffusion,10 AccrualNet offers the oncology field a far superior method to access accrual solutions (relative advantage), by offering a Web platform to obtain hundreds of accrual resources quickly and easily (compatible with clinicians' workflow, simple). Through the ability to comment on existing materials and add new content, AccrualNet also increases the ability of clinicians to learn vicariously through others' experiences and feedback. AccrualNet should continue to be evaluated to assess its impact on the oncology research field and how to maximize the value it adds to efforts to improve trial accrual. Although the information architecture has been subjected to expert review, for example, it has not been validated in the field. With continued attention to diffusion characteristics and building a community of practice, AccrualNet should improve the likelihood that the accrual resources will diffuse and be used by the clinical trial research community.
Acknowledgment
AccrualNet was developed by the National Cancer Institute. Portions of this study were presented at the NCI-ASCO Cancer Clinical Trial Accrual Symposium, Rockville, MD, April 30, 2010; the Community Clinical Oncology Program (CCOP), Minority-Based CCOP (MBCCOP), and Research Base Principal Investigator (PI) and Administrator Meeting, Rockville, MD, September 16, 2010; as a poster session at the Society of Clinical Research Associates, Dallas, TX, September 24-26, 2010; and as a poster session at the American Association for Cancer Education/Cancer Patient Education Network, San Diego, CA, October 25-27, 2010.
Appendix
Figure A1.
AccrualNet organizational model.
Figure A2.
AccrualNet screen shot, stage level.
Authors' Disclosures of Potential Conflicts of Interest
The authors indicated no potential conflicts of interest.
Author Contributions
Conception and design: Holly A. Massett, Linda Parreco, Rose Mary Padberg, Ellen Richmond, Colleen Ryan Leonard, Whitney Quesenbery, H. William Killam, Lenora E. Johnson, David M. Dilts
Financial support: Holly A. Massett, Lenora E. Johnson
Administrative support: Holly A. Massett, Rose Mary Padberg, Ellen Richmond, Marie Rienzo, Colleen Ryan Leonard, Lenora E. Johnson
Provision of study materials or patients: Holly A. Massett, Rose Mary Padberg, Ellen Richmond, Marie Rienzo, David M. Dilts
Collection and assembly of data: Holly A. Massett, Linda Parreco, Rose Mary Padberg, Ellen Richmond, Marie Rienzo, Colleen Ryan Leonard, Whitney Quesenbery, H. William Killam, Lenora E. Johnson, David M. Dilts
Data analysis and interpretation: Holly A. Massett, Linda Parreco, Colleen Ryan Leonard, Whitney Quesenbery, H. William Killam, Lenora E. Johnson, David M. Dilts
Manuscript writing: Holly A. Massett, Linda Parreco, Rose Mary Padberg, Ellen Richmond, Colleen Ryan Leonard, Whitney Quesenbery, David M. Dilts
Final approval of manuscript: All authors
References
- 1.Friedman MA, Cain DF. National Cancer Institute sponsored cooperative clinical trials. Cancer. 1990;65:2376–2382. doi: 10.1002/1097-0142(19900515)65:10+<2376::aid-cncr2820651504>3.0.co;2-a. [DOI] [PubMed] [Google Scholar]
- 2.Sateren W, Trimble E, Abrams J, et al. How sociodemographics, presence of oncology specialists, and hospital cancer programs affect accrual to cancer treatment trials. J Clin Oncol. 2002;20:2109. doi: 10.1200/JCO.2002.08.056. [DOI] [PubMed] [Google Scholar]
- 3.Curran W, Jr, Schiller J, Wolkin A, et al. Addressing the current challenges of non–small-cell lung cancer clinical trial accrual. Clin Lung Cancer. 2008;9:222–226. doi: 10.3816/CLC.2008.n.033. [DOI] [PubMed] [Google Scholar]
- 4.Vickers AJ. Do we want more cancer patients on clinical trials? If so, what are the barriers to greater accrual? Trials. 2008;9:31. doi: 10.1186/1745-6215-9-31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Klabunde CN, Keating NL, Potosky AL, et al. A population-based assessment of specialty physician involvement in cancer clinical trials. J Natl Cancer Inst. 2011;103:384–397. doi: 10.1093/jnci/djq549. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Cheng SK, Dietrich MS, Dilts DM. A sense of urgency: Evaluating the link between clinical trial development time and the accrual performance of CTEP-sponsored studies. Clin Cancer Res. 2010;16:5557–563. doi: 10.1158/1078-0432.CCR-10-0133. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Schroen AT, Petroni GR, Wang H, et al. Challenges to accrual predictions to phase III cancer clinical trials: A survey of study chairs and lead statisticians of 248 NCI-sponsored trials. J Clin Onc. 2009;27:338s. doi: 10.1177/1740774511419683. abstr 6562. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Go RS, Meyer M, Mathiason MA, et al. Nature and outcome of clinical trials conducted by the Eastern Cooperative Group (ECOG) from 1977 to 2006. J Clin Onc. 2010;28:464s. abstr 6069. [Google Scholar]
- 9.Korn EL, Freidlin B, Mooney M, et al. Accrual experience of National Cancer Institute cooperative group phase III trials activated from 2000 to 2007. J Clin Oncol. 2010;28:5197–5201. doi: 10.1200/JCO.2010.31.5382. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Rogers EM. Diffusion of Innovations. ed 5. New York, NY: The Free Press; 2003. [Google Scholar]
- 11.Lave J, Wenger E. Cambridge, England: Cambridge University Press; 1991. Situated Learning: Legitimate Peripheral Participation. [Google Scholar]
- 12.McDermott R, Archibald D. Harnessing your staff's informal networks. Harvard Bus Rev. 2010;88:82–89. [PubMed] [Google Scholar]
- 13.AccrualNet. 2010. https://accrualnet.cancer.gov.
- 14.National Cancer Institute. Making Health Communication Programs Work. 2002. http://www.cancer.gov/cancertopics/cancerlibrary/pinkbook/page1.
- 15.Research-Based Web Design and Usability Guidelines. 2004. http://www.usability.gov/guidelines/
- 16.Larman C. Agile & Iterative Development. Reading, MA: Addison-Wesley Professional; 2003. [Google Scholar]
- 17.Morville P, Rosenfeld L. Sebastopol, CA: O'Reilly; 2006. Information Architecture for the World Wide Web. [Google Scholar]
- 18.Newman MW, Landay JA. Sitemaps, storyboards, and specifications: A sketch of Web site design practice. In: Boyarski D, Kellogg WA, editors. Proceedings of the 3rd Conference on Designing Interactive Systems; New York, NY: ACM; 2000. pp. 263–274. [Google Scholar]
- 19.Ambler SW. Cambridge, England: Cambridge University Press; 2004. The Object Primer: Agile Modeling-Driven Development With UML 2. [Google Scholar]
- 20.Institute of Medicine: A National Cancer Clinical Trials System for the 21st Century. Washington, DC: National Academies Press; 2010. Reinvigorating the NCI Cooperative Group Program. [PubMed] [Google Scholar]