Lessons Learned Designing and Using an Online Discussion Forum for Care Coordinators in Primary Care

Abstract

While an increasing number of researchers are using online discussion forums for qualitative research, few authors have documented their experiences and lessons learned to demonstrate this method’s viability and validity in health services research. We comprehensively describe our experiences, from start to finish, of designing and using an asynchronous online discussion forum for collecting and analyzing information elicited from care coordinators in Patient-Centered Medical Homes across the United States. Our lessons learned from each phase, including planning, designing, implementing, using, and ending this private online discussion forum, provide some recommendations for other health services researchers considering this method. An asynchronous online discussion forum is a feasible, efficient, and effective method to conduct a qualitative study, particularly when subjects are health professionals.

With the proliferation of health-related online discussion forums for information sharing and socialization (Lederman, Fan, Smith, & Chang, 2014), an increasing number of social and health science researchers are recognizing the internet as a rich source of information. Many healthcare studies using online forums conduct retrospective or secondary analyses of archived messages to capture patient experiences, which preclude researchers from asking follow-up questions (Brown & Altice, 2014; Butt, Cox, Oyebode, & Ferner, 2012; Gill & Whisnant, 2012; Hua, Alfi, & Talbot, 2013; Mao et al., 2013; Sarrazin, Cram, Mazur, Ward, & Reisinger, 2014). While an increasing number of health researchers are designing online discussion forums to qualitatively collect data from patients (Horgan, McCarthy, & Sweeney, 2013; Im et al., 2010; Im, Lee, & Chee, 2011; Tour et al., 2014), few have studied how useful online forums are for knowledge sharing and social networking among health professionals (Curran & Abidi, 2007; Dieleman & Duncan, 2013; Kenny, 2005; Stewart & Abidi, 2012). Additionally, there is limited literature documenting experiences and lessons learned with using online forums to inform other researchers interested in using this method (Im & Chee, 2006; Im & Chee, 2012; Wilkerson, Iantaffi, Grey, Bockting, & Rosser, 2014; Williams, Clausen, Robertson, S., & McPherson, 2012).

We comprehensively describe our experiences and lessons learned from designing and using an asynchronous on-line discussion forum for collecting and analyzing information elicited from care coordinators in Patient-Centered Medical Homes (PCMH). The PCMH is widely endorsed as a model of advanced primary care that holds promise in transforming the United States healthcare system to one that is more patient-centered, coordinated, accessible, effective, safer, and efficient (Jackson et al., 2013; Rosenthal, 2008). In the following sections, we detail all our experiences during the three phases of our research study: (a) Planning and development, (b) Implementation and active usage, and (c) Ending the forum. We then discuss participants’ evaluation of the forum and our lessons learned. Finally, we offer recommendations for other health services researchers interested in using asynchronous online discussion forums for qualitative research. Our research team included a family physician, three sociologists, a medical informatics expert and a communications expert. The Institutional Review Board at Rutgers-Robert Wood Johnson Medical School approved this study.

Planning and Development

Choosing an Asynchronous Online Forum to Study Care Coordinators in Primary Care

Many PCMH demonstration projects are instituting care coordinators within primary care practices to help eliminate gaps in shared information and communication among providers and between patients and providers (Patient-Centered Primary Care Collaborative, 2009). However, there is limited research documenting care coordination processes and coordinators’ experiences from their own perspectives (Fagnan et al., 2011; Ferrante, Cohen, & Crosson, 2010; Henderson, Princell, & Martin, 2012). Furthermore, the role of care coordinator is relatively new in primary care settings, and how this role manifests and is integrated into the practice is unknown. An online discussion forum could be used to acquire a breadth of perspectives on these research questions. It could also act as a method for social interaction and collaborative learning in that care coordinators can answer questions, ask questions, and view and react to other participants’ answers (Seale, Charteris-Black, MacFarlane, & McPherson, 2010).

We chose an online asynchronous discussion forum versus a face-to-face or synchronous online focus group for the following reasons. The online asynchronous format allowed busy health care workers from around the nation to participate in their own time and space (Sintjago & Link, 2012). It allowed us to conduct real-time qualitative analyses as part of an iterative process in which the care coordinators were also involved in determining the meaning and significance of findings. We monitored the online dialogue in real time and shared preliminary findings with coordinators, which allowed them to help build on or clarify emerging concepts. In this way, coordinators were not just research subjects, but also participants engaged in shaping the research product. Preliminary findings served as a narrative of the research project, so that concepts under development and changes in process or focus as the project evolved were transparent (Wakeford & Cohen, 2008).

Design of Site and Functionality

Our university’s Office of Information Technology (OIT) guided us through a six-sigma process (Harry, Prem, De Hodgins, Hulbert, & Lacke, 2010) to plan and design the web site. Through this process, we determined the structures and functions of the web application that would be needed for both the collaborative learning and research components of the study. OIT created a custom private web application on a secure server. The website had an Overview page that included several tabs: Discussion Forum, Resources, Instructions, FAQ, and Reports (Table 1 and Supplemental Figure 1).

Table 1.

Structure and Function of Website

Main Tabs	Features
Overview	Description of different sections of website
Discussion Forum	Posting of Topics for Discussion and Researcher’s Analysis Blog
Resources	Allowed participants to upload documents and resources for sharing
Instructions	General instructions about using and posting on the website, expectations, ground rules, confidentiality and etiquette
FAQ	Information on technical and research aspects and role of moderators
Reports	Allowed researchers to export discussion threads and usage tracking reports

The Discussion Forum tab was the main interactive window that contained topics for discussion and the researcher’s analysis blog, where preliminary findings of content were shared (Supplemental Figure 2). The reports tab, accessible only by the researchers, gave us the ability to export discussion threads directly into Word documents as well as usage tracking reports directly into an Excel spreadsheet (Supplemental Figure 3). Additionally, an account management feature allowed participants to add information and edit their user profiles. Automatic standardized email notifications were sent to participants when there was a new question posted, when participants did not post after one week, and when there was a comment made to one of their posts. Additionally, there was an automatic assignment of incentive points for participation (discussed below) as well as a search function.

Defining Roles and Privileges

As part of the design process, we defined the roles and privileges for participants and researchers/moderators (e.g., care coordinator participants had the ability to post, respond, and upload documents to the website, but only researchers could edit, hide, delete postings, or add/delete users). Other logistics such as the process for creating accounts, monitoring and moderating the forum, formatting and posting questions, and analyzing and posting comments on the researcher’s blog were also determined. These guidelines were written and posted on the website in the instructions and FAQ documents.

Recruitment of participants

We initially sent emails with a flyer about the study to medical directors in practices that were participating in PCMH demonstration projects and utilizing care coordinators as listed on the Patient-Centered Primary Care Collaborative website (www.pcpcc.org) in 2010. We asked the medical directors to forward the information to care coordinators, and snowball sampling was also used to identify and contact other programs using care coordinators (Wilkerson et al., 2014).

The care coordinators who responded to our invitation were enthusiastic about learning and sharing with others. We enrolled 25 care coordinators out of 33 who expressed interest, anticipating a 50% dropout rate so that we would end up with at least 12–14 participants, which is the mean sample size of other online focus groups (Williams et al., 2012). We screened potential participants through e-mail and purposively chose care coordinators from primary care practices of varying organization types (six private practice; seven academic; six hospital or integrated health system- affiliated; six community health centers), locations (eight urban, eight suburban, nine rural), and size (1–40 clinicians) to maximize the richness and variability of experiences.

Implementation and Active Usage

Consent and Sign-in Process

We emailed a consent form to participants for them to sign and fax back, before a link to the website was given with instructions to create their account. There was an initial sign-in process, which required participants at first log in to read the user agreement and answer a grand tour question before they were allowed to enter the website (“What is your job title and description? Please describe a typical workday for you.”) Similar to the process of conducting a face-to-face interview or focus group (Dicicco-Bloom & Crabtree, 2006), we used a preliminary interview guide, with questions based on the Organizational Design Framework (McDonald et al., 2007). In this model, the care coordinator is the “good fit” that matches the demands of the setting and patients with the coordinating mechanisms that facilitate information flow and information processing for coordinating activities (Table 2). These questions were modified and additional questions were added based on the content of discussions. Each question was posted serially approximately once per week, to the Topics for Discussion page of the forum, and topics remained for the entire study period.

Table 2.

Sample Questions based on Organizational Design Framework

Framework Concepts	Sample Questions
Demands of Setting and Patients
Identification and assessment of patients in need of coordination	How do you decide which patient population to focus on (e.g., is it based on demographics, complexity of diseases, uncertainty of patient compliance, frequent use of medical care in multiple settings)?
Role identification	We are surprised at how many different titles you have. From your own personal experience, do you see some important differences between the different titles and roles or is this a similar role that different organizations are naming differently? What relationships with other people need to be developed for your job to be successful?
Coordinating Mechanisms
Operational processes	How often are you included in patient visits with their PCP’s? What/who determines when you’re included? What strategies have you used to engage patients?
Structural linking	What strategies or tools are you and your organizations using to reduce hospital and emergency department readmissions? Please describe the flow of communication that is involved. Some of you mentioned being involved in team meetings or huddles. Can you tell us more about these meetings, such as how often they are held, who are involved, your role in the meetings, what is discussed?
Grouping	How are you integrated with other practice members and with practice-wide care processes? For example, do you attend staff meetings, feel like you are part of the practice, and communicate daily with other practice members?

Encouraging Participation

To encourage participation, we used automatic email reminders, a point system, and financial rewards, which have been commonly used in other discussion forums. (Birnholtz, Horn, Finholt, & Bae, 2004; Farzan et al., 2008; Garnefeld, Iseke, & Krebs, 2012) We held monthly drawings for a $100 retail gift card for all who contributed to the forum at least once weekly. Reminder emails about these monthly drawings were sent to all participants at the end of each week. We also used a point system in which participants received four points for posting a question or resource, and three points for answering a question or commenting to someone else’s post. At the end of each month, the three participants with the highest number of points for the month were entered twice into the gift card drawing. At the end of the forum, the three participants with the highest number of overall points also received $100, $75, and $50 gift cards. The incentive system and email notifications appeared to foster participation on the forum. We found these weekly email reminders about the monthly gift card drawings generated a wave of postings at the end of each week.

Technical Challenges

Several technological issues arose in the beginning of implementation, such as users having difficulty in creating their own user name and passwords, getting error messages when logging in or attempting to upload a profile picture, or seeing blank pages because of web browser compatibility problems. To address these issues, we assigned user names and temporary passwords for all participants and worked closely with OIT to resolve application errors. Ongoing OIT support was needed throughout the study period. For example, participants were being timed out after 15 minutes without warning. If someone took a long time to write a detailed post, that person would unexpectedly be logged out and the post would be lost. We found a workaround (using the back button on the web browser and then copying and pasting the text) until OIT added in a five-minute warning, with the option to extend the time for 20 minutes. Another initial problem was a limit to the length of responses, so if someone wrote a long comment, not all of it was visible. OIT changed the setting so that there was no limit to the number of characters per response. An additional issue necessitating OIT support was the reordering of the list of topics so that the most recent question or topic of discussion appeared at the top of the page rather than at the bottom. Finally, there were several technical glitches with the automatic email reminders. For example, they were initially sent from an unfamiliar email address with an unfamiliar subject heading, they were ending up in the spam folder, the forum link embedded in the email was incorrect, and they stopped being automatically sent. Having a test account for the moderators alerted us to these issues.

Moderating the Forum

During the planning phase, we defined the moderator’s role as someone who would clarify postings, elicit elaboration or explanation, foster interaction, encourage reflection, keep participants on track, offer support, and enforce the rules of engagement. We provided a guide for each moderator with sample wording for responses (Table 3). One person was assigned to moderating the site each day. Our research team met weekly to discuss and reflect on postings and to decide what to post on the researchers’ blog. We provided comments to the group, added a new topic to the discussion forum, or placed our reactions and follow-up questions on the researcher’s blog about once a week during the active period. This seemed sufficiently frequent enough to avoid ending a conversation prematurely while adequately prompting the group to maintain engagement. Participants were also encouraged to add topics they wanted to discuss with the group and to address each other as well as to us.

Table 3.

Sample wording for moderator responses

Moderator Functions	Sample Responses
Clarify postings and elicit elaboration or explanation	That was really interesting; we’d like to hear more about this. Can you give us an example/a story of what you mean? Please describe what you mean. Could you be more specific about xxx? What experiences have you had to make you feel that way?
Foster interaction	Have other people experienced this? What about the rest of you? We want to hear all the different points of view. Does anyone have a different perspective/experience? Do others have similar/other stories to share? Any reactions to what (person) just shared? How do others feel about that? How have the rest of you dealt with this?
Encourage reflection	What did you learn/gained/take away from what you just shared? How did you feel when this happened? Any reactions to what (name) just shared? How would you respond differently now?
Keep participants on track	It seems like a lot of you are interested in this- let’s start a new thread on this topic. We can start a new topic on this, but let’s finish this first. Remember before we started talking about this, we were talking about xx. Earlier, (name) mentioned x. What do you think about that?
Offer support	That must have been very difficult/distressing/embarrassing, etc…Any reactions to what (person) just shared? What can we learn/take away from this? Any suggestions from the rest of the group? Any ideas on how to handle this the next time it comes up?
Enforce rules of engagement	Just a reminder, please spell out acronyms. Please don’t use all capital letters/avoid text speak.

Participation in Forum

The forum was open for responses from January to May 2012. Out of the 25 care coordinators who signed consent forms, created accounts, and answered the first grand tour question, three formally dropped out (two due to personal reasons and one left the position), 17 continued to log in through the fifth month, and 13 participants posted messages into the fifth month. We monitored participation via the usage tracking reports, downloaded directly from the website, which described when each participant logged in, the number of times they logged in, the date of their last post, the number of posts they made, and their total number of incentive points earned. The number of posts per participant ranged from 1–54 (mean of 19) and number of log-ins ranged from 5–84 (mean, 36). There was no limit to the number or length of responses to each topic. Number of messages per topic ranged from 7 to 31, and number of words in each message ranged from 5–884 (mean, 129). The shorter responses tended to be statements of agreement with someone’s post, while longer responses were usually stories and personal reflections about patient cases. Participants had the option to post messages anonymously, but no one did so during the forum.

The majority of the responses, particularly the early ones, focused on knowledge sharing and answers to our questions. The forum became more active as time went on. During Months two and three we saw more cross-talk among participants. These consisted of participants agreeing with or elaborating on someone else’s post, providing praise and affirmation about someone’s success, answering someone’s question, or expressing envy of others’ resources. Participants began to post questions of their own, seeking advice, for example, regarding training and certification, selection of patients for services, and strategies in managing population health and noncompliant patients. They also shared resources, which included: sample job descriptions; treatment plans; transition of care logs; PCMH implementation guides; resources for continuing education, skill development and self-care; and links to resources for patients.

There was also evidence of networking and provision of emotional support, as many participants seemed to recognize and value the community building aspects of the forum. The following examples highlight these aspects:

For me there is more work that needs to be done than I have time to do. I would have time to better manage our patient population if we were not so tightly staffed…. We have no access to extra support.… So, some days I end up helping answer phones, draw/process lab specimens, process referrals, even room patients. Having to do those things prevents some phone calls that I would be making to our complex patients.… I get frustrated when I can’t dedicate more time to do the job I was hired for.

EC-thank you for sharing such personal things. It is inspiring to hear how you approach each day - we often forget to take care of ourselves, body and soul, and we forget how lucky we are. Regardless of one’s faith, personal reflection can make you a better practitioner, spouse, mother, daughter etc.

One of the most beneficial aspects of this experience for me has been the ability to hear other stories and perspectives, the good and the more challenging. It has helped me to know that many of my daily struggles are experienced by several of us and that I’m not “alone.” That sense of community has been invaluable! There are numerous benefits to participating, but some days, I just need to know that I can log on and read what other folks are up to, knowing that it helps me to feel connected (even when I’m so exhausted after long day and I don’t respond to discussion forum at that time, it’s nice to get on and just read away!) This has had such a fabulous positive benefit!! THANKS, everyone!

Emerging Roles of Participants

As found in other Internet communities, several roles emerged based on participants’ log-in and post frequencies (van Mierlo, 2014). A few participants were clear leaders who logged on many times and commented frequently, encouraging others and uploading resources to share with the group. These Superusers logged in over 60–84 times and commented over 50 times. Others were active Contributors, logging in 20–40 times and posting almost every time they logged in. There were also several Lurkers who logged in 50–60 times but posted about 10 times. Three participants never fully engaged, logging in 5–7 times and posting only once.

Real Time Analysis

While at least one moderator monitored the discussions daily, our research team met weekly to review and reflect on the messages, decide on additional questions to post, and generate preliminary analyses for the researchers’ blog. This blog was used to summarize discussion threads and to ask participants for feedback on our emerging findings. After each thread of conversation had ended, we imported the data into ATLAS.ti (Atlas.ti Scientific Software Development GmbH, Berlin, Germany), a qualitative data analysis and research software, for coding and more in-depth content analysis.

Ending the Forum

We spent much time planning the process for winding down the forum. We did not want to shut it down abruptly, nor was it feasible to keep it open past the facilitated study period. We posed this forum continuation question to the participants, and several wanted another medium to continue the collaboration and community. One of the more active members volunteered to begin a public professional group on LinkedIn (Care Coordination in Primary Care). While 20 members joined the LinkedIn group, to date there have been only two discussions started. This may be due to the lack of an active moderator to seed conversations and/or the presence of other similar LinkedIn or social media groups.

Participants’ Evaluation of Forum

At the end of the study, we sent a web-based survey, external to the discussion forum, evaluating the online forum to all 25 participants who enrolled; we received 17 responses. Overall, most participants agreed or strongly agreed that they gained a sense of community and support and were inspired with new ideas. However, many responded that they did not have enough time to log in and post, keep up with others’ posts, and respond before a new topic was posted (Table 4).

Table 4.

Participants’ Evaluation of Forum (N=17)

Statements	Strongly Disagree N (%)	Disagree N (%)	Neutral N (%)	Agree N (%)	Strongly Agree N (%)	Rating Average (scale 1–5)
Participating in this forum has inspired me with new ideas.	0 (0)	0 (0)	5 (29.5)	10 (58.8)	2 (11.8)	3.82
I found the resources that participants posted helpful.	0 (0)	0 (0)	5 (29.4)	10 (58.8)	2 (11.8)	3.82
The gift card drawings motivated me to post on the forum.	0 (0)	5 (29.4)	4 (23.5)	6 (35.3)	2 (11.8)	3.29
It was difficult for me to find the time to log in and post on the forum.	9 (9)	2 (11.8)	1 (5.9)	7 (41.2)	7 (41.2)	4.12
I experienced a sense of support and community from participating in this forum.	0 (0)	0 (0)	3 (17.6)	11 (64.7)	3 (17.6)	4.00
What I learned by participating in this online community helped me in my day-to-day work.	0 (0)	0 (0)	8 (47.1)	7 (41.2)	2 (11.8)	3.65
The technological problems on the website kept me from participating on the forum.	5 (29.4)	6 (35.3)	4 (23.5)	2 (11.8)	0 (0)	2.18
I did not have enough time to read and respond between new discussion topics.	0 (0)	3 (17.6)	4 (23.5)	5 (29.4)	5 (29.4)	3.71
I appreciated participating in the researcher’s analysis section.	0 (0)	0 (0)	6 (35.3)	11 (64.7)	0 (0)	3.65
I could not keep up with reading all participants’ postings.	0 (0)	3 (17.6)	3 (17.6)	8 (47.1)	3 (17.6)	3.65

Lessons Learned and Recommendations

We provide recommendations for other health services researchers considering using an asynchronous online discussion forum for collecting qualitative data.

1. Allow sufficient time upfront to design the site: The planning, development of the application, and review and revision of mock-ups lasted about 5 months before we were able to “go live.” It was crucial to work with the web designer to determine the structure and functionality of the site, define the roles and privileges of users, and develop instructions and rules of engagement.

2. Allow time to pilot-test the website and have a test account: Because the planning and development phase took longer than we expected and we had a one-year grant to complete this project, we weren’t able to pilot test the site before going live as planned. Piloting the website could have prevented several technological issues that arose during implementation. Monitoring of a test account alerted us to some of these issues.

3. Do not recruit participants too early: As it was fairly easy to recruit participants, we had to email each of them several times during the planning months to ensure continued interest during the delay in “going live.” We may have lost some potential participants during this delay.

4. Ensure there is ongoing technological support available: We found we needed ongoing technological support throughout the study period, for errors such as those related to creating accounts, editing profiles, using different web browsers, being timed out, and sequencing of discussion threads. Although 11 of the 17 survey respondents felt the technological problems did not keep them from participating in the forum, we do not know whether the technological problems affected the non-respondents or those participants who never became engaged.

5. Use incentives and email reminders to encourage participation: While survey results found the impact of gift cards on participation to be mixed, participants mentioned that automated email reminders were helpful, and we observed a flurry of postings in response to weekly email reminders about the gift card drawings.

6. Moderate discussions to guide and foster participation: The lack of participation in the subsequent LinkedIn group suggests that this is necessary. We found that one person was sufficient to moderate the discussion at one time. Most of the moderating that was needed pertained to clarifying postings, eliciting elaborations, fostering interactions, and encouraging reflection. There was not much need to keep participants on track or enforce rules of engagement, probably because we provided clear instructions and expectations up front. It was also not necessary for moderators to offer support, as the participants took that on themselves.

7. Probe for deeper reflection by asking follow-up questions to the entire group: We initially probed brief one-line responses to each individual, but found that some would be ignored or missed as discussions continued on. Posting follow-up questions to the entire group, either by a new discussion thread or on the researcher’s analysis blog, generated more detailed responses and cross-talk among participants.

8. Allow adequate time between new topics for discussion: We posted new topics for discussion about once a week because there seemed to be a lull in the conversation after one week. However, 10 out of 17 survey respondents felt this was not enough time to read and respond. Moderators need to be prepared to be patient and wait several days for responses. How often moderators should prompt the group to maintain engagement should be balanced with giving enough time to avoid ending a conversation prematurely.

9. Plan ahead and advise participants what will happen when the study is over: Several participants were interested in continuing the online community after the end of the study period. Be clear that once the study period is over, participants will no longer have access to the site, and it will be necessary for them to continue the group elsewhere.

10. Choose a moderator to maintain a subsequent continuation group: While starting another group on a public social media site is fairly easy and would avoid the need for technological support, a moderator is needed to foster continued participation. Perhaps having the group select a moderator or encouraging one or more Superusers to serve as moderators would have increased the vitality and success of the subsequent group.

Discussion

The use of an asynchronous online discussion forum is particularly useful as a research method for data collection from hard to reach populations (Tates et al., 2009), such as care coordinators in primary care, who may frequently be the sole person in that role within an office. While health professionals in general are often too busy in their day-to-day work to participate in research studies (Herber, Schnepp, & Rieger, 2009; Sahin, Yaffe, Sussman, & McCusker, 2014), these care coordinators, who were relatively new and often isolated, were enthusiastic to participate and network professionally with others in a similar role. We found that taking the time upfront and working with an interdisciplinary team to plan and think through each phase of the study, including moderating and ending the forum, were crucial to success.

There are many advantages to using an asynchronous online discussion forum. Compared with a face-to-face focus group, it allows for easier recruitment and knowledge sharing and networking among participants that are geographically dispersed. It also leads to faster and more cost efficient data collection, by easing the process of coordinating busy schedules and eliminating the need for researchers to travel, rent space, provide refreshments, moderate, and take notes. Additionally, the data is already transcribed, with less chance of error, and is immediately ready to analyze. An asynchronous web forum may also allow for more participation and interaction than a face-to-face or synchronous online focus group. With fewer time pressures, participants can carefully construct their responses, so data can be equally rich or potentially more detailed, elaborate, and reflective (Fox, Morris, & Rumsey, 2007; Hewson & Laurent, 2008; Wilkerson et al., 2014). For example, several participants’ posts, in particular, the lengthier ones over 500 words, contained detailed stories, description of patient cases, and respondent’s thoughtful reflections. The relative anonymity of an online discussion also helps encourage more honest self-disclosure and participation from people who are shy or reluctant to speak up when in front of a group (Williams et al., 2012). Comparing the quality of data using this method to other types of data collection methods, for instance, face-to-face or synchronous online focus groups, is a step for future research.

Another key benefit of this research format was our ability to analyze the content of discussion threads as it was being collected, and to move iteratively between data collection and data analysis. This allowed us more time to reflect on previous responses before asking new questions and to share emerging findings with participants, thereby allowing for member checking on preliminary analyses. This helped us build rapport and allowed participants to feel engaged with the research process. By having our OIT design a web application that met our research needs, we were also able to track usage by participants. Finally, participants in our discussion forum had the opportunity to upload documents and resources, ask questions of their own, and learn collaboratively.

Despite these strengths, a web-based discussion forum does present a number of limitations for the collection of qualitative data. Like other focus groups, data are from one person’s perspective per practice, and the group may influence responses (Tates et al., 2009). Additionally, we were not able to see body language and nonverbal cues, although we did receive some detailed written descriptions that explicitly expressed emotion, for example, through use of capital letters, exclamation points, and emoticons. These descriptions may be more easily and accurately analyzed and interpreted than nonverbal cues (Williams et al., 2012). Partly because of this lack of access to non-verbal cues, we also found the online forum more challenging to moderate then a face-to-face focus group. We were not able to probe individual responses as we would in a face-to-face conversation, so some answers may not be as deep or rich. We also found it challenging to prompt more quiet participants because we did not know whether or not they were online. More research is needed on best strategies to engage inactive participants and lurkers in online discussion forums (Sun, Rau, & Ma, 2014).

Another limitation to the asynchronous format is the potential decrease in spontaneous responses and fluid discussions (Wilkerson et al., 2014). For some, having discussions and expressing opinions may be easier when there is face-to-face contact (Zwaanswijk & van Dulmen, 2014). Moreover, in conducting internet-based research, we may have introduced a sampling bias, as participation required users to be comfortable and competent with using the internet in general and with navigating our site in particular. However, with the proliferation of internet and cell phone use for social networking (Pew Research Center, 2013), progressively more people are accustomed to communicating electronically via text-based platforms. Finally, the need for technological support to design the web application and respond to problems throughout the study period are additional costs not present in traditional face-to-face focus groups, and could be a limiting factor for some. While our team used a custom development approach, others considering doing this work might find existing supported software could save on programing time, personnel, and costs. Free online social platforms, such as LinkedIn and Yahoo or Google groups, or nominal cost ones, such as Ning or eXo Platform, may provide opportunities for researchers on limited resources and timelines, so that full attention can be paid to the study forum itself, rather than technology issues. Our study demonstrated the kind of functionality that existing software would need, such as privacy and moderation controls, account management, blog features, automated email notifications of new content, automated reminder emails, export of conversation threads, tracking of participant usage, and automated rewards system.

In conclusion, an asynchronous online discussion forum is a feasible and effective method to conduct a qualitative study, particularly when subjects are health professionals or hard to reach. Our experience and lessons learned with care coordinators in primary care provide recommendations to other researchers who are interested in using this method for health services research. As the concept of the PCMH and practice transformation sweeps the US, and new or changing workforce roles become commonplace (Naughton, Adelman, Bricker, Miller-Day, & Gabbay, 2013; Willard-Grace et al., 2013), an online forum can be an efficient tool to understand perceptions of innovative health professional roles.

Biographies

Jeanne M. Ferrante, MD, MPH, FAAFP, is a professor in the Department of Family Medicine and Community Health, Rutgers-Robert Wood Johnson Medical School and Rutgers-Cancer Institute of New Jersey, New Brunswick, New Jersey, USA.

Asia M. Friedman, PhD, is an assistant professor in the Department of Sociology and Criminal Justice, University of Delaware, Newark, Delaware, USA.

Jenna Howard, PhD, is a research analyst in the Research Division, Department of Family Medicine and Community Health, Rutgers-Robert Wood Johnson Medical School, New Brunswick, New Jersey, USA.

Eric K. Shaw, PhD, is an associate professor in the Department of Community Medicine, Mercer University School of Medicine, Savannah, Georgia, USA.

Deborah J. Cohen, PhD, is an associate professor in Department of Family Medicine, Oregon Health & Science University, Portland, Oregon, USA.

Laleh Shahidi, PhD, is Vice President of California Health Medical Reserve Corps and Principal Consultant of APS Consulting, San Ramon, California, USA.