From the Eyes of the Front Line: BCBAs Evaluate BAP

Abstract

Master's-level Board Certified Behavior Analysts^® were emailed an anonymous webbased survey regarding the journal Behavior Analysis in Practice (BAP). Following a 96-hour response window, 284 completed surveys were obtained. Data revealed that many participants regard the journal as beneficial to their profession, yet considered it too expensive and in need of a sharper focus on practitioner issues. Most supported BAP's emphasis on empirical research, and many recommended additional content areas such as clinical case formulations, objective product information, and a layout that blends the features of an empirical journal and a popular magazine. In summary, this survey indicates that, as a practitioner-oriented journal, BAP has the potential to occupy a valuable niche for master's-level behavior analysts and that journal leaders might enhance its value by modifying its marketing, content, and structure.

As we dive headfirst into the electronic age of instant information and access to content anytime anywhere, many traditional print publications are finding it challenging to attract readers. For example, because of falling numbers of subscribers, Newsweek magazine, after 79 years in print, became a web-only publication this past year (Haughney & Carr, 2012). Scientific journals have struggled as well. Important scientific discoveries are much less likely to appear in prestigious journals than they were a generation ago (Lozano, Lariviere, & Gingras, 2012). Publication in such journals has become less critical as web-based search engines like Google Scholar have given readers unprecedented access to articles in a wide array of journals.

In addition to adapting to the electronic age, journals must always maximize their market. As with any other consumer product, leaders of a journal need to understand the market, target the market analytically, and endeavor to create a product that edges out competition. While one may not think of a scholarly enterprise like a journal as a business, it is a business indeed (Van Noorden, 2013). Revenue generated through subscriptions and citations that increase impact factor are akin to the outcome measures found in other marketplaces such as net profits, growth margins, and return on investments. These dollars and numerical metrics reflect the many intertwined processes that make the peer-reviewed journal what it is, such as the prestige of its authors, the topics it covers, and the quality and quantity of its content. In order to survive, a journal, like any other product, needs to have value to its intended market.

In 2008, the Association for Behavior Analysis International founded the peer-reviewed journal Behavior Analysis in Practice (BAP) as an outlet for delivering scientific information to practitioners. Its mission has been to target the exponentially increasing marketplace of Board Certified Behavior Analysts^® (BCBAs^®) and promote dissemination of practical information to all frontline, clinically orientated behavior analysts. Over the past 5 years, BAP has published 10 issues for the practicing community with a total of 21 empirical articles, 3 product reviews, and 20 discussion papers.

An easy and cost-effective approach to understanding the needs of the consumer is through market-based surveys (Cook, Heath, & Thompson, 2000). Such surveys can help organizations align their product with consumers' wishes and identify novel ways to improve the product (Manzi, 2012). In the case of BAP, such a survey could yield data-based information on impediments to market growth and strategies to make a strong product even stronger. Organizational leaders are prone to erroneously assuming they know best about what consumers want. For example, J. C. Penney lost 20% in sales during the first quarter of 2011 after radical pricing policy changes (Loeb, 2013); Dish Network outraged customers when it dropped popular channels; and Nokia's stock dropped 60% after the company embraced the Microsoft operating system and ignored iPhone (Trefis Team, 2013). In the end, the voice of the target audience determines the eventual success of any product or company (Belk, 1975).

To obtain consumer input on BAP, we surveyed master's-level BCBAs (the primary target audience for the journal) regarding impediments to subscribing/resubscribing, content preferences, physical layout, and general opinions about BAP. To reduce the risk of response bias such as answering based on social desirability, we presented survey questions in diverse formats, including Likert-scale ratings, forced-choice rankings, and sliding scales, with some questions focused on barriers and others focused on preferences (Furr & Bacharach, 2008). We conducted the survey during a 96-hour window, closing it when we began receiving fewer than 10 new responses per day.

Method

Email Campaign

We obtained permission from the Behavior Analyst Certification Board (BACB) to target our email campaign to master's-level Board Certified Behavior Analysts (BCBAs), who comprise the primary audience of Behavior Analysis in Practice (BAP). On our behalf, the BACB emailed a brief message inviting recipients to participate in a brief survey regarding BAP. The instructions indicated that the “survey will be used to inform the future of the journal and understand BAP's readership base.” Recipients were told that the results would remain confidential. The body of the message contained a hyperlink to the survey (described below). The email campaign began at approximately 8:00 am EST on Wednesday, April 10, 2013, and closed approximately 96 hr later (8:00 am on Sunday, April 14, 2013) after the rate of new responses fell below 10 per day. We obtained 284 responses during this interval. Because BCBAs can opt out of email solicitations on their BACB portal, the number of addresses to which the email was sent is unknown (there were 9,445 BCBAs at the time of the email campaign, according to http://bacb.com; the number consenting to receive third-party email solicitations through the BACB mass email service is undisclosed).

Survey Hosting

The survey was generated through Qualtrics Research Suite (http://www.qualtrics.com/researchsuite/#enterprise), which is an online survey software package that enables users to generate surveys for large Internet distribution campaigns and use descriptive statistics to summarize the findings.

Participants

Participant demographics were obtained through a series of questions at the conclusion of the survey but are presented first here in order to describe the individuals who provided data. Of the 284 participants, the majority was female (N = 209, 74%). Participants reported an average age of 36.98 years (SD = 10.07). Participants overwhelmingly reported that “applied behavior analysis” was their primary professional discipline (78%), with “education” and “special education” tied at a distant second (6%). Most of the remaining participants listed their primary discipline as social work (2%), behavioral psychology (2%), or school psychology (2%). Nearly half (43%) indicated that “consultant/staff trainer” most closely describes their current professional position. Other positions consisted of “psychologist/therapist/clinician” (27%) and “administrator” (12%). Six percent of the remaining participants reported being “students,” with another 5% indicating that they are “school teachers.” Most participants' main professional activity was “clinical/therapeutic service” (34%) or “consulting” (30%). Other notable primary pro fessional activity responses were “administrative/management” (11%), “teaching” (8%), and “staff/parent training” (6%).

Survey Items

All survey items featured a forced response option such that participants had to make a selection before proceeding with the survey.

Subscription status. Participants were asked to indicate whether they had ever paid for a subscription for BAP in the past and whether they were currently a subscriber using yes/no multiple choice item types (with only one answer permitted) across the two questions.

Barriers to [re]subscription. Participants were asked to indicate whether any of the following statements [re] subscribing using a matrix table with a Likert-rating system ranging from Strongly Disagree to Strongly Agree (5 scale points, including Disagree, Neither Agree nor Disagree, and Agree): (a) price is too high, (b) I have no interest in practitioner issues, (c) the content of the journal has not fit my needs, (d) I do not find the journal to be a contribution to the field, (e) I do not value the journal's mission, (f) I do not see the relevance of the journal given other competing journals (e.g., RASD, JABA, the Behavior Analyst, Behavioral Interventions), (g) the direction of the journal was not what I expected, (h) the quality of the writing was poor, and (i) the rigor of the research was subpar.

A second multiple choice item (one answer permitted) asked participants to indicate one aspect that the BAP editorial board should change in the journal to make practitioners more excited about BAP. Options included: (a) nothing, (b) the price of subscription, (c) focus more on practitioner issues, (d) change the format to become more of a scientific journal, (e) change the format to become more of a magazine, (f) feature more papers on autism or developmental disabilities, (g) feature more papers on behavior analytic issues unrelated to autism/DD, or (g) “other.” If participants selected “other,” they were allowed to enter a textual response indicating what change they would most like to see.

Reasons to [re]subscribe. Participants were asked to indicate whether any of the following changes would increase their chance of [re]subscribing using a multiple choice item (more than one answer permitted): (a) more papers written at a level practicing behavior analysts could understand; (b) topics that are broader than what has been found in prior issues of BAP; (c) product reviews of the latest behavioral gear, stimulus materials, technology, and software; (d) shorter articles with clear summaries; (e) papers that appear similar to those found in the Journal of Applied Behavior Analysis (JABA); (f) an on-time publication cycle; (g) the bundling of the journal with your renewal of membership to ABAI; (h) an option to subscribe when you are renewing your BCBA; or (i) not applicable because they already subscribe to BAP.

Preference for future content. Participants were asked to indicate their preference for future BAP content across three questions. In the first question, participants were asked to rank order (by sliding the items up and down in a list) the article types they would most prefer to see in BAP. Article types corresponded to existing BAP manuscript categories: (a) discussion papers, (b) empirical papers, (c) product reviews, (d) review papers, and (e) tutorials. The second question regarding preference for future content read, “If BAP were redesigned visually and in content, what would you like to see the final product resemble?” To answer the question, participants were instructed to slide a marker across a bar with JABA on the left anchor and Time Magazine on the right. The marker was initially anchored exactly between the two options in an attempt to reduce possible anchoring effects toward either. For the sake of scoring, an extreme left selection indicating 100% JABA was scored as “0,” while an extreme right selection indicating 100% Time magazine was scored as “100.” The final item in this category was presented in a multiple choice format (more than answer permitted) that asked participants to indicate whether they would like to see any of the following published in future issues of BAP: (a) advertisements for products, (b) advertisements for agencies, (c) advertisements for job openings, (d) interviews about products, (e) interviews with practitioners, (f) discussion of resources available to practitioners, (g) highlights of published research outside of BAP, (h) announcements of upcoming conferences of interest to practitioners, (i) announcements of news of relevance to practitioners, or (j) subscriber/reader submitted questions and answers to prominent behavior analyst.

Results

Subscription status. Of the 284 participants, only 32% (N = 90) reported ever subscribing for BAP. Only 20% of respondents indicated that they are presently subscribers. Data on whether participants ever subscribed and are currently subscribed are depicted in the left and right panels of Figure 1, respectively.

Figure 1.

Participant responses to questions regarding subscriptions to BAP. The figure on the left represents the proportion of participants who have ever subscribed to BAP, while the figure on the right depict the proportion of participants with active subscriptions.

Barriers to [re]subscription. Figure 2 depicts the number of participants reporting levels of agreement concerning various barriers to subscribing or resubscribing to BAP. Likert-ratings were used to obtain numerical values of agreement. Scores could range from 1 (Strongly Disagree) to 6 (Strongly Agree), with a score 3 representing indifference. As depicted in the figure, the strongest levels of agreement were found for “Price is too high.” Conversely, the item feature strongest disagreement read, “I have no interest in practitioner issues.” “Price is too high” was the only barrier with a mean rating above 3 (M = 3.35, SD = 0.92), representing indifference or slight agreement with the statement. Participants strongly disagreed with the statement “I have no interest in practitioner issues” with a mean rating of 1.47 (SD = .71). The other items represent general disagreement were “I do not value the journal's mission” (M = 1.86, SD = 0.83) and “I do not find the journal to be a contribution to the field” (M = 1.95, SD = 0.86). All remaining items fell in the disagreement to indifference range.

Figure 2.

Participants' ratings of agreement to statements regarding potential barriers to BAP subscriptions. Numbers of agreement category endorsements (e.g., Strongly Agree) are plotted across the x-axis for each barrier statement plotted on the y-axis.

When asked to identify one aspect of the above barriers that the BAP editorial board should change, 33% of participants (N= 93) selected “Lower the price of subscription.” This represented the most endorsed item. The second most endorsed item was “Focus more on practitioner issues,” constituting 21% of responses. These results corroborate the concerns raised in the barriers to subscription question. Interestingly, the third most endorsed item was “Feature more papers on behavior analytic issues unrelated to autism/DD,” with 13% of participants indicating this response. The least endorsed item was “Change the format to become more of a magazine” (4%). A graphical depiction of these data is presented in Figure 3. “Other” was endorsed by 8% of participants. In the open-ended response section, many participants reported a desire for a more regular publication cycle with shorter periods between issues. Some participants also indicated that the peer review process could be expedited. Many participants reported that the journal should strictly adhere to its stated mission and publish articles that make behavior analysis consumable to practitioners and care providers both within and outside of behavior analysis. Finally, a number of participants reported that BAP should do more marketing, as they had never seen or heard of the journal before. (Of note, 4 participants emailed us directly to say that they were unaware of the journal's existence; it is unknown whether these respondents were the same participants reporting unfamiliarity with BAP in the open-ended response item).

Figure 3.

Horizontal bar plot depicting the percentage of participants endorsing items as the number one change that would excite subscribers.

Reasons to [re]subscribe. Figure 4 depicts participants' endorsements of items that would increases their likelihood of subscribing or resubscribing to BAP. The most frequently endorsed item was “An option to subscribe when you are renewing your BCBA” (53%). A close second was “The bundling of the journal with your renewal of membership to ABAI” (46%). The least endorsed item was “Topics that are broader than what has been found in prior issues of BAP” (11%). All other items ranged between 15% and 18%. The sum of percentages is greater than 100% because participants were asked to select all items that applied.

Figure 4.

Horizontal bar plot depicting the percentage of participants endorsing items as potential changes to BAP that would increase the likelihood of subscription. Participants were allowed to endorse any applicable items. Thus, the sum of percentages is greater than 100%.

Preference for future content. When asked to rank order article types that they would most like to see in future issues, participants overwhelmingly rated Empirical Reports as the top choice. With a value of 1 representing the top ranked article type, Empirical Reports featured a mean of 1.87 (SD = 1.18). The least preferred article type was Product Reviews (M = 4.19, SD = 1.07), with all others generating approximately equal ratings with a slight bias for Discussion Papers (M = 2.61, SD = 1.18). Figure 5 depicts the raw number of times each article type was ranked as number one by a participant, corroborating the quantitative data described above.

Figure 5.

Stacked bar graph depicted total number of participants rating each article type as the top preferred.

The next item asked participants to slide a bar between the words JABA and Time Magazine to indicate their preference for what BAP should visually resemble if it were to be redesigned. A selection to the far left (JABA) was scored as a 0 with a far right selection scored as 100 (Time Magazine). As depicted in Figure 6, the average rating was 42.88 (SD = 23.03), indicating a preference for a combination of scientific journal and magazine attributes, with a slight preference for scientific journal features.

Figure 6.

Average participant response to rating of preference for visual redesign of BAP—between JABA (0) and Time Magazine (100)—is depicted by the vertical black bar (SD depicted by error bars).

The final item asked participants to select features that they would like to see in future issues of BAP. Responses are depicted in Figure 7. The most endorsed item (66%) was “Discussion of resources available to practitioners,” followed by other major endorsement such as, “Announcements of news of relevance to practitioners” (55%), “Subscriber/reader submitted questions and answers to prominent behavior analysts” (49%), and “Highlights of research published outside of BAP” (49%). The least endorsed items pertained to advertisements for agencies (5%) and products (10%).

Figure 7.

Horizontal bar plot depicting the percentage of participants endorsing items as additions they would like to see in future issues of BAP. Participants were allowed to endorse any applicable items. Thus, the sum of percentages is greater than 100%.

Discussion

With the assistance of the BACB, we surveyed masters-level BCBAs to find out what information would be valuable to them as professionals and how this journal could better serve them. Most of the 284 participants expressed interest in obtaining information about interventions, resources, news, and practice issues, consistent with the current mission of BAP. They gave the highest priority to empirical reports, which have been the most common type of article published in the journal. However, participants also identified a need to reduce barriers to subscribing, improve the quality and relevance of articles, and feature new types of information. We consider each of these issues in turn.

Barriers to Subscribing

Only about one-third of the participants had ever subscribed to BAP, and a substantial minority of this subgroup had let their subscriptions lapse. Many reported that the price was too high. In written comments, some participants commented that they routinely received issues late or not all. A few remarked that they were completely unaware of the journal until they received the survey.

Fortunately, participants' responses also suggest a potential solution to these difficulties. About half said they would like to see subscriptions to the journal bundled with renewal of their BCBA. A similar proportion supported bundling the journal with renewal of their membership in ABAI. Although many logistical challenges would need to be overcome in order to set up this kind of “package deal,” it is a possible way to control costs, ensure that BCBAs or ABAI members are informed that the journal is available every time they renew, and encourage publication of each issue on schedule. (Subscribers are much more likely to renew if they have received all the issues included in the subscription.) Thus, while we cannot guarantee success, we are actively exploring ways to bundle BAP subscriptions with renewals of the BCBA or ABAI membership.

Quality and Relevance of BAP

Many participants were undecided or doubtful that the journal maintains high standards, meets their needs, and occupies a niche that sets it apart from journals aimed at researchers. Indeed, some written comments characterized the journal as a replica of the Journal of Applied Behavior Analysis and exhorted the editors to make the content more practitioner-friendly. More than one-fifth of the participants ranked this change as their top priority. A handful of participants highlighted specific BAP articles that they found especially useful, notably articles that offered step-by-step guidelines on how to deliver particular ABA interventions in “real world” settings. Many participants also expressed a preference for shorter, more colloquial articles.

In combination with the finding that empirical reports are the most valued type of article in the journal, we interpret the call for greater relevance to practitioners as evidence that we need to give clearer guidance to authors of such reports. The instructions to authors already state that empirical reports are appropriate for BAP when “[t]he procedures and findings are directly transferable to service delivery” and “concrete guidelines for practice are included.” We plan to operationalize this requirement by requiring authors to include a list of bullet points summarizing what practitioners should take away from the article and by instructing reviewers to evaluate submitted manuscripts based on whether they are practitioner-oriented (i.e., emphasizing how to use the findings in practice, as opposed to discussing theoretical implications or directions for future research).

New Types of Articles

Although skeptical of advertisements for products or agencies, most participants professed a desire for information about professional resources, news, and recent research findings, as well as features such as question-and-answer with experts and interviews with practitioners. Participants were divided on whether to feature more articles on autism and developmental disabilities or whether to increase the focus on other clinical populations. The implications we draw are that we do not have a clear mandate to change the proportion of articles on autism or developmental disabilities relative to articles on other populations, but we should seek to broaden the types of articles published in the journal. Accordingly, with this issue, we are introducing two new features: (1) an objective forum for reviewing new products, and (2) field reports from community practice settings that illustrate successful deployment of ABA interventions outside of specialized centers. We will continue to seek ways to provide a range of information that will be useful to practitioners.

Limitations

The main caveat about our survey findings is that the participants were only a subset of masters-level BCBAs. The BACB emailed the survey on our behalf to a list of BCBAs whose identities are confidential. As a result, we have no way of calculating what percentage of BCBAs who were invited to complete the survey actually did so, nor can we determine whether the views of survey completers are similar to or different from the views of noncompleters. Nevertheless, we did obtain responses from a large number of participants and believe that their feedback helps establish a future direction for the journal.

Concluding Remarks

Our survey shows that a large majority of the participants value BAP's mission to serve ABA practitioners and esteem the types of articles published in the journal. Our tasks ahead may be to improve customer service, boost the visibility and accessibility of the journal, sharpen the focus on practitioner issues, and expand the range of practitioner-oriented information contained in the journal. When consumers indicate the means by which a valued product can become even better, it seems logical to let the marketplace guide subsequent versions of the product.