An Evaluation of Multiple SAFMEDS Procedures

Abstract

Lindsley developed the “say all fast minute every day shuffled” (SAFMEDS) procedure in the late 1970s to enhance the typical use of flash cards (Graf & Auman, 2005). The acronym specifically guides the learner’s behavior when using flash cards. A review of SAFMEDS research indicates its successful use with children, college students, and older adults with and without disabilities. The literature also indicates that SAFMEDS procedures are not well documented and have multiple variations, limiting practitioners’ ability to know what procedures to use and when. The purpose of this study was to evaluate the effects of a basic SAFMEDS procedure and four supplementary SAFMEDS procedures on the rates of correct and incorrect responding to unfamiliar Russian words and Chinese characters in college students. The results of the study suggest that the basic SAFMEDS procedure produced some learning (i.e., increases in correct responding and decreases in incorrect responding), but all of the supplementary procedures led to greater increases in the number of correct responses per 1-min timing. Further research evaluating differences in performance across the supplementary procedures is warranted.

Precision teaching (PT) is a supplemental strategy emphasizing the frequent measurement of student performance, which in turn allows teachers to make instructional decisions (Merbitz, Vieitez, Merbitz, & Pennypacker, 2004; West, Young, & Spooner, 1990). Hallmarks of PT are the frequent measurement of learning, standard graphing of data, analysis of data to guide instructional decisions, and instructional changes that maximize learning (West et al., 1990). One strategy linked to PT is “say all fast minute every day shuffled” (SAFMEDS).

SAFMEDS is an instructional and assessment strategy similar to flash cards. Lindsley coined the term in the late 1970s to specify procedural differences between SAFMEDS and traditional flash card procedures (Eshleman, 2000a; Graf & Auman, 2005; Lindsley, 1996). Flash card procedures are less structured and do not necessarily emphasize important areas of instruction, such as making an observable response (i.e., saying the answer). It is difficult to determine the “gold standard” SAFMEDS procedure because so many procedural variations exist. However, Eshleman (2000a) and Graf and Auman (2005) described a basic procedure for SAFMEDS that is often referred to in the literature. Their procedure consists of the following steps, in which the learner

1. Holds the deck,

2. Shuffles the cards,

3. Starts the timer (typically 1 min),

4. “Sees” the front of the card and “says” the term on the back of the card out loud (a see/say learning channel, although other learning channels can be used; see Graf & Auman, 2005, for further discussion),

5. Turns the card over and checks the answer for immediate feedback,

6. Sorts the cards for correct and incorrect responses into piles,

7. Counts the number of responses per pile after the time expires, and

8. Charts their own performance for review and instructional changes (see Calkin, 2005, for further information regarding a standard celeration chart [SCC]).

The simplest SAFMEDS procedure (based strictly on the guidelines suggested within the acronym) involves conducting a single timing per day using the aforementioned procedures. However, Eshleman and Graf and Auman provided in their how-to guides other examples of how SAFMEDS instruction and assessment might be implemented. They suggested that multiple timings per day could be used instead of just one, the number of cards within the deck could be increased and decreased based on learner performance, and the length of the timing could be increased or decreased.

Since its inception, SAFMEDS has been evaluated to determine its effects on learning (e.g., Bolich & Sweeney, 1996; Byrnes, Macfarlane, Young, & West, 1990; Eaton & Fox, 1983; Eshleman, 1985) and the maintenance of learning (Kim, Carr, & Templeton, 2001; Olander, Collins, McArthur, Watts, & McDade, 1986). In addition, comparisons between different SAFMEDS procedures (e.g., teacher vs. student deck development; McDade & Olander, 1990) and modalities (e.g., computer-based vs. paper deck; McDade, Austin, & Olander, 1985) have been made, as well as between factors affecting performance outcomes (e.g., error rate; Bower & Orgel, 1981; stimulus control; Meindl, Ivy, Miller, Neef, & Williamson, 2013). A review of this literature indicates that SAFMEDS increases the number of correct answers across varied content (e.g., math facts, reading fluency, positive self-statements), various populations (i.e., elementary education, secondary education, university, and geriatric), and various settings (e.g., education, home, and community; Quigley, Peterson, Frieder, & Peck, 2017).

Anecdotally, we have used SAFMEDS in our own instruction with college students, and we are aware of others who do so as well. It is not uncommon for instructors to “tinker” with the SAFMEDS procedure to meet the needs of students. For example, our own students have expressed that it is overwhelming to be presented with a large deck of cards at the beginning of a semester that must be mastered by the end of the semester. As a result, we have divided the cards into sets so that the students could work on smaller subsets of the cards. Sometimes, these subsets are combined as the semester progresses so that students have an increasing number of cards in their decks. In addition to making the task less daunting at the onset of the semester, this procedure also has the benefit of not introducing cards for concepts that have not yet been taught in the semester. This “tinkering” with the procedure has produced a literature base that is not entirely clear as to what the “standard” SAFMEDS strategy is. Furthermore, it is unclear what modifications to the procedure can be made, when those modifications should be made, and whether those modifications reliably produce better learning.

To illustrate, Quigley et al. (2017) reviewed 27 SAFMEDS publications and noted the SAFMEDS procedures implemented by researchers. Of the 27 publications, none of the researchers utilized a SAFMEDS procedure that aligned with the basic procedure described previously. Further, 11 of the publications did not contain a sufficient description of the SAFMEDS procedures to ascertain what procedure was implemented. Of the 16 studies with a description of the SAFMEDS procedures, the procedures varied along the parameters of the number timings conducted per day, whether the learner had the opportunity to review the cards prior to the timing, the length of the timings (e.g., 30 s, 1 min), the total number of cards in a deck, and the error correction procedures used. Quigley et al. also noted that researchers also occasionally implemented multiple SAFMEDS procedures within the same research project. For example, Beverly, Hughes, and Hastings (2009) added additional timings, altered the total number of cards reviewed, and introduced error correction procedures based on student performance, all within the same study. A clear understanding of intervention procedures is critical for any sort of discussion of validity. Yet, all of the procedural variations found in the literature make it difficult to determine what the effects of various SAFMEDS procedures are and when to utilize each variation to maximize learning.

According to Kazdin (2003), internal validity is the ability to rule out explanations of results other than the designated intervention. Additionally, Kazdin (2003) stated that external validity is the extent to which the results are applicable to other populations, settings, and circumstances beyond the research. A clear description of the intervention is necessary to evaluate potential confounding variables for internal validity. Furthermore, a clear description is needed when evaluating how the intervention might be applicable to other populations, settings, and circumstances beyond the research. Internal and external validity is increased when studies are replicated across time using the same intervention procedures. Clear descriptions of the intervention impact our ability to establish internal and external validity, which in turn impacts effective practice.

Currently, the literature base on SAFMEDS provides an unclear description of the SAFMEDS procedure, making it difficult for a practitioner to determine what constitutes evidence-based practice when implementing them. Research is needed to clearly identify how different variations of the SAFMEDS procedure impact learning. Thus, the purposes of this study were to clearly define multiple SAFMEDS procedures and evaluate the effects of various SAFMEDS procedures on the correct and incorrect responses to vocabulary terms of a novel language.

Method

Participants

Nineteen participants enrolled at a midwestern university were recruited for participation in the study through flyers and in-class recruitment scripts. Participants had varied educational backgrounds (e.g., GPA, enrollment status), varied exposure levels to SAFMEDS, and varied histories of exposure to Chinese and Russian languages. Criteria for inclusion in the study included the participants having basic reading skills, vocal language, no physical disabilities that limited hand/finger dexterity, no visual impairments, and no history of learning related to the target languages (i.e., the participants had no formal training in the Chinese or Russian language, neither the Chinese nor Russian language was spoken at home, and the participant had not resided in a location where the Russian or Chinese language was primarily spoken). The inclusionary criteria were assessed via a questionnaire. All 19 volunteers met these inclusionary criteria and enrolled in the study. However, only 14 of the volunteers completed the study. Attrition was largely due to conflicts with the time commitment of daily research sessions. See Table 1 for additional participant demographics.

Table 1.

Participant Demographics

Education information					Language history			SAFMEDS exposure
BA/BS	Master’s	PhD	GPA < 3.0	GPA > 3.0	Second-language training	Years of training < 2	Years of training > 2	Total number
8	5	1	2	12	12	3	9	9

Note. Each number reflects the total number of participants meeting the demographic criteria. The degree reflects what degree level the participant was enrolled in at the time of participation.

Settings and Materials

Sessions were conducted in a small room at the university. The room measured approximately 3 m × 2 m and contained tables and chairs. The participant and one or two researchers were present for every session; a second researcher was occasionally present for live scoring of interobserver agreement (IOA) data. Materials consisted of two timers (i.e., stopwatch and digital countdown time), three different decks of SAFMEDS cards (i.e., training deck, Chinese deck, and Russian deck), a demographics questionnaire for assessing inclusionary criteria, and video-recording equipment. All sessions were video recorded for later data review purposes.

The SAFMEDS cards were approximately 7.6 cm × 12.7 cm and were created by printing the relevant information on both sides of card stock and then cutting the cards out. There were three decks of cards: a training deck, a Russian deck, and a Chinese deck. The training deck consisted of 20 cards, each with a picture on one side and a term on the other. The picture/term combinations were created to train participants on how to complete the SAFMEDS procedure. In an attempt to train participants in the procedures for when they made a correct response, some cards had a common picture on one side (e.g., a picture of a car) and a corresponding common term on the other side (e.g., “car”). In an attempt to train participants in the procedures for when they made an incorrect response, some cards had a common picture on one side (e.g., a picture of an 18-wheeler truck) with an obscure corresponding term on the other side (e.g., “big rig”). Last, in an attempt to train participants in the procedures for when they wanted to “pass” on a response, some cards had an unfamiliar and obscure picture on one side (e.g., a picture of a dragon fruit) with the corresponding term on the other side (e.g., “dragon fruit”).

The Russian and Chinese decks were created in a similar manner and differed from the training deck only in content and total number of cards. One deck had Mandarin Chinese characters on one side with the English translation on the opposite side. The other deck had Russian words on one side with the English translation on the opposite side. Both decks contained 60 cards with the same English words across decks (e.g., 月 = moon and Луна = moon; 山 = mountain and ropa = mountain). Correspondence between languages was determined via Google Translate (http://translate.google.com/). All information on the fronts and backs of the cards appeared in the center of the cards. To avoid stimulus control issues such as smudge marks or bent corners evoking responses instead of the printed words (see Eshleman, 2000a, 2000b, and Meindl et al., 2013, for further discussion), multiple identical decks were created and rotated throughout sessions. Cards with any visible marks, smudges, tears, or other defects were replaced with new cards.

Dependent Variables and Data Analysis

The dependent variables were the number of correct responses and incorrect responses participants made to the cards in each of the decks. A correct response occurred when the participant stated out loud the term associated with the character/word on the other side of the card. For example, if the participant saw the Chinese character for “boy,” they vocally stated “boy” for a correct response. An incorrect response occurred when the participant stated a term other than what was printed on the card. For example, if the participant saw the Chinese character for “boy” and they said “girl,” this was counted as incorrect. If the participant provided more than one response for a single card, the final response was scored according to the aforementioned directions. For example, if the participant saw the Chinese character for “boy” and said “Girl . . . no, boy,” it was counted as correct so long as the final response occurred prior to seeing the back side of the card. Participants could state “pass” if they were unsure of the correct answer. If the participant passed, they still flipped the card over to see the correct response. We counted passes as incorrect responses. There were no time constraints within the 1-min timing to produce a response for each individual card. The primary observer watched video recordings to record responses.

Metrics for level, celeration, and bounce were calculated using the statistical software PrecisionX (CentralReach, 2020). These values were analyzed in combination with visual analysis.

Interobserver Agreement

IOA data were recorded by having a second observer independently watch the videos and record participant responses to the cards. IOA data were calculated using a point-by-point agreement method (Kazdin, 1982) across a minimum of 25% of sessions, distributed equally across all phases for each participant. Overall agreement averaged 99% (range 89%–100%) across all participants and phases of the study.

Independent Variables

Five independent variables were implemented and evaluated. The first independent variable was a package of variables that composed the basic SAFMEDS procedure. The other four independent variables consisted of this basic procedure along with one variation of the procedure that had been documented in Quigley et al. (2017). In these cases, all aspects of the procedure were the same as the basic procedure except for the variable that was modified. These four modifications consisted of a whole-deck presentation of cards, an incremental whole-deck presentation of cards, three 1-min sprints prior to timings, and incremental sprints (three 1-min timings with a subset of cards prior to the timing). All participants experienced the basic SAFMEDS procedure for both the Chinese and Russian decks. Each participant then experienced one of the four modified SAFMEDS procedures for one of the decks while continuing the basic SAFMEDS procedure for the other deck. For example, Participant 1 first experienced daily 1-min timings using the basic procedure for both the Chinese and Russian decks for 2 weeks. At the start of the 3rd week, Participant 1 continued using the basic procedure for the Russian deck but experienced the whole-deck procedure for the Chinese deck. The selection of which additional SAFMEDS procedure each participant experienced was randomly determined. Table 2 provides a summary of each independent variable; more detail is provided in the descriptions of each independent variable. In the following descriptions, the testing timing refers to the 1-min timing conducted as part of the basic procedure. For any supplemental procedures that included other timings (e.g., sprints), the testing timing refers to the final 1-min timing that followed the practice timings. We only report the results of the testing timing for each participant.

Table 2.

Description of IV Components

IV	Material review outside of test time?	Time for outside review	Practice timings?	Number of practice timings before the test timing	All cards in the deck?	Test time
Basic	No		No		Yes	60 s
Whole deck	Yes	≤10 m	No		Yes	60 s
Incremental whole deck	Yes	≤10 m	No		No	60 s
Sprints	No		Yes	3	Yes	60 s
Incremental sprints	No		Yes	3	No	60 s

Note. IV = independent variable. If a cell is blank, the components were not present for that independent variable

Basic SAFMEDS Procedure

The basic SAFMEDS procedure was based on the procedures described by Eshleman (2000a) and Graf and Auman (2005). Participants were told,

You will have 1 minute to say as many terms as you can. If you don’t know the term, you can pass on the card. If you pass, you should still flip the card over to see the correct response.

Participants then held the deck, shuffled the deck, started a countdown timer set for 1 min, looked at the stimulus on the front side of the card (i.e., see), provided a vocal response (i.e., say the answer or say “pass”), flipped the card over to expose the correct answer and to determine whether the stated answer was correct or incorrect, and placed the card onto a corresponding pile (i.e., correct, incorrect, or pass). The participants repeated this procedure with as many cards as possible during the 1-min timing.

Whole-Deck Procedure

This procedure was based on the procedures described by Eshleman (2000a). Participants were provided up to 10 min to study the cards in any manner. If the participant studied the cards for the entire 10 min, the researcher stopped the participant and conducted the 1-min test timing. The participant could stop the 10-min study period at any point and proceed to the 1-min test timing. Participants were told, “Review each card as quickly as possible. If your review continues for 10 minutes, you will be stopped.”

Incremental Whole-Deck Procedure

This procedure was based on the procedures described in Kim et al. (2001). The incremental whole-deck procedure was identical to the whole-deck procedure (i.e., study time followed by test timing), except that participants were only given a portion of the cards from the deck. Rather than being given all 60 cards in the deck, participants were given a subdeck of 15 cards at a time. However, 4 duplicates of each card were provided so that a total of 60 cards were still presented. The cards for the test timing were the same ones used during the practice period. Subdecks containing novel cards (i.e., 15 new cards with 4 duplicates) were introduced at different points. After the participant had experienced each subdeck, the 15 cards from each deck were combined together (i.e., 60 cards, each containing a different word pair). Participants were told, “Review each card as quickly as possible. If your review continues for 10 minutes, you will be stopped.”

Sprints Procedure

This procedure was basedupon procedures described by Eshleman (2000a) and Stockwell and Eshleman (2010). Participants practiced the word pairs by conducting three 1-min timings following the basic procedure (described in the introduction), using the full deck. Each timing was completed back-to-back with less 1-min between timings. Following the three practice timings, a fourth timing, the test timing, was conducted. Participants were told,

You will have 1 minute to say as many terms as you can. If you don’t know the term, you can pass on the card. If you pass, you should still flip the card over to see the correct response.

Incremental Sprints Procedure

This procedure was an extension of the procedures in Kim et al. (2001). The procedure was identical to the sprints procedure, except that the participant was only given a portion of the cards from the deck. As with the incremental whole-deck procedure, participants were only given 15 cards at a time with 4 duplicates of each of the 15 cards for a total of 60 cards. There were four different subdecks (i.e., each with 15 cards with 4 duplicates) and the complete deck (i.e., 60 cards, each containing a different word pair). After the participant had experienced each subdeck, the 15 cards from each deck were combined together (i.e., 60 cards, each containing a different word pair). Participants were told,

You will have 1 minute to say as many terms as you can. If you don’t know the term, you can pass on the card. If you pass, you should still flip the card over to see the correct response.

Fidelity of the Independent Variable

A fidelity checklist was used to determine if key procedural aspects of each SAFMEDS procedure (e.g., shuffling, number of timings, duration of timing, sorting of cards into correct piles) were followed across sessions and participants. Fidelity was assessed for at least 25% of the sessions across all phases. IOA data were also taken on fidelity measures by having a second observer simultaneously but independently complete the same fidelity checklist. All fidelity checks were conducted by watching video recordings of sessions. Fidelity IOA was calculated using a point-by-point agreement method (Kazdin, 1982). Fidelity averaged 98% (range 80%–100%) across all participants and phases. Fidelity IOA averaged 99% (range 80%–100%) across all participants and phases.

Experimental Design and Procedures

The research questions were evaluated using a simultaneous-treatment design embedded within in a concurrent multiple-baseline across-participants design (Kazdin, 1982). That is, each participant experienced two variations of SAFMEDS simultaneously, one with each deck (Chinese and Russian). However, the implementation of the variation strategy was staggered across participants. The simultaneous-treatment design allowed for within-subject comparisons of the basic SAFMEDS procedure and one of the supplementary SAFMEDS procedures. The multiple-baseline design provided the opportunity for replication of effects across participants and SAFMEDS procedures. The multiple baselines were discontinuous.

Basic SAFMEDS Procedure Training

Participants were trained on the basic SAFMEDS procedure with the training deck. Participants were told how to hold the deck, shuffle the deck, start the countdown timer, look at the stimulus on the front side of the card (i.e., see), provide a vocal response (i.e., say), flip the card over to determine whether the answer was correct or incorrect, place the card into a pile (i.e., correct, incorrect, or pass), and repeat the procedure with as many cards as possible during the 1-min timing. In addition to vocal instruction, participants watched a video demonstration of the basic SAFMEDS procedure, and the first researcher modeled the procedure. Next, participants were asked to implement the SAFMEDS procedure. Feedback was provided for correct and incorrect implementation. This was repeated until participants made at least 20 responses per minute with 80% or higher fidelity. All participants met these criteria in four or fewer practice timings.

Basic SAFMEDS Procedure for Chinese and Russian Decks

Following successful completion of training, participants met daily (if possible) with a researcher to complete test timings using the basic SAFMEDS procedure described previously for each of the Chinese and Russian decks. Participants only had access to the cards during the 1-min timing. This served as a baseline for both decks for each participant.

Introduction of Additional SAFMEDS Procedure

In a staggered fashion across participants, participants then experienced one of the four variations of the SAFMEDS procedure. Participants were randomly assigned to a procedure based on their order of enrollment in the study. The SAFMEDS variation was introduced for only one of the decks, whereas the participant continued to use the basic SAFMEDS procedure for the other deck. First, participants were trained on the new procedure using the same strategies as were used for the training deck. The deck for which the participant used the SAFMEDS variation (i.e., Russian or Chinese) was counterbalanced across participants.

Follow-Up Phase

The purpose of this phase was to briefly assess the short-term maintenance of performance following at least 10 days without access to the cards. Following at least a 10-day break, participants completed a single 1-min timing for each deck following the basic SAFMEDS procedure.

Results

Results for participants are provided in Figs 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 and 14, with summary data presented in Tables 3, 4, 5 and 6. Figures are grouped by participant and SAFMEDS procedure (i.e., the first figure is the basic SAFMEDS procedure, and the second figure is the supplemental procedure for the participant); the tables summarize the comparative metrics of each supplemental SAFMEDS procedure with the basic SAFMEDS procedure across corrects and incorrects for each participant. Figures represent the number of correct and incorrect (i.e., including pass responses) responses per 1-min timing across days. The duration of baseline was staggered across participants to create a discontinuous multiple-baseline across-participants design; however, this may not be evident due to the presentation of charts by participant. Results are depicted using SCCs to demonstrate real-time changes in day-to-day performance for each participant.

Fig. 1.

Results for Karly. Note. The top chart shows the raw data for the basic procedure using the Chinese deck. The bottom chart shows the raw data for the whole-deck procedure using the Russian deck

Fig. 2.

Results for Terri. Note. The top chart shows the raw data for the basic procedure using the Russian deck. The bottom chart shows the raw data for the whole-deck procedure using the Chinese deck

Fig. 3.

Results for Louise. Note. The top chart shows the raw data for the basic procedure using the Chinese deck. The bottom chart shows the raw data for the whole-deck procedure using the Russian deck

Fig. 4.

Results for Jackson. Note. The top chart shows the raw data for the basic procedure using the Russian deck. The bottom chart shows the raw data for the incremental whole-deck procedure using the Chinese deck

Fig. 5.

Results for Bailey. Note. The top chart shows the raw data for the basic procedure using the Chinese deck. The bottom chart shows the raw data for the incremental whole-deck procedure using the Russian deck. Only one session of the final intervention phase (i.e., all cards) was completed

Fig. 6.

Results for Joan. Note. The top chart shows the raw data for the basic procedure using the Russian deck. The bottom chart shows the raw data for the incremental whole-deck procedure using the Chinese deck

Fig. 7.

Results for Pam. Note. The top chart shows the raw data for the basic procedure using the Chinese deck. The bottom chart shows the raw data for the incremental whole-deck procedure using the Russian deck

Fig. 8.

Results for Kit. Note. The top chart shows the raw data for the basic procedure using the Russian deck. The bottom chart shows the raw data for the sprints procedure using the Chinese deck. Follow-up was not completed for Kit

Fig. 9.

Results for Diane. Note. The top chart shows the raw data for the basic procedure using the Chinese deck. The bottom chart shows the raw data for the sprints procedure using the Russian deck

Fig. 10.

Results for Myra. Note. The top chart shows the raw data for the basic procedure using the Chinese deck. The bottom chart shows the raw data for the sprints procedure using the Russian deck

Fig. 11.

Results for Kylie. Note. The top chart shows the raw data for the basic procedure using the Russian deck. The bottom chart shows the raw data for the sprints procedure using the Chinese deck

Fig. 12.

Results for Nathan. Note. The top chart shows the raw data for the basic procedure using the Russian deck. The bottom chart shows the raw data for the incremental sprints procedure using the Chinese deck

Fig. 13.

Results for Sarah. Note. The top chart shows the raw data for the basic procedure using the Chinese deck. The bottom chart shows the raw data for the incremental sprints procedure using the Russian deck

Fig. 14.

Results for Lacy. Note. The top chart shows the raw data for the basic procedure using the Chinese deck. The bottom chart shows the raw data for the incremental sprints procedure using the Russian deck

Table 3.

Level, Celeration, and Bounce Metrics of Correct and Incorrect Responses Across Participants in the Whole-Deck Supplemental SAFMEDS Procedure

Participant name	Condition	Language	Corrects			Incorrects
			Level	Celeration	Bounce	Level	Celeration	Bounce
Karly	Basic SAFMEDS	Chinese	2.52	x1.0	x4.2	17.8	÷1.01	x1.3
	Whole deck	Russian	7.35	x1.27	x2.7	11.71	÷1.11	x1.9
Terri	Basic SAFMEDS	Russian	1.27	x1.33	x4.3	23	x1.05	x1.6
	Whole deck	Chinese	14.91	x1.61	x3.6	6.59	÷1.42	x2.3
Louise	Basic SAFMEDS	Chinese	1.85	x1.33	x3.6	20.73	x1.05	x1.3
	Whole deck	Russian	15.62	x1.63	x2.4	10.54	÷1.1	x1.5

Table 4.

Level, Celeration, and Bounce Metrics of Correct and Incorrect Responses Across Participants in the Incremental Whole-Deck Supplemental SAFMEDS Procedure

Participant name	Condition	Language	Corrects			Incorrects
			Level	Celeration	Bounce	Level	Celeration	Bounce
Jackson	Basic SAFMEDS	Russian	0.60	x1.09	x2.3	10.76	÷1.03	x1.6
	Incremental whole deck	Chinese
	Subdeck 1		33.18	x3.68	x1.2	1.82	÷47.8	x1.2
	Subdeck 2		37.51	x2.14	x1.1	1.82	÷11.3	x2.5
	Subdeck 3		21.06	x3.12	x2.5	1.51	÷2.39	x8.1
	Subdeck 4		28.99	x1.06	x1.1	1.26	÷2.44	x1.7
	All decks		21.79	x1.07	x1.8	3.74	÷1.2	x3.0
Bailey	Basic SAFMEDS	Chinese	3.02	x1.09	x3.1	33.84	x1.02	x1.4
	Incremental whole deck	Russian
	Subdeck 1		14.59	x3.83	x1.3	6.34	÷3.0	x2.8
	Subdeck 2		18.2	x1.37	x1.6	7.63	÷1.4	x1.5
	Subdeck 3		29.95	x4.13	x1.6	2.88	÷46.8	x2.0
	Subdeck 4		36.65	x1.33	x1.0	1.15	÷11.3	x4.5
	All decksa		—	—	—	—	—	—
Joan	Basic SAFMEDS	Russian	1.16	x1.18	x3.3	32.93	x1.05	x1.4
	Incremental whole deck	Chinese
	Subdeck 1		31.77	x1.13	x1.4	1.0	x1.0	x1.0
	Subdeck 2		32.17	x3.8	x1.3	0.63	x1.0	x2.4
	Subdeck 3		34.55	x1.74	x1.5	0.84	÷1.45	x3.7
	Subdeck 4		33.97	x1.16	x1.3	1.0	÷1.83	x2.0
	All decks		27.31	x1.28	x1.8	2.57	÷1.63	x2.8
Pam	Basic SAFMEDS	Chinese	2.79	x1.17	x4.5	15.12	x1.03	x1.6
	Incremental whole deck	Russian
	Subdeck 1		23.33	x2.62	x1.4	0.71	÷2.83	x1.8
	Subdeck 2		26.72	x10.2	x1.6	0.79	÷128	x2.4
	Subdeck 3		28.31	x1.18	x1.2	1.26	÷1.41	x1.1
	Subdeck 4		31.66	x1.12	x1.0	0.5	÷1.0	x1.0
	All decks		20.49	x1.52	x1.3	2.74	÷6.16	x2.9

^a Levels cannot be computed due to only one session with all subdecks combined

Table 5.

Level, Celeration, and Bounce Metrics of Correct and Incorrect Responses Across Participants in the Sprints Supplemental SAFMEDS Procedure

Participant name	Condition	Language	Incorrects			Corrects
			Level	Celeration	Bounce	Level	Celeration	Bounce
Kit	Basic SAFMEDS	Russian	1.03	x1.17	x3.3	28.14	x1.03	x1.3
	Sprints	Chinese	9.37	x1.34	x3.2	17.13	÷1.08	x1.9
Diane	Basic SAFMEDS	Chinese	2.46	x1.35	x4.0	31.32	x1.04	x1.3
	Sprints	Russian	7.90	x1.44	x3.5	31.02	x1.01	x1.4
Myra	Basic SAFMEDS	Chinese	1.52	x1.12	x4.7	31.35	x1.04	x1.4
	Sprints	Russian	1.72	x1.27	x3.7	27.55	x1.03	x1.5
Kylie	Basic SAFMEDS	Russian	2.29	x1.31	x3.1	39.45	x1.08	x1.2
	Sprints	Chinese	17.77	x1.48	x3.5	32.52	x1.05	x1.2

Table 6.

Level, Celeration, and Bounce Metrics of Correct and Incorrect Responses Across Participants in the Incremental Sprints Supplemental SAFMEDS Procedure

Participant name	Condition	Language	Corrects			Incorrects
			Level	Celeration	Bounce	Level	Celeration	Bounce
Nathan	Basic SAFMEDS	Russian	1.38	x1.14	x3.8	23.74	÷1.05	x1.8
	Incremental sprints	Chinese
	Subdeck 1		8.55	x3.08	x3.9	13.74	÷1.8	x1.9
	Subdeck 2		11.58	x89.8	x2.5	5.73	÷320	x3.2
	Subdeck 3		13.12	x2.82	x2.5	7.08	÷2.51	x3.2
	Subdeck 4		15.58	÷1.09	x2.5	6.32	÷1.33	x3.0
	All decks		14.69	x1.61	x1.7	7.49	x1.02	x3.5
Sarah	Basic SAFMEDS	Chinese	3.01	x1.11	x5.3	20.45	x1.04	x1.6
	Incremental sprints	Russian
	Subdeck 1		15.08	x15	x15	3.98	÷907	x3.8
	Subdeck 2		8.07	x10.6	x10.6	6.79	÷9.13	x4.5
	Subdeck 3		18.54	x2.52	x2.52	4.03	÷8.36	x6.2
	Subdeck 4		15.13	x5.25	x5.25	6.31	÷21.3	x4.7
	All decks		14.4	x1.28	x1.28	12.68	÷1.0	x2.2
Lacy	Basic SAFMEDS	Chinese	4.74	x1.1	x3.3	29.42	x1.01	x1.01
	Incremental sprints	Russian
	Subdeck 1		12.32	x3.91	x1.9	17.61	÷1.84	x1.6
	Subdeck 2		18.24	x2.26	x1.4	17.27	÷2.22	x1.4
	Subdeck 3		16.18	x1.46	x1.7	18.96	÷1.36	x1.3
	Subdeck 4		16.92	x2.17	x1.7	16.18	÷2.13	x1.5
	All decks		10.14	x1.05	x1.6	22.17	x1.09	x1.3

Whole Deck

Level

Across the three participants in the whole-deck supplemental SAFMEDS procedure, participants demonstrated an average of 12.62 (range 7.35–15.62) correct responses per day throughout the duration of this procedure (see Table 3). Incorrect responses in the whole-deck procedure occurred at an average rate of 9.61 (range 6.59–11.71) per day across all sessions. In comparison, in the basic SAFMEDS procedure, these same participants yielded an average rate of 1.88 (range 1.27–2.53) correct responses per day. The basic SAFMEDS procedure showed an average of 20.51 (range 17.8–23) incorrect responses per day. In general, correct response rates were low in the basic SAFMEDS conditions, especially when compared to incorrect responses.

Celeration

Celerations for the number of correct responses in the whole-deck procedure ranged from ×1.27 (27% acceleration in daily corrects over time) to ×1.63 (63% acceleration in daily corrects over time), with an average of ×1.5. Celerations for the number of incorrect responses in the whole-deck procedure ranged from ÷1.1 (10% decay in incorrects) to ÷1.4 (40% decay in incorrects), with an average of ÷1.21. In comparison, celerations for the number of correct responses in the basic SAFMEDS procedure ranged from ×1.0 (no daily increase in corrects over time) to ×1.33 (33% acceleration in daily corrects over time), with an average of ×1.22. Celerations for the number of incorrect responses in the basic SAFMEDS procedure ranged from ÷1.0 (no decay in incorrects over time) to ×1.05 (5% acceleration in daily incorrects over time), with an average of ×1.03.

Bounce

The bounce of correct responses in the whole-deck procedure ranged from ×2.4 (140% acceleration in the variability of corrects over time) to ×3.6 (260% acceleration in the variability of corrects over time), with an average of ×2.9. The bounce of incorrect responses in the whole-deck procedure ranged from ×1.5 (50% acceleration in the variability of incorrects over time) to ×2.3 (130% acceleration in the variability of incorrects over time), with an average of ×1.9. In comparison, the bounce of correct responses in the basic SAFMEDS procedure ranged from ×3.6 (260% acceleration in the variability of corrects over time) to ×4.3 (330% acceleration in the variability of corrects over time), with an average of ×4.03. The bounce of incorrect responses in the basic SAFMEDS procedure ranged from ×1.3 (30% acceleration in the variability of incorrects over time) to ×1.6 (60% acceleration in the variability of incorrects over time), with an average of ×1.4.

Incremental Whole Deck

Level

Across the four participants in the incremental whole-deck supplemental SAFMEDS procedure, participants demonstrated an average of 28.91 (range 14.59–37.51) correct responses per day across subdecks (see Table 4). Participants demonstrated an average of 20.20 (range 12–27.31) correct responses per day using the entire deck (Bailey is not included in the all-deck calculations due to limited data). Incorrect responses across subdecks occurred at an average rate of 1.94 (range 0.50–7.63) per day across all sessions. Incorrect responses using the entire deck occurred at an average rate of 5.76 (range 2.57–14) per day across all sessions. In comparison, in the basic SAFMEDS procedure, these same participants yielded an average rate of 1.89 (range 0.60–3.02) correct responses per day. The basic SAFMEDS procedure yielded an average of 23.16 (range 10.76–33.84) incorrect responses per day. In general, correct response rates were low in the basic SAFMEDS conditions, especially when compared to incorrect responses.

Celeration

Celerations for the number of correct responses in the incremental whole-deck procedure, across all subdecks, ranged from ×1.06 (6% acceleration in daily corrects over time) to ×10.2 (920% acceleration in daily corrects over time), with an average of ×2.72. Celerations for the number of correct responses using the entire deck ranged from ×1.07 (7% acceleration in daily corrects over time) to ×1.52 (52% acceleration in daily corrects over time), with an average of ×1.29. Celerations for the number of incorrect responses, across all subdecks, ranged from ×1.0 (no change in incorrects) to ÷128 (1,280% decay in incorrects), with an average of ÷16.56. Celerations for the number of incorrect responses using the entire deck ranged from ÷1.2 (20% decay in daily corrects over time) to ÷6.16 (516% decay in daily incorrects over time), with an average of ×2.99. In comparison, celerations for the number of correct responses in the basic SAFMEDS procedure ranged from ×1.09 (9% daily increase in corrects over time) to ×1.18 (18% acceleration in daily corrects over time), with an average of ×1.13. Celerations for the number of incorrect responses in the basic SAFMEDS procedure ranged from ×1.05 (5% acceleration of incorrects over time) to ÷1.03 (3% decay in daily incorrects over time), with an average of ×1.03.

Bounce

The bounce of correct responses in the incremental whole-deck procedure, across all subdecks, ranged from ×1.0 (no daily change in the variability of corrects over time) to ×2.5 (150% acceleration in the variability of corrects over time), with an average of ×1.3. The bounce of correct responses using the entire deck ranged from ×1.3 (30% acceleration in the variability of daily corrects over time) to ×1.8 (80% acceleration in the variability of daily corrects over time), with an average of ×1.63. The bounce of incorrect responses in the incremental whole-deck procedure, across all subdecks, ranged from ×1.0 (no daily change in the variability of corrects over time) to ×8.1 (710% acceleration in the variability of incorrects over time), with an average of ×2.5. The bounce of incorrect responses using the entire deck ranged from ×2.8 (180% acceleration in the variability of daily incorrects over time) to ×3.0 (200% acceleration in the variability of daily incorrects over time), with an average of ×2.8. In comparison, the bounce of correct responses in the basic SAFMEDS procedure ranged from ×3.6 (260% acceleration in the variability of corrects over time) to ×4.3 (330% acceleration in the variability of corrects over time), with an average of ×4.03. The bounce of incorrect responses in the basic SAFMEDS procedure ranged from ×1.3 (30% acceleration in the variability of incorrects over time) to ×1.6 (60% acceleration in the variability of incorrects over time), with an average of ×1.4.

Sprints

Level

Across the four participants in the sprints supplemental SAFMEDS procedure, participants demonstrated an average of 9.19 (range 1.72–17.77) correct responses per day throughout the duration of this procedure (see Table 5). Incorrect responses in the sprints procedure occurred at an average rate of 27.05 (range 17.13–32.52) per day across all sessions. In comparison, in the basic SAFMEDS procedure, these same participants yielded an average rate of 1.82 (range 1.03–2.46) correct responses per day. The basic SAFMEDS procedure showed an average of 32.56 (range 28.14–39.45) incorrect responses per day. In general, correct response rates were low in the basic SAFMEDS conditions, especially when compared to incorrect responses.

Celeration

Celerations for the number of correct responses in the sprint procedure ranged from ×1.27 (27% acceleration in daily corrects over time) to ×1.48 (48% acceleration in daily corrects over time), with an average of ×1.38. Celerations for the number of incorrect responses in the sprint procedure ranged from ×1.05 (5% acceleration in incorrects) to ÷1.08 (8% decay in incorrects), with an average of ×1.04. In comparison, celerations for the number of correct responses in the basic SAFMEDS procedure ranged from ×1.12 (12% acceleration in daily corrects over time) to ×1.35 (35% acceleration in daily corrects over time), with an average of ×1.23. Celerations for the number of incorrect responses in the basic SAFMEDS procedure ranged from ×1.03 (3% acceleration in incorrects over time) to ×1.08 (8% acceleration in daily incorrects over time), with an average of ×1.04.

Bounce

The bounce of correct responses in the sprint procedure ranged from ×3.2 (220% acceleration in the variability of corrects over time) to ×3.7 (270% acceleration in the variability of corrects over time), with an average of ×3.47. The bounce of incorrect responses in the sprint procedure ranged from ×1.2 (20% acceleration in the variability of incorrects over time) to ×1.9 (90% acceleration in the variability of incorrects over time), with an average of ×1.5. In comparison, the bounce of correct responses in the basic SAFMEDS procedure ranged from ×3.1 (210% acceleration in the variability of corrects over time) to ×4.7 (370% acceleration in the variability of corrects over time), with an average of ×3.77. The bounce of incorrect responses in the basic SAFMEDS procedure ranged from ×1.2 (20% acceleration in the variability of incorrects over time) to ×1.4 (40% acceleration in the variability of incorrects over time), with an average of ×1.3.

Incremental Sprints

Level

Across the three participants in the incremental sprints supplemental SAFMEDS procedure, participants demonstrated an average of 14.11 (range 8.07–18.54) correct responses per day across subdecks (see Table 6). Participants demonstrated an average of 13.07 (range 10.14–14.69) correct responses per day using the entire deck. Incorrect responses across subdecks occurred at an average rate of 10.33 (range 3.98–18.96) per day across subdecks. Incorrect responses using the entire deck occurred at an average rate of 14.11 (range 7.49–22.17) per day across all sessions. In comparison, in the basic SAFMEDS procedure, these same participants yielded an average rate of 3.04 (range 1.38–4.74) correct responses per day. The basic SAFMEDS procedure showed an average of 24.54 (range 20.45–29.42) incorrect responses per day. In general, correct response rates were low in the basic SAFMEDS conditions, especially when compared to incorrect responses.

Celeration

Celerations for the number of correct responses in the incremental sprints procedure, across all subdecks, ranged from ÷1.09 (9% decay in daily corrects over time) to ×89.8 (8,880% acceleration in daily corrects over time), with an average of ×11.48. Celerations for the number of correct responses using the entire deck ranged from ×1.05 (5% acceleration in daily corrects over time) to ×1.61 (61% acceleration in daily corrects over time), with an average of ×1.31. Celerations for the number of incorrect responses, across all subdecks, ranged from ÷1.33 (33% decay in incorrects) to ÷907 (9,060% decay in incorrects), with an average of ÷106.58. Celerations for the number of incorrect responses using the entire deck ranged from ÷1.09 (9% decay in daily corrects over time) to ×1.02 (2% acceleration in daily incorrects over time), with an average of ×0.37. In comparison, celerations for the number of correct responses in the basic SAFMEDS procedure ranged from ×1.1 (10% daily increase in corrects over time) to ×1.14 (40% acceleration in daily corrects over time), with an average of ×1.12. Celerations for the number of incorrect responses in the basic SAFMEDS procedure ranged from ÷1.05 (5% decay of incorrects over time) to ×1.04 (4% acceleration in daily incorrects over time), with an average of ×1.0.

Bounce

The bounce of correct responses in the incremental sprints procedure, across all subdecks, ranged from ×1.4 (40% acceleration in the variability of corrects over time) to ×3.9 (290% acceleration in the variability of corrects over time), with an average of ×2.11. The bounce of correct responses using the entire deck ranged from ×1.6 (60% acceleration in the variability of daily corrects over time) to ×2.4 (140% acceleration in the variability of daily corrects over time), with an average of ×1.9. The bounce of incorrect responses in the incremental whole-deck procedure, across all subdecks, ranged from ×1.3 (30% acceleration in the variability of incorrects over time) to ×6.2 (520% acceleration in the variability of incorrects over time), with an average of ×3.03. The bounce of incorrect responses using the entire deck ranged from ×1.3 (30% acceleration in the variability of daily incorrects over time) to ×3.5 (250% acceleration in the variability of daily incorrects over time), with an average of ×2.33.

Discussion

SAFMEDS is an assessment and instructional strategy based on PT principles (Eshleman, 2000a; Graf & Auman, 2005). The name suggests a standard application for use, but a literature review suggests vague and varied applications of the procedure. The purpose of this study was to define a basic SAFMEDS procedure, along with four variations of SAFMEDS procedures, and to determine the effects of each procedural variation on correct and incorrect responses. Taken together, the data obtained in this study indicate the basic SAFMEDS procedure, as defined previously, produces low levels of correct responses and higher levels of incorrect responses. The supplemental procedures, in general, produced higher levels of correct responses and lower levels of incorrect responses. These general findings are somewhat difficult to interpret in relation to previous findings because it is not clear if these procedures are the same as or different from those in previous studies. As noted previously, many earlier studies of SAFMEDS have vague definitions of the procedures (see Quigley et al., 2017) or include what we defined as variations of SAFMEDS procedures (e.g., additional timings).

It should be noted that no additional instruction was provided on the Russian or Chinese characters other than exposure to the SAFMEDS cards. One might criticize the study for this lack of instruction. However, SAFMEDS is described as both an assessment and an instructional procedure (e.g., Eshleman, 2000a; Lindsley, 1996). The previously published literature is unclear as to whether SAFMEDS procedures were used strictly for assessment of learning or as part of the overall instruction. The relation between SAFMEDS and ongoing instruction has not been made clear in the literature. It is unclear whether these procedures would have been more effective if they had been used in the context of ongoing, direct instruction. Future research should evaluate the effects of the SAFMEDS procedures in conjunction with other instructional opportunities (e.g., lecture, book reading) to determine how these additional, non-fluency-based instructional methods impact performance with multiple SAFMEDS procedures.

Interestingly, in most cases, the variations of the SAFMEDS procedure produced increases in correct responding and decreases in incorrect responding, suggesting that they were sufficient to produce at least some learning in the absence of other, ongoing, explicit, direct instruction. This is in stark contrast to the results for the basic SAFMEDS procedure. The whole-deck, incremental whole-deck, sprints, and incremental sprints procedures all produced an improvement in correct responding for all participants, except one. The change in level when each of the supplemental procedures was introduced was large, especially when compared to performance during the basic SAFMEDS procedure. Of the variations of SAFMEDS procedures, the incremental whole-deck and incremental sprints procedures appeared to have the most impact on performance. This finding is interesting given Lindsley’s (1996) rationale for his acronym choice. Specifically, “all,” “fast,” and “minute” were chosen to indicate limitations of previous flash card methods. “All” and “fast” indicated a need to review “all” material at one time rather than smaller chunks of material (e.g., learning all 60 cards instead of 15 at a time). His hypothesis was that doing so would prevent slow, accurate responding with the later need to develop fluency. Results of the current study suggest that reviewing the cards in smaller units (i.e., 15 at a time) in a relatively slow manner (i.e., studying them for a 10-min period) with subsequent 1-min timings may lead to quick acquisition and steep increases in fluency. This conclusion is based upon anecdotal data and should be viewed cautiously, however.

Participants were not required to use the full 10 min allotted for study. Often, participants did not use the full 10 min of study and, furthermore, their study time decreased with each subsequent day. Additionally, participants studied the cards in a “fast” manner during the review period. Specifically, participants often studied the cards in a similar format to the timing procedure—that is, quickly going through the deck, saying cards quickly, and checking their answers. Finally, although we did not program for it, the participants often sued error self-correction procedures that had been implemented in other SAFMEDS studies (e.g., Hughes, Beverly, & Whitehead, 2007). For example, the participants often set aside the cards on which they made errors and then conducted additional practice on those cards during their study time. This anecdotal information suggests that perhaps participants were using a practice procedure that exemplified the “fast” and “minute” components of SAFMEDS.

This study has its limitations and, thus, results should be interpreted with caution. First, participants did not complete daily timings because they did not practice on weekends. It is reasonable to assume that participants would have had a higher number of corrects and fewer errors if they had had additional learning trials that would have been afforded through true daily practice. We did not allow participants to take home their decks and do timings on the weekends because we could not be there to conduct those timing sessions. Furthermore, allowing participants to take decks home could have added confounds due to additional practice the participants may have done outside of research sessions.

Another limitation was that participants were not allowed to continue practicing with their decks until rates of performance stabilized. This limitation is most important for the participants who experienced the incremental procedures. Best practice probably would have been for participants to practice each subdeck until a specific mastery criterion was met. In this study, we provided new decks based on the passage of time, not performance. If participants had continued practicing their subdecks until high and stable rates of correct responding were observed, the decrease in correct responses when reintroducing the entire deck might have been less likely. Another possible limitation is that many participants in this study reported having multiple years of experience with a second language (though not these languages), were high-performing college students (i.e., most had GPAs of 3.0 and above), and had used SAFMEDS procedures in their coursework prior to enrolling in the study. It is unknown how these factors affected performance. Future studies may want to include participants who are more naive to SAFMEDS, are lower performing, and/or have less experience with skills related to those being taught.

Another potential limitation of the study is that we did not counterbalance the order in which participants completed timings with the different decks across phases. That is, we did not control for whether participants completed their Chinese deck or their Russian deck first each day, and we did not control for this across the SAFMEDS variations. If participants repeatedly completed the Chinese deck timing after the Russian deck timing, performance could have been higher on the Chinese deck because the Russian deck served as a “warm-up” (see Donahoe & Palmer, 2004, for a discussion of priming). If this phenomenon were observed, it would be especially problematic after the variations of SAFMEDS procedures were introduced because the phenomenon might have partially accounted for the improved results for the variation strategies. To assess this possible limitation, we reviewed the sequence of timings for each participant across each phase. Order was classified as balanced (i.e., each deck was practiced first between 40% and 60% of the time) or unbalanced (i.e., either deck was practiced first greater than 60% of opportunities). When both decks were exposed to the basic SAFMEDS procedure, 10 of 14 participants (i.e., Karly, Terry, Louise, Diane, Kit, Jackson, Nathan, Lacy, Joan, and Myra) had a balanced order. The performance of these 10 participants was compared to the remaining 4 participants, whose practice was more unbalanced (i.e., one deck more routinely followed another). We observed similar performance across all individuals, indicating no warm-up effects. When the SAFMEDS variations were introduced, 9 of the 14 participants (i.e., Louise, Terri, Dianne, Jackson, Bailey, Nathan, Sarah, and Kylie) had an unbalanced order. In these cases, the deck being practiced with the basic procedure was typically practiced first, followed by the deck being practiced with the SAFMEDS variation. However, comparisons of levels and trends across these participants as compared to those whose practice was more balanced indicated similar performance. Again, this suggested that no warm-up effect was observed. There was one exception, Myra (i.e., crossover never occurred after the introduction of the supplemental procedure). Based on this information, it is believed that the order effects are minimal, if present at all.

Fluency-based instruction has proposed specific outcomes. These outcomes are retention, endurance, application, and performance standards (REAPS; Merbitz et al., 2004). Although previous studies have attempted to evaluate the REAPS outcomes of SAFMEDS (e.g., Kim et al., 2001), this study did not. The current study indicates celeration differences across procedures, but future studies should identify whether procedures produce different REAPS outcomes. These learning outcomes are the core feature of using fluency-based instruction methods.

In summary, the basic SAFMEDS procedure, as we defined it, did not appear to facilitate fluency development with Chinese characters and Russian words in the absence of other instruction with college students. However, variations of the basic SAFMEDS procedure all produced relatively robust increases in correct responses per 1-min timing, as well as decreases in incorrect responses. These findings suggest that there is still much to learn about the utility of SAFMEDS and the effects of variations of this procedure on the accuracy and fluency of student responding. Furthermore, the refinement of the SAFMEDS procedure is only the beginning of needed SAFMEDS research (Quigley et al., 2017). Further replication of these results is warranted before firm conclusions can be drawn. Future research should focus on further analyses of these and other supplemental procedures to determine which procedures lead to quicker fluency development and under what conditions these procedures are most effective (i.e., in the presence or absence of other explicit instruction). Replications of this work with different materials, content areas, populations, and learning channels are also warranted. Beyond these areas of future research, there is also a need for researchers to parse out the roles of practice effects and reinforcement rates, if any, with SAFMEDS. Doughty, Chase, and O’Shields (2004) provided an in-depth discussion of the potential interactions between practice effects, rate of reinforcement, and fluency-building exercises that have not been thoroughly explored to date. There is still much work to be done to understand SAFMEDS and all the variations of the general procedure.

Declarations

Ethical approval

All procedures performed in this study involving human participants were in accordance with the ethical standards of Western Michigan University (HSIRB Project No. 13-07-27) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Conflict of interest

The authors declare that they have no conflict of interest.

Informed consent

Informed consent was obtained from all individual participants included in the study.