Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Apr 1.
Published in final edited form as: J Contextual Behav Sci. 2014 Apr 1;3(2):148–153. doi: 10.1016/j.jcbs.2014.03.003

The promise of mobile technologies and single case designs for the study of individuals in their natural environment

Roger Vilardaga 1, Jonathan Bricker 2, Michael McDonell 3
PMCID: PMC4059557  NIHMSID: NIHMS587394  PMID: 24949285

Abstract

Mobile technologies are growing rapidly around the world to broad demographics of society. These technologies hold great promise for their integration with Single Case Designs (SCDs) and the study of individuals in their natural environment. This paper discusses the theoretical, methodological and analytic implications of these tools for the advancement of the contextual behavioral etiology of behavioral disorders, and their remediation. We hope this paper will highlight the scientific advantages of combining mobile technologies and SCDs and encourage their adoption among CBS scientists.

Keywords: mobile technologies, single case designs, contextual behavioral science

Psychology is rooted in the study of individuals. B.F. Skinner, whose work is very familiar to behavioral scientists, derived the principles of positive and negative reinforcement (1938) from the study of individuals from two single species (rattus norvegicus and columba livia). The development of applied interventions in areas such as education (e.g., Layng et al., 2004), psychopathology (e.g., Barlow et al., 1984; Kazdin, 1977; Wolpe, 1968), addiction (e.g., McDonell et al., 2013; Petry et al., 2006), or developmental disabilities (e.g., Sisson et al., 1993), draw directly from the principles derived from such individual analysis. Further, many experiments in other fields of psychology relied on single individuals to validate scientific hypotheses and theories. For example, Ebbinghaus (1913), discovered new insights into memory and cognition through experiments he conducted primarily on himself.

Creating models to explain individual variation has the advantage of generating knowledge that is directly transportable to clinical practice. In areas such as medicine, these designs have become increasingly popular (e.g., Backman & Harris, 1999; Shamseer et al., 2012; Kravitz et al., 2008), such that an extension of the CONSORT guidelines are currently being developed in the medical field (e.g., CENT guidelines: Shamseer et al., 2012).

This revival of SCD methods has its origin in ethical, financial and methodological arguments. Authors such as Lillie et al. (2011) argue that these methods are central for the development of personalized medicine. In their view, despite the fact that a central axiom in medicine is that of providing treatment to the individual patient, it is surprising that SCD methods remained exclusive to areas such as education, a point also made by Kravitz et al. (2008). Riley et al. (2013) discussed the benefits of SCDs for pilot trials, given their flexibility, cost-efficiency and ability to quickly generate data. In the authors’ view, SCDs play a central role in what they call rapid, responsive and relevant research, since these methods can reduce the costs of exploring the feasibility of new interventions and accelerate scientific innovation. Finally, the use of randomization tests (or permutation tests), allows the statistical analysis of SCDs without relying on the assumptions of most frequentist statistical techniques (i.e., random sampling, independence, normality). As this special issue shows, randomization tests have a long history (Dugard), are quickly evolving, and can be run with freely available software packages (Heyvaert).

Mobile technologies: a 21st century tool for CBS scientists

Throughout history, scientists have taken advantage of any tools available to interact with their subject of study. In astronomy, the telescope substituted for the bare eye in the observation of stars and planets, leading to a greater precision in the definition of astronomical terms. Likewise, the microscope was a critical source of innovation in biology. Computer technology had a similar effect across scientific disciplines. The software and hardware revolution of the 1950’s had a great impact in science, both in terms of computation power and of speed of transfer of information (Fertig, 1985). However, aside of the availability of word processors, data analysis and communication software (e.g., internet, email), this revolution did not have a profound impact on the basic tools for studying the individual in its natural environment. Mobile technologies are changing this.

Mobile technologies have been defined as “wireless devices and sensors (including mobile phones) that are intended to be worn, carried, or accessed by the person during normal daily activities” (Kumar et al., 2013, p. 228). This definition includes a broad range of devices, from cell-phones and portable digital assistants (PDAs) to state of the art smartphones. It is also broad enough to include devices that can be used for a variety of purposes, such as help people monitor behavior, provide therapeutic content or respond to time sensitive self-report data. For example, smartphones are enabling behavioral scientists to model, instigate and reinforce individual behavior change in “real-world” settings, and the active and passive assessment of outcomes. The sophistication and capabilities of these new mobile technologies is such that they are “a dream come true” from a contextual behavioral science standpoint. Further, mobile devices are already carried by 83% of individuals in the United States, and smartphones are used by 34% of the population (Smith, 2012). The popularity of mobile technology is reflected by the fact that there are currently 12,000 health related apps in the market (Mobihealthnews, n.d.), and by 2016 it is estimated that there will be 146 million downloads of mobile health apps (iHealthBeat, n.d.).

Despite the fact that very few mobile health apps have been empirically tested (D?glise et al., 2012; Chen et al., 2012), these technologies have created great excitement in the field (e.g., Miller, 2012), including major research funding institutions (e.g., Nilsen et al., 2012; Kumar et al., 2013). Mobile technologies allow for the feasible implementation of SCDs in an individual’s natural environment, providing real time assessment and testing of contextual behavioral hypothesis and theories.

Currently, a number of papers have discussed the use of mobile technologies in the social and behavioral sciences (e.g., Aguilera & Muench, 2012; Morris & Aguilera, 2012; Miller, 2012; Nilsen et al., 2012; Kumar et al., 2013; Dallery et al., 2013; Chen et al., 2012), but none has elaborated on their value from a CBS perspective. Furthermore, some studies have discussed the implications of SCDs for the rapid testing of technology-based interventions (e.g., Dallery et al., 2013), but not the scientific implications of mobile technologies for the implementation of SCDs. Therefore, the goals of this paper are to discuss the theoretical, methodological and analytic implications of combining mobile technologies and SCDs for the study of individuals in their natural environment.

Theoretical: Examining the impact of verbal influence in natural contexts

Mobile technologies are a means to an end, rather than an end themselves. From a scientific perspective these technologies are an empty shell if not at the service of testing specific hypotheses or theories. One of the most innovative behavioral theories that came out of the contextual behavioral tradition is Relational Frame Theory (RFT; Hayes et al., 2001). RFT was built upon Skinner’s classic work on verbal behavior (1957) and Kantor’s interbehaviorism (1963). This scientific approach to language and cognition targets the ecological determinants of individual behavior, and studies how over time, the verbal context experienced by a single individual and its consequences, form a relational repertoire (a.k.a. “relational network”). RFT reignited the naturalistic and non-reductionist study of areas such as self-awareness (e.g., Dymond & Barnes, 1997), cognition (e.g., Golijani-Moghaddam et al., in press), values-driven behaviors (e.g., Hooper et al., 2012) and metaphor (e.g., Stewart et al., 2004), and has informed the development of psychotherapy models such as Acceptance and Commitment Therapy (Hayes et al., 2011). A more detailed description of RFT and its historical and current status can be found elsewhere (e.g., Hughes et al., 2012; Dymond, 2013).

RFT attempts to understand how specific verbal cues (e.g., naming, comparing, contrasting) put forward by individuals in the social environment have an impact on individual’s experiences (thinking, feeling, wanting) and overt behavior. These verbal cues have been traditionally experienced through different forms of writing, but more recently through other media, such as text messaging or websites. One example of verbal event is the common experience of receiving and responding to a text message. A specific verbal cue (e.g., “how are you doing”), in combination with a specific physical context (e.g., a restaurant, an empty classroom or an open space), has the function of prompting an emotional, relational or overt behavioral response (e.g., feeling connected, responding back, ignoring it, providing another verbal cue, etc.). These antecedents, behaviors and consequences will shape the overall verbal context of this individual and influence future interactions.

In that way, mobile technologies have the potential to serve as a platform to study the impact of verbal cues on individuals’ behavior in their natural environment. Furthermore, the combination of mobile technologies and measurement procedures such as ecological momentary assessments (e.g., Wenze & Miller, 2010) and a variety of mobile sensors (Ali et al., 2012; Plarre et al., 2011; Preece et al., 2009) have enabled the gathering of real time environmental and observational data, a possibility that haunted the field almost from its inception (Dougher & Dougher, 2000).

These devices provide an excellent opportunity to RFT and clinical CBS researchers. By deliverately manipulating a variety of daily verbal cues, researchers can extend the reach of their experimental verbal manipulations to real world settings. These verbal cues can be designed to enhance more flexible repertoires such as deictic relations (Vilardaga et al., 2012), motivative augmentals (Dahl et al., 2009), or combinations of relational cues (e.g., in the form of metaphors). These mobile technologies, when combined with an RFT analysis of language can also be used as stand-alone interventions or to enhance existing face-to-face interventions. In addition, mobile technologies can enable researchers to study the contextual antecedents and consequences of specific verbal cues and potentially design functionally appropriate schedules of verbal prompting that are unique to a particular individual. For example, machine learning algorithms can be programmed to track specific sequences of events, behaviors and consequences to readjust the ratio of motivative augmentals provided to a specific individual for a specific target behavior. Furthermore, if deictic relations are at the core of rigid forms of sense of self and a range of clinical phenomena, as argued by ACT and RFT researchers (e.g., Barnes-Holmes et al., 2000; Vilardaga et al., 2012), then it would be expected that a mobile texting intervention in which machine learning algorithms are used to modulate the delivery of deictic contextual cues could have the effect of increasing individual’s levels of psychological flexibility. Machine learning algorithms have already been used to improve behavioral interventions (e.g., Burns et al., 2011), although to our awareness, not using RFT.

Methodological: Increasing the precision, scope and depth of contextual behavioral research

Precision, scope and depth are important qualities of scientific theories (Biglan & Hayes, 1996). Precision is attained by having a limited number of concepts referring to a given phenomenon. This requires tools that can bridge the concept and the phenomena closer to each other. Scope is achieved when certain scientific principles are applicable to a broad range of phenomena, and depth when those principles cohere across levels of analysis (e.g., biological or social).

Current contextual behavioral tools have limited capacity to evaluate the precision, scope and depth of our theoretical models of intervention. The lack of instruments to directly observe behaviors of interest, such as levels of activation, social interactions, or moment to moment mood symptoms, is typically approached by designing laboratory and analog studies (e.g., Cooke, 1966). These studies test “analogs” of a clinical intervention through tightly controlled procedures. However, analog studies are difficult to generalize to real-world settings (Kazdin, 1978), and this research approach cannot fully substitute for a precise evaluation of specific clinical problems in a “real-world setting,” or the scope and depth of our strategies to remediate them.

Given that precision, scope and depth are important features of CBS (Hayes et al., 1993, 2012), clinical contextual behavioral science’s growth and potential impact will be limited as long as we continue to rely on tools that do not allow for the direct observation of the contextual determinants of individual’s behavior in their natural environment (Vilardaga et al., 2009).

Mobile technologies offer the opportunity to measure behaviors with greater precision. Some of these behaviors can be self-reported (e.g., a specific mood item), while others automatically tracked (e.g., GPS location). Capturing data in an individual’s natural context allows an examination of the specific antecedents and consequences of certain behaviors and this is an important requisite for the behavioral definition of constructs with higher precision. Further, the capacity of mobile technologies to circumvent the retrospective bias that comes along with most retrospective global self-reports (e.g., Sato & Kawahara, 2011; Ben-Zeev et al., 2012) cannot be minimized. Mobile technologies also enable researchers to experimentally examine the scope of specific interventions, as these technologies can illuminate to what degree certain interventions (e.g., motivative augmental cues) generalize across behaviors, emotional responses and physical locations. For example, clinical experience shows that patients often use the psychological strategies they learn (e.g., smoking cessation strategies) to target other problem behaviors (e.g., excessive eating), but there might be situations in which these skills are not warranted. This process of generalization is often not evaluated due to the difficulties of accessing individual’s natural environment. Thus mobile technologies can help researchers evaluate the limits and scope of these principles. Finally, mobile technologies can be used to assess the degree to which certain principles of change have depth, cohering across a variety of levels of analysis. An evaluation of the depth of scientific theories facilitates important communication among disciplines and cross-fertilization of ideas. For example, the impact of certain contextual behavioral interventions on mood ratings might extend to an individual’s heart rate variability, demonstrating its health benefits. Heart rate variability can be measured using mobile sensor technology. Likewise, contextual behavioral interventions at the level of an individual might have an impact on the larger social environment (e.g., social networks) and facilitate pro-social behavior or cooperation.

Gordon Paul’s statement, “what treatment, by whom, is most effective for this individual with that specific problem under which set of circumstances, and how does it come about” (1969, p.44), is difficult to answer with the measurement and intervention tools currently available (e.g., global self-report measures, face-to-face interventions). The synergy between mobile technologies and CBS research has the potential to improve its methodological rigor and help advance its underlying theory and models of intervention.

Analytic: Empowering visual and statistical analysis of SCDs

The advantages of mobile technologies do not end with the contextual assessment and delivery of verbal interventions in their natural environment, and a better examination of their precision, scope and depth. Mobile technologies also have analytic implications for the interpretation of SCDs.

From Ecological Momentary Assessments to Ecological Momentary Experiments

The use of real-time data collection tools such as Ecological Momentary Assessments (EMAs) and the Experience Sampling Method (ESM), have been present in the field for several decades now (e.g., Stone & Shiffman, 1994; Csikzentmihalyi et al., 1977). Although both tools have their origin in different philosophical frameworks (e.g., ESM was inspired by phenomenology; Hektner et al., 2007), both share the approach of measuring psychological and contextual variables in order to examine their association with a variety of outcomes.1 This approach, highly consistent with the CBS framework, has contributed to the environmental and naturalistic analysis of psychological phenomena. The repeated measurement of psychological and contextual events improves the ecological validity and measurement reliability of the data collected (Hektner et al., 2007). Unfortunately, the majority of these studies have used observational designs in which there is no experimental manipulation of independent variables (e.g., Csikszentmihalyi & Larson, 1987; Hektner et al., 2007), limiting their utility and conclusions.

EMAs gradually merged into earlier forms of mobile technologies, such as pagers, wristwatches, portable digital assistants and cell phones. However, the advent of smartphones with wireless and programming capability enabled the use of these devices as proactive intervention tools, not just for assessment. Since smartphones can be used to deliver a variety of verbal cues, prompts and signals, EMAs have the potential to evolve into what we call ecological momentary experiments: studies in which specific intervention components are deliberately manipulated to observe an effect. Furthermore, because SCDs are less logistically complex than randomized controlled trials, and can be flexibly combined with other SCD experiments (see Heyvaert’s tutorial on meta-analysis of SCDs in this special issue), their potential to rapidly advance treatment development is at hands reach (Riley et al., 2013; Nilsen et al., 2012; Kumar et al., 2013).

Fine grain visual inspection

The ability to detect reliable treatment effects over time greatly depends on the amount of observations. In laboratory experiments, where multiple measures can be repeatedly taken in a short period of time under highly controlled conditions, measurement reliability is not difficult to achieve, especially when the subjects involved are not humans. However, in clinical settings, SCDs encounter several measurement and logistical barriers. First, behavioral disorders can be observed in their natural environment only in some settings (e.g., school settings, in-patient units). This is generally not possible for the majority of mood, anxiety or psychotic disorders (Dougher & Dougher, 2000). Second, due to difficulties of observing individuals in their natural environment, observations of adults with a clinical behavioral disorder are restricted to weekly encounters and retrospective reports. This adds two problems: recall bias and low measurement frequency. Asking individuals a battery of questions with strong internal validity partially solves those problems, but such global and decontextualized measures ignore the dynamic patterns of behavior dependent upon a specific history of antecedents and consequences.

Weekly and global self-report measurements also complicate the analysis of SCDs. Visual inspection is one of the most common strategies to examine data gathered from SCDs (e.g., Hayes et al., 1999). The presence of a specific baseline pattern is used to decide whether or not to apply a new treatment phase. An ideal baseline is generally defined as two or three datapoints in which there is a stable level or a trend in the opposite direction than the desired level or trend of the targeted behavior (e.g., a reduction in tantrums Hayes et al., 1999). However, even when participants consent to participate in research studies, baselines must have a reasonable limit (e.g., 2 to 4 weeks). Moreover, when baselines can be extended, the total amount of observations derived from a clinical trial in which measures are collected on a weekly basis is small (e.g., 12 measurement points). Thus the use of weekly global self-reports in SCDs limits the ability of these designs to reliably detect the effect of interventions on targeted problems.

Mobile technologies can enhance the visual analysis of SCDs. First, mobile devices can be implemented in real-world settings without interfering with the normal course of a behavioral intervention in a clinical setting. For example, between an intake appointment and an intervention session, global self-report measures can gather between 1 and 2 data-points of the targetted behavior. Instead, mobile devices can enable the gathering of at least 6 datapoints of the desired target behavior (e.g., daily mood ratings). Carefully designed EMA studies sometimes involve 10 measurements per day (e.g., Kimhy et al., 2006; Hektner et al., 2007), so this number can be doubled or tripled by adding two to three measurements per day, which should not interfere with individual’s normal day-to-day activities. In this manner, a researcher can obtain a more reliable level and trend in only 1 week. This can help draw stronger conclusions about changes in level and trend as a result of the intervention. Furthermore, if sensing technology were to be used (e.g., motion trackers for behavioral activation), a more continuous and effortless flow of data could be obtained in the same time period. Figure 1 shows simulated data of global self-report measures of a mood outcome. These are compared to an EMA chart of the same outcome. Based on the global self-report chart, it is difficult to conclude that self-reported mood ratings were due to the experimental manipulation. These data might reflect an existing trend. The daily EMA chart provides more conclusive evidence that there was a disruption in the level and trend between the baseline and the intervention phases.

Figure 1.

Figure 1

Open in a new tab

Simulated chart showing the advantages of EMAs as compared to global self-report measures for the visual and statistical analysis of SCDs.

Increased statistical power

Global self-report measurements can become a barrier for the statistical analysis of SCDs using randomization tests (see Heyvaert in this issue). This non-parametric statistical procedure relies on considering all the observations from a SCD as a population of datapoints, and the random assignment of the beginning of treatment to one of those observations within that time continuum. The statistical power of any given dataset is directly proportional to the amount of available observations (Todman & Dugard, 2001; Edgington & Onghena, 2007).

The limited measurement frequency attained with weekly global self-reports limits the use of these analytic methods. For example, in an AB SCD with 12 data points in which the beginning of treatment is randomly assigned to weeks 1 through 12 and we require in each phase a minimum of 3 observations, the probability of obtaining a similar result given the assumption that there were no treatment effects is p = 1/7 = 0.142. In other words, given the small number of permutations, there is little power to detect an effect (Todman & Dugard, 2001; Edgington & Onghena, 2007; Ferron & Sentovich, 2002; Levin et al., 2012). A meta-analysis of sequential SCDs would increase statistical power (as described by Heyvaert et al in this special issue), but this would also require more subjects and resources.

Researchers interested in conducting randomization tests of SCD experiments with mobile technology could highly increase their statistical power. Taking the example described above and assuming that data was gathered once a day, there would be a total of 84 observations (see Figure 1). This would lead to an 11-fold increase in the experimenter’s ability to detect a treatment effect p = 1/79 = 0.012. Statistical power could be further increased by simply adding more than one measurement per day or by using sensing technology.

As we have seen, there is a synergy between mobile technologies and SCDs fro the experimental study of individuals in their natural environment. The amounts of data gathered by these devices can enhance the visual and statistical analysis of SCDs. Integrating mobile technologies, SCDs and randomization tests can open up a new wave of cost-effective, real-world, and experimentally and clinically-useful research at the individual level. Table 1 presents a summary of these implications together with specific examples.

Table 1.

Theoretical, methodological and statistical advantages of using mobile technologies

Advantages Description
Theoretical
Contextual delivery Interventions can be provided in their natural context (e.g., prior to a feared situation)
Contextual assessment Assessment of antecedents and consequences of target behaviors (e.g., specific triggers
and short term consequences)
Manipulation of verbal context Design of specific schedules of verbal prompting (e.g., a specific rate of motivative
augmentals)
Methodological
Increased Precision Constructs can be more precisely defined and measured (e.g., intensive repeated assessment,
minimal recall bias and sensing technology)
Evaluation of Scope Assessment of the impact of behavioral interventions across target behaviors and
situations (e.g., the effect of smoking cessation intervention on non-targetted behav-
iors)
Evaluation of Depth Assessment of the impact of behavioral interventions across levels of analysis (e.g.,
heart rate variability, social networks)
Statistical
Improved visual inspection Fine grained analysis of level and trend phase changes (see Figure 1)
Increased statistical power Drastic increase in measurement frequency enhances the use of randomization tests
for SCDs
Open in a new tab

Summary and conclusions

The task of creating a powerful behavioral science to tackle problems of human concern requires a combination of strong theory, experimental methods and technical innovation. In other words, in the same way that the results of our studies cannot be separated from our scientific questions and the content areas we choose to study from our methods, scientific progress and innovation cannot be separated from its technical means to achieve it. Thus the aims of this paper were to describe the synergy between mobile technologies and single case designs for the contextual behavioral understanding of the behavior of individuals in their natural environment.

We argued that mobile technologies allow CBS researchers to empirically examine hypotheses derived from learning theories of language and cognition (such as RFT). This can be accomplished by manipulating the verbal context of the individual and observing its effect in their day to day experience. Methodologically, mobile technologies can be used to evaluate the antecedents and consequences of specific behaviors and thus provide a more precise assessment of specific target behaviors. These contextual assessments can also be used to evaluate the scope of specific principles of change (i.e., their impact across a range of behaviors) and their depth across levels of analysis (e.g., physiology, social networks). Finally, we discussed the analytic implications of mobile technologies. We described how mobile technologies can enhance traditional strategies to analyze SCDs. Thanks to increased number of observations, visual analysis can be improved with a fine grain charting of phase changes, and randomization tests can dramatically improve their statistical power.

Mobile technologies are not without limitations. First, the development of these interventions still requires either grant funding or specialized expertise. Despite this, if Moore’s law (Wikipedia, 2013) continues to hold in the years to come, the cost of mobile behavioral health technologies will continue to drop, making more feasible the development of these tools by a wider range of contextual behavioral scientists. Second, even though SCDs can improve our understanding of the contextual behavioral etiology of behavioral problems, SCDs experiments should not be a substitute for larger population based trials. Randomized controlled trials have helped us improve our behavioral interventions. Therefore, we think the choice between SCDs and group designs is not an “either/or” decision. An increased use of SCDs would simply have the effect of diversifying our methods of exploration and make the process of discovery more agile (Riley et al., 2013). Finally, standalone mobile contextual behavioral health interventions can extend the reach of our treatments to a larger section of the population. However, these interventions have reduced ability to foster engagement and have notorious drop out rates (e.g., Eysenbach, 2005; Christensen & Mackinnon, 2006; Christensen et al., 2009). Combining mobile contextual behavioral interventions with face-to-face interventions might offer the advantage of capitalizing on a therapeutic relationship (e.g., direct modeling of processes, provision of non-verbal contingencies), while at the same time providing contextual behavioral assessments of target behaviors and an extension of the verbal context of the therapeutic relationship with targeted verbal cues.

In conclusion, we believe that if behavior therapy were created today, mobile technologies would probably be a primary tool of choice for behavioral scientists. However, behavior therapy emerged during the 1950s (e.g., Lindsley et al., 1953), not in 2013. The widespread growth of mobile technologies gives behavioral scientists a readily available platform to conduct single case behavioral experiments, and thus mobile technologies holds great promise for the next generation of contextual behavioral science. We hope this article provided a theoretical framework to understand their scientific advantages and encourages their adoption.

Footnotes

1

From now on we will refer to both as EMAs.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Contributor Information

Roger Vilardaga, University of Washington & Fred Hutchinson Cancer Research Center

Jonathan Bricker, Fred Hutchinson Cancer Research Center & University of Washington

Michael McDonell, University of Washington

References

  1. Aguilera A, Muench F. There’s an app for that: Information technology applications for cognitive behavioral practitioners. The Behavior Therapist. 2012;35(4):65–73. [PMC free article] [PubMed] [Google Scholar]
  2. Ali A, Hossain S, Hovsepian K, Rahman M, Kumar S. Smoketrack: Automated detection of cigarette smoking in the mobile environment from respiration measurements. ACM IPSN; 2012. [Google Scholar]
  3. Backman CL, Harris SR. Case studies, single-subject research, and n of 1 randomized trials: comparisons and contrasts. American journal of physical medicine & rehabilitation / Association of Academic Physiatrists. 1999 Apr;78(2):170–176. doi: 10.1097/00002060-199903000-00022. (PMID: 10088595) [DOI] [PubMed] [Google Scholar]
  4. Barlow DH, Hayes SC, Nelson RO. The scientist practitioner: Research and accountability in clinical and educational settings. Vol. 128. Cambridge Univ Press; 1984. [Google Scholar]
  5. Barnes-Holmes D, Stewart I, Dymond S, Roche B, Dougher MJ. Clinical behavior analysis. Context Press; Reno, NV, US: 2000. A behavior-analytic approach to some of the problems of the self: A relational frame analysis; pp. 47–74. [Google Scholar]
  6. Ben-Zeev D, McHugo GJ, Xie H, Dobbins K, Young MA. Comparing retrospective reports to real-Time/Real-Place mobile assessments in individuals with schizophrenia and a nonclinical comparison group. Schizophrenia Bulletin. 2012;38(3):396–404. doi: 10.1093/schbul/sbr171. 10.1093/schbul/sbr171. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Biglan A, Hayes SC. Should the behavioral sciences become more pragmatic? the case for functional contextualism in research on human behavior. Applied & Preventive Psychology. 1996;5(1):47–57. [Google Scholar]
  8. Burns MN, Begale M, Duffecy J, Gergle D, Karr CJ, Giangrande E, Mohr DC. Harnessing context sensing to develop a mobile intervention for depression. Journal of Medical Internet Research. 2011;13(3) doi: 10.2196/jmir.1838. 10.2196/jmir.1838. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Chen C, Haddad D, Selsky J, Hoffman JE, Kravitz RL, Estrin DE, Sim I. Making sense of mobile health data: An open architecture to improve individual- and population-level health. Journal of Medical Internet Research. 2012;14(4) doi: 10.2196/jmir.2152. (PMID: 22875563 PMCID: PMC3510692) 10.2196/jmir.2152. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression: Systematic review. Journal of medical Internet research. 2009;11(2) doi: 10.2196/jmir.1194. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Christensen H, Mackinnon A. The law of attrition revisited. Journal of medical Internet research. 2006;8(3) doi: 10.2196/jmir.8.3.e20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Cooke G. The efficacy of two desensitization procedures: An analogue study. Behaviour Research and Therapy. 1966;4(1):17–24. [Google Scholar]
  13. Csikszentmihalyi M, Larson R. Validity and reliability of the experience-sampling method. Journal of Nervous and Mental Disease. 1987;175(9):526–536. doi: 10.1097/00005053-198709000-00004. [DOI] [PubMed] [Google Scholar]
  14. Csikzentmihalyi M, Larson R, Prescott S. Ecology of adolescent activity and experience. Journal of Youth and Adolescence. 1977;6(3):281–294. doi: 10.1007/BF02138940. [DOI] [PubMed] [Google Scholar]
  15. Dahl J, Plumb J, Stewart I, Lundgren T. The art and science of valuing in psychotherapy. New Harbinger; Oakland: 2009. [Google Scholar]
  16. Dallery J, Cassidy RN, Raiff BR. Single-case experimental designs to evaluate novel technology-based health interventions. Journal of medical Internet research. 2013;15(2):e22. doi: 10.2196/jmir.2227. (PMID: 23399668) 10.2196/jmir.2227. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. D?glise C, Suggs LS, Odermatt P. Short message service (SMS) applications for disease prevention in developing countries. Journal of medical Internet research. 2012;14(1):e3. doi: 10.2196/jmir.1823. (PMID: 22262730) 10.2196/jmir.1823. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Dougher MJ, Dougher MJ. Clinical behavior analysis. Context Press; Reno, NV, US: 2000. [Google Scholar]
  19. Dymond S. Advances in relational frame theory: research and application. New Harbinger Pub; Oakland, CA, USA: 2013. [Google Scholar]
  20. Dymond S, Barnes D. Behavior-analytic approaches to self-awareness. Psychological Record. 1997;47(2):181–200. [Google Scholar]
  21. Ebbinghaus H. Memory: A contribution to experimental psychology. No. 3. Teachers college, Columbia university; 1913. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Edgington E, Onghena P. Randomization tests. Chapman & Hall; Boca Raton, FL, USA: 2007. [Google Scholar]
  23. Eysenbach G. The law of attrition. Journal of medical Internet research. 2005;7(1) doi: 10.2196/jmir.7.1.e11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Ferron J, Sentovich C. Statistical power of randomization tests used with multiple-baseline designs. The Journal of Experimental Education. 2002 Jan;70(2):165–178. 10.2307/20152674. [Google Scholar]
  25. Fertig RT. The software revolution: trends, players, market dynamics in personal computer software. Elsevier North-Holland, Inc; New York, NY, USA: 1985. [Google Scholar]
  26. Golijani-Moghaddam N, Hart A, Dawson DL. The implicit relational assessment procedure: Emerging reliability and validity data. Journal of Contextual Behavioral Science. (in press) 10.1016/j.jcbs.2013.05.002. [Google Scholar]
  27. Hayes SC, Barlow DH, Nelson-Gray RO. The scientist practitioner: Research and accountability in the age of managed care. 2nd ed Allyn & Bacon; 1999. [Google Scholar]
  28. Hayes SC, Barnes-Holmes D, Roche B. Relational frame theory: A post-skinnerian account of human language and cognition. Kluwer Academic/Plenum Publishers; New York, NY, USA: 2001. [DOI] [PubMed] [Google Scholar]
  29. Hayes SC, Barnes-Holmes D, Wilson KG. Contextual behavioral science: Creating a science more adequate to the challenge of the human condition. Journal of Contextual Behavioral Science. 2012 Dec;1(1?2):1–16. (Cited by 0007) 10.1016/j.jcbs.2012.09.004. [Google Scholar]
  30. Hayes SC, Hayes LJ, Reese HW, Sarbin TR. Varieties of scientific contextualism. Context Press; Reno, NV, USA: 1993. [Google Scholar]
  31. Hayes SC, Strosahl KD, Wilson KG. Acceptance and commitment therapy, second edition: The process and practice of mindful change. Guilford Press; New York, NY, USA: 2011. [Google Scholar]
  32. Hektner JM, Schmidt JA, Csikszentmihalyi M. Experience sampling method: measuring the quality of everyday life. SAGE; Chicago, IL, USA: 2007. [Google Scholar]
  33. Hooper N, Stewart I, Duffy C, Freegard G, McHugh L. Modelling the direct and indirect effects of thought suppression on personal choice. Journal of Contextual Behavioral Science. 2012;1(1–2):73–82. 10.1016/j.jcbs.2012.06.001. [Google Scholar]
  34. Hughes S, Barnes-Holmes D, Vahey N. Holding on to our functional roots when exploring new intellectual islands: A voyage through implicit cognition research. Journal of Contextual Behavioral Science. 2012;1(1–2):17–38. 10.1016/j.jcbs.2012.09.003. [Google Scholar]
  35. iHealthBeat. (n.d.) [Retrieved 2013-07-31];44M mobile health apps will be downloaded in 2012, report predicts - iHealthBeat. from http://www.ihealthbeat.org/articles/2011/12/1/44m-mobile-health-apps-will-be-downloaded-in-2012-report-predicts.
  36. Kantor J. The scientific evolution of psychology. Principia Press; Chicago: 1963. [Google Scholar]
  37. Kazdin AE. Assessing the clinical or applied importance of behavior change through social validation. Behavior Modification. 1977;1(4):427–452. [Google Scholar]
  38. Kazdin AE. Evaluating the generality of findings in analogue therapy research. Journal of Consulting and Clinical Psychology. 1978;46(4):673–686. doi: 10.1037//0022-006x.46.4.673. (PMID: 670514) [DOI] [PubMed] [Google Scholar]
  39. Kimhy D, Delespaul P, Corcoran C, Ahn H, Yale S, Malaspina D. Computerized experience sampling method (ESMc): assessing feasibility and validity among individuals with schizophrenia. Journal of Psychiatric Research. 2006;40(3):221–230. doi: 10.1016/j.jpsychires.2005.09.007. 10.1016/j.jpsychires.2005.09.007. PMID: 16300791. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Kravitz RL, Duan N, Niedzinski EJ, Hay MC, Subramanian SK, Weisner TS. What ever happened to n-of-1 trials? insiders’ perspectives and a look to the future. The Milbank Quarterly. 2008;86(4):533–555. doi: 10.1111/j.1468-0009.2008.00533.x. (PMID: 19120979 PMCID: PMC2690377) 10.1111/j.1468-0009.2008.00533.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Kumar S, Nilsen WJ, Abernethy A, Atienza A, Patrick K, Pavel M, Swendeman D. Mobile health technology evaluation: The mHealth evidence workshop. American Journal of Preventive Medicine. 2013;45(2):228–236. doi: 10.1016/j.amepre.2013.03.017. 10.1016/j.amepre.2013.03.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Layng T, Twyman JS, Stikeleather G. Engineering discovery learning: The contingency adduction of some precursors of textual responding in a beginning reading program. Analysis of Verbal Behavior. 2004;20:99–109. doi: 10.1007/BF03392997. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Levin JR, Ferron JM, Kratochwill TR. Nonparametric statistical tests for single-case systematic and randomized ABAB?AB and alternating treatment intervention designs: new developments, new directions. Journal of school psychology. 2012 Oct;50(5):599–624. doi: 10.1016/j.jsp.2012.05.001. 10.1016/j.jsp.2012.05.001. [DOI] [PubMed] [Google Scholar]
  44. Lillie EO, Patay B, Diamant J, Issell B, Topol EJ, Schork NJ. The n-of-1 clinical trial: the ultimate strategy for individualizing medicine? Per Med. 2011;8(2):161–173. doi: 10.2217/pme.11.7. (PMID: 21695041) [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Lindsley O, Skinner B, Solomon H. Studies in behavior therapy (status report i) Metropolitan State Hospital; Walthama, MA: 1953. [Google Scholar]
  46. McDonell MG, Srebnik D, Angelo F, McPherson S, Lowe J, Sugar A, Ries R. Randomized controlled trial of contingency management for stimulant use in community mental health patients with serious mental illness. American Journal of Psychiatry. 2013;170(1):94–101. doi: 10.1176/appi.ajp.2012.11121831. 10.1176/appi.ajp.2012.11121831. PMID: 23138961. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Miller G. The smartphone psychology manifesto. Perspectives on Psychological Science. 2012;7(3):221–237. doi: 10.1177/1745691612441215. 10.1177/1745691612441215. [DOI] [PubMed] [Google Scholar]
  48. Mobihealthnews. (n.d.) [Retrieved 2013-07-31];An analysis of consumer health apps for apple?s iPhone 2012 ∣ mobihealthnews. from http://mobihealthnews.com/research/an-analysis-of-consumer-health-apps-for-apples-iphone-2012/
  49. Morris ME, Aguilera A. Mobile, social, and wearable computing and the evolution of psychological practice. Professional Psychology-Research and Practice. 2012;43(6):622–626. doi: 10.1037/a0029041. 10.1037/a0029041. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Nilsen W, Kumar S, Shar A, Varoquiers C, Wiley T, Riley WT, Atienza AA. Advancing the science of mHealth. J Health Commun. 2012;17(Suppl 1):5–10. doi: 10.1080/10810730.2012.677394. (PMID: 22548593) [DOI] [PubMed] [Google Scholar]
  51. Paul G. Behavior modification research: Design and tactics. In: Franks CM, editor. Behavior therapy: Appraisal and status. McGraw-Hill; Oxford: 1969. pp. 29–62. [Google Scholar]
  52. Petry NM, Alessi SM, Carroll KM, Hanson T, MacKinnon S, Rounsaville B, Sierra S. Contingency management treatments: Reinforcing abstinence versus adherence with goal-related activities. Journal of Consulting and Clinical Psychology. 2006;74(3):592–601. doi: 10.1037/0022-006X.74.3.592. [DOI] [PubMed] [Google Scholar]
  53. Plarre K, Raij A, Hossain S, Ali A, Nakajima M, Al’absi M, Wittmers L. Continuous inference of psychological stress from sensory measurements collected in the natural environment. Information processing in sensor networks (ipsn), 2011 10th international conference.2011. pp. 97–108. [Google Scholar]
  54. Preece SJ, Goulermas JY, Kenney LPJ, Howard D, Meijer K, Crompton R. Activity identification using body-mounted sensors???a review of classification techniques. Physiological Measurement. 2009;30(4):R1. doi: 10.1088/0967-3334/30/4/R01. [DOI] [PubMed] [Google Scholar]
  55. Riley WT, Glasgow RE, Etheredge L, Abernethy AP. Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clin Transl Med. 2013;2(1):10. doi: 10.1186/2001-1326-2-10. (PMID: 23663660) [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Sato H, Kawahara J.-i. Selective bias in retrospective self-reports of negative mood states. Anxiety, stress, and coping. 2011;24(4):359–367. doi: 10.1080/10615806.2010.543132. (PMID: 21253957) 10.1080/10615806.2010.543132. [DOI] [PubMed] [Google Scholar]
  57. Shamseer L, Sampson M, Bukutu C, Barrowman N, Altman D, Moher D, Vohra S. P05.50. CONSORT extension for n-of-1 trials (CENT) guidelines. BMC Complementary and Alternative Medicine. 2012;12(Suppl 1):P410. (PMID: null PMCID: PMC3373855) 10.1186/1472-6882-12-S1-P410. [Google Scholar]
  58. Sisson LA, Van Hasselt VB, Hersen M. Behavioral interventions to reduce maladaptive responding in youth with dual sensory impairment. An analysis of direct and concurrent effects. Behavior Modification. 1993;17(2):164–188. doi: 10.1177/01454455930172005. (PMID: 8471010) [DOI] [PubMed] [Google Scholar]
  59. Skinner B. The behavior of organisms: an experimental analysis. Appleton-Century; Oxford, England: 1938. [Google Scholar]
  60. Skinner B. Verbal behavior. Appleton-Century-Crofts; East Norwalk: 1957. [Google Scholar]
  61. Smith A. [Retrieved 2012-08-01];Americans and their cell phones. Pew Internet & American Life Project. 2012 Aug; from http://pewinternet.org/~/media//Files/Reports/2011/Cell%20Phones%202011.pdf.
  62. Stewart I, Barnes-Holmes D, Roche B. A functional-analytic model of analogy using the relational evaluation procedure. Psychological Record. 2004;54(4):531–552. [Google Scholar]
  63. Stone AA, Shiffman S. Ecological momentary assessment (ema) in behavorial medicine. Annals of Behavioral Medicine. 1994 [Google Scholar]
  64. Todman J, Dugard P. Single-case and small-n experimental designs. Lawrence Erlbaum; New York, NY, USA: 2001. [Google Scholar]
  65. Vilardaga R, Estevez A, Levin ME, Hayes SC. Deictic relational responding, empathy, and experiential avoidance as predictors of social anhedonia: Further contributions from relational frame theory. Psychological Record. 2012;62(3):409–432. [Google Scholar]
  66. Vilardaga R, Hayes SC, Levin ME, Muto T. Creating a strategy for progress: A contextual behavioral science approach. The Behavior Analyst. 2009;32(1):105–133. doi: 10.1007/BF03392178. (PMID: 22478516 PMCID: PMC2686981) [DOI] [PMC free article] [PubMed] [Google Scholar]
  67. Wenze SJ, Miller IW. Use of ecological momentary assessment in mood disorders research. Clinical Psychology Review. 2010;30(6):794–804. doi: 10.1016/j.cpr.2010.06.007. 10.1016/j.cpr.2010.06.007. [DOI] [PubMed] [Google Scholar]
  68. Wikipedia [Retrieved 2013-06-28];Moore’s law. 2013 Jun; from http://en.wikipedia.org/w/index.php?title=Moore%27s_law&oldid=561477629 (Page Version ID: 561477629)
  69. Wolpe J. Psychotherapy by reciprocal inhibition. Conditional reflex: a Pavlovian journal of research & therapy. 1968;3(4):234–240. doi: 10.1007/BF03000093. [DOI] [PubMed] [Google Scholar]

RESOURCES