Development and Preliminary Validation of the Patient Outcome Planning Calculator (POP-C): A Tool for Determining Treatment Dosage in Applied Behavior Analysis

Abstract

Board certified behavior analysts (BCBA) are responsible for determining the medically necessary treatment dosage for patients (i.e., the number of hours of therapy a patient should receive per week to optimize progress) during applied behavior analysis (ABA) therapy. However, because there is currently no standard method for making these determinations, BCBAs must rely on their own clinical judgment. Given that clinical judgment may be underdeveloped in some BCBAs, particularly those who are newly certified, more formal strategies are needed to guide decision making as it relates to medical necessity and treatment dosage. In this article we describe the development of the Patient Outcome Planning Calculator (POP-C), a standardized decision-making tool designed to assist novice practitioners in determining the medically necessary ABA treatment intensity and appropriate treatment setting for individuals with autism spectrum disorder (ASD). We present preliminary reliability data as well as construct validity data indicating statistically significant correlations between the POP-C and several norm-referenced and criterion-referenced assessments commonly used to estimate skill level and the corresponding degree of support needed within the ASD population to inform the ABA treatment model and goals.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40617-023-00861-6.

Autism spectrum disorder (ASD)—a neurodevelopmental disorder characterized by deficits in social communication skills, restricted interests, and repetitive behavior—has experienced a significant increase in prevalence during the last several decades, with current estimates indicating that 1 in 36 individuals will qualify for an ASD diagnosis (Centers for Disease Control & Prevention [CDC], 2023). As a result, treatment approaches that can help improve symptoms of ASD that negatively affect quality of life for affected individuals are clearly needed. Although treatment options are many (Green et al., 2006; Hess et al., 2008), none have quite the breadth of research to support its use as applied behavior analysis (ABA) intervention (CDC, 2022). ABA intervention for individuals with ASD was initially popularized by the publication of the landmark study reporting outcomes from the UCLA-Young Autism Project, which demonstrated dramatic improvements in children with ASD following intensive (i.e., average of 40 hr per week) long-term treatment (Lovaas, 1987). Since then a great number of studies have been published lending additional support for the effectiveness of ABA treatment, including at least 11 meta-analytic studies (Eckes et al., 2023; Eldevik et al., 2009, 2010; Kuppens & Onghena, 2012; Makrygianni et al., 2018; Makrygianni & Reed, 2010; Peters-Scheffer et al., 2011; Reichow, 2012; Reichow & Wolery, 2009; Spreckley & Boyd, 2009; Virués-Ortega, 2010). As a result of the robust literature establishing the efficacy of ABA and the work of many parent advocates, insurance-funded therapy is now the norm rather than the exception. As of 2019, all 50 U.S. states require insurance coverage for autism treatment. Thus, many insurance providers have adopted highly specific policy coverage criteria that is used to review and make benefit decisions related to treatment intensity or treatment dosage1 recommendations for members with ASD.

Research generally indicates that ABA treatment provided at high dosages produces the largest improvements for young children with ASD (Eldevik et al., 2009; Klintwall et al., 2015; Virués-Ortega et al., 2013; cf. Rogers et al., 2021). At least 36 hr of direct ABA treatment per week for at least 2 years has been associated with clinically significant, reliable changes in cognitive and adaptive skills (Eldevik et al., 2010). Much research suggests that low-dose ABA treatment yields smaller improvements than ABA treatment at higher dosages (Eldevik et al., 2006, 2012; Green, 2011; Peters-Scheffer et al., 2010). Although most published outcome research on ABA treatment seems to suggest that higher dosages (i.e., 20 or more weekly hours of treatment) produce the best outcomes, recent investigations have questioned this assertion, citing methodological limitations with many of the outcome studies supporting the link between high treatment dosage and better outcomes for young children with ASD (Daniolou et al., 2022; Rogers et al., 2021) and highlighting that little is known about the relation between dosage and outcomes for older children, adolescents, and adults with ASD. Indeed, ASD is a complex and heterogeneous disorder that affects individuals differently. Therefore, treatment dosage recommendations must reflect these differences and must be customized to the strengths and needs of the individual and their environment (Council for Autism Service Providers [CASP], 2020).

A central issue in understanding the relationship between treatment dosage and outcomes is the degree to which individual patient factors influence response to treatment and how these factors should be incorporated into dosage recommendations. Research suggests that several factors could influence a patient’s response to therapy. In particular, IQ is often cited as a moderator to treatment outcomes, such that patients with lower cognitive ability or comorbid intellectual disability tend to make slower progress towards goals in therapy (Hedvall et al., 2015; Magiati et al., 2011). In addition, research suggests that patients who display greater social avoidance and withdrawal behavior at baseline often make less progress in their communication and social skills during treatment than patients with less social avoidance behavior (Robain et al., 2020). On the other hand, those with higher levels of communication and higher toy engagement often display quicker progress and more treatment gains over time (Contaldo et al., 2020; Fossum et al., 2018). Challenging behavior also plays a role in patient progress. Research suggests that the presence of challenging behavior may be a barrier to significant treatment gains (Hedvall et al., 2015; Robain et al., 2020).

Finally, although it has been widely accepted that early intervention leads to “best outcome” (that is, the earlier treatment begins, the better the outcome), recent research has questioned the strength of evidence to support the relationship between age at onset of treatment and treatment outcome (Cerasuolo et al., 2022). Further, ABA practice guidelines are clear that treatment should not be constrained by age and that ABA is effective across the lifespan (i.e., research has not established an age limit beyond which ABA is ineffective; CASP, 2020). Therefore, it may not be appropriate to use chronological age at treatment onset as a determining variable in making dosage recommendations. For these reasons, recommendations and treatment often vary considerably across individuals in terms of dosage, duration, complexity of treatment goals, and the extent of direct treatment provided.

The process of determining the degree of symptom severity, the number of behavioral targets requiring support, and the corresponding treatment dosage is often referred to as the medical necessity determination. The term medical necessity was initially developed by Congress in the 1960s to define the limits of insurance coverage (Bergthold, 1995). It became increasingly popular with the growth of managed care and integrated health, and now commonly extends to the coverage of ABA therapy for patients with ASD. On a general level, medical necessity refers to a group of characteristics that must be met in order to qualify for medical service coverage. However, provider-specific determinations about medical necessity are often less specific in mental and behavioral health settings compared to other areas of medicine. At present, there is no singular definition of medical necessity as it relates to coverage for ABA treatment and interpretation of medical necessity varies across managed care organizations. Disagreements about the medical necessity of treatment or dosage recommendations call into question who is most qualified to make these important determinations (Sabin & Daniels, 1994). Whereas medical necessity may be defined by state Medicaid or commercial health plans or by federal laws, it is the professional behavior analyst who is ultimately responsible for determining and justifying the medical necessity of their treatment dosage recommendations.

For practitioners providing insurance-funded ABA therapy, determining the medical necessity and corresponding dosage and treatment recommendations for a patient may come as a challenge due to a paucity of resources or guidance available on this aspect of clinical practice. In 2016, Papatola and Lustig published guidance for clinicians in navigating medical necessity in the context of a peer-review, i.e., a scheduled phone conversation during which a clinician discusses the need for treatment with a representative from the insurance company in order to obtain authorization. The authors provide a short list of elements they deem necessary for establishing medical necessity for ABA treatment, including: (1) a DSM-5 diagnosis supported by symptoms; (2) symptoms that cause impairment in functioning; and (3) family involvement as evidence of motivation for change. Despite these guidelines, which provide some initial clarity for ABA providers on medical necessity criteria, the author's suggestions leave much open for interpretation by way of clinical judgment.

Clinical judgment is often synonymous with other terms such as clinical reasoning, critical thinking, problem solving, and diagnostic reasoning (Brentnall et al. 2022). It is defined as, “an interpretation or conclusion about a patient’s needs, concerns, or health problems, the decision to take action (or not), use or modify standard approaches, or improve new ones as deemed appropriate by the patient’s response” (Tanner, 2006, p. 66). When it comes to clinical judgment, intuition may provide us confidence in our ability to make the “correct” decision, however, the literature suggests this may not be the case. In fact, there is a significant body of literature in nursing, psychology, and forensics on the topic, much of which has focused on the development of tools to shape a strong clinical judgment repertoire among novices. These tools aim to align decision making with best practice, regardless of experience and history in the role (Adamson et al., 2001; Banner & Olney, 2007; O’Neill et al., 2005). Benner (1984) acknowledged that inexperienced nurses often use decision-making tools, procedures, and guidelines to augment clinical decisions and reduce errors in judgment that could negatively affect patient outcomes. Like nursing, the field of behavior analysis needs resources to support novice practitioners in developing and refining their clinical judgment because, by nature of being a novice, there is little prior history and experience to guide decision making.

With more than half of all behavior analysts becoming certified in the last 5 years (55% of the overall number of BCBA certificants in 2022 were certified between 2018 and 2022; BACB, n.d.), it is no surprise to see that the field of behavior analysis has also begun developing decision-making models to help ease and improve clinical decision making of practitioners2 and corresponding outcomes (Geiger et al., 2010; LeBlanc et al., 2016; Kipfmiller et al., 2019; Brodhead & Truckenmiller, 2021; see also Suarez et al., 2022). LeBlanc et al. (2016) indicate that these models “consist of a series of questions that can be answered to lead a practitioner to recommendations about interventions that are optimally matched to clinical considerations” (p.77). Such models aim to reduce the probability of bias and increase the practitioner’s sensitivity to relevant variables linked to the likely consequences of the decision (Marya et al., 2022).

The purpose of the current study is to describe the development of the Patient Outcome Planning Calculator (POP-C), a standardized decision-making tool designed to (1) guide BCBAs in synthesizing information relevant to ABA treatment outcomes to support medical necessity determinations; and (2) provide individualized recommendations about optimal treatment dosage ranges for patients seeking ABA therapy. Although other similar tools currently exist in the field (e.g., the Medical Necessity Assessment; Davis-Wilson, 2019) to our knowledge, the current study of the POP-C is the first to report preliminary reliability and validity data to establish strong psychometric properties before recommending widespread use.

Method

Tool Development

The POP-C was developed and tested at a mid-size, ABA provider in the Midwest. This organization provides treatment to patients across the lifespan with no age restrictions or exclusionary criteria, in both center and home-based settings, at dosages ranging from 20–40 hr per week for comprehensive services.3 Additional options for patients who require more focused support include weekly social skills groups for children and adolescents, and caregiver training and consultation services.

The POP-C was developed by the first author, a doctoral-level licensed psychologist, diagnostician, and BCBA with expertise in the assessment and treatment of ASD and related disorders. The development of the tool was informed by an in-depth review and synthesis of the current literature on individual patient factors that influence response to ABA therapy at various dosage levels (see Cerasuolo et al., 2022; Contaldo et al., 2020; Hedvall et al., 2015; Robain et al., 2020), along with the first author’s expert knowledge of the diagnostic criteria for autism and its accompanying symptom severity levels. Tool items focus primarily on autism symptom severity in the core areas of social communication and social interactions that affect a patient's ability to be successful. Accompanying symptoms that lead to further challenges with independence were included such as adaptive skill level, challenging behavior, sensory sensitivities, comorbid diagnoses, and safety concerns.

Content validity was established through consultation with experts in the field of ABA, as well as a formal survey submitted to six director-level BCBAs in September 2021. Respondents had an average of 10 years of experience making dosage determinations as practicing BCBAs and were considered to be internal subject matter experts on the topic. Respondents rated the degree to which each variable influenced their ultimate dosage determination with patients on a 5-point rating scale, where scores of 1 indicated a variable had “no influence” on dosage determinations and scores of 5 indicated a variable had a “major influence” in dosage determinations. Results are displayed in Table 1. Variables with a mean rating of 3 or more were considered for inclusion on the POP-C, whereas those with a mean rating of less than 3 were not included in the tool.

Table 1.

Variables ranked from most-to-least influential in dosage determinations by subject matter experts

Survey Item Number	Item Description	Mean Item Rank	Mean Rating	SD
17	History of behavioral hospitalizations in the last year	1	4.83	0.41
19	Presence of repetitive behaviors	1	4.83	0.41
21	Level of verbal communication	1	4.83	0.41
23	Absence of age-appropriate social skills	1	4.83	0.41
24	Absence of age-appropriate coping skills	1	4.83	0.41
30	Presence of challenging behavior in the home or school setting	1	4.83	0.41
31	Presence of challenging behavior in the community	1	4.83	0.41
18	Presence of restricted interests	2	4.67	0.52
26	Absence of age-appropriate leisure skills	3	4.50	0.84
27	Absence of age-appropriate self-care skills	3	4.50	0.84
20	Normative level of delay (e.g., Vineland, IQ score)	4	4.33	0.82
29	Presence of food selectivity or mealtime/eating difficulties	4	4.33	0.82
11	Presence of comorbid intellectual disability	5	4.17	0.75
32	Patient “readiness to learn”	5	4.17	1.17
13	Presence of comorbid psychiatric disorders (e.g., ADHD, ODD, PTSD)	6	3.67	0.52
28	Presence of sleep difficulties	6	3.67	1.03
22	Use of Augmentative and Alternative Communication (AAC)	7	3.50	1.97
1	Patient age at therapy onset	8	3.33	1.03
12	Presence of comorbid genetic disorder	8	3.33	0.52
25	Absence of age-appropriate academic skills	8	3.33	1.37
33	Response to previous therapy interventions (if applicable)	9	2.83	1.33
10	Financial/insurance variables (e.g., specific funding source, limited age-range coverage, daily copays)	10	2.67	1.03
14	Physical limitations (e.g., nonambulatory, use of assistive equipment)	11	2.50	1.64
6	Extracurricular or community-based activities that support quality of life	12	2.33	0.82
16	History of medical hospitalizations in the last year	13	2.17	1.17
5	Caregiver knowledge/experience with ABA at intake	14	1.83	1.33
2	Current school schedule and school-based supports	15	1.67	0.82
3	Caregiver commitment and willingness to participate in therapy at intake	15	1.67	0.82
4	Caregiver preference for therapy location (i.e., clinic or home-based)	15	1.67	0.82
8	Family geography (i.e., distance from family home to nearest clinic)	15	1.67	0.82
9	Other family characteristics (e.g., caregiver’s response to diagnosis and treatment, physical home environment, caregiver’s physical, psychosocial, or socio-economic status)	15	1.67	0.52
7	Family scheduling barriers (e.g., parent work schedule, sibling extracurricular activities)	16	1.50	0.84
15	History of seizures	16	1.50	0.84

The items included in the POP-C were carefully selected, designed, and refined over a 12-month period, during which time several iterations were piloted, and feedback was solicited from all authors as well as several additional external subject matter experts who practice, teach, and conduct research in the field of behavior analysis.

The POP-C uses an electronic form to prompt the BCBA to answer a series of questions about their patient’s presenting symptoms and concerns within two scoring sections: a Treatment Dosage section (16 questions) and a Treatment Setting section (13 questions). Additional information is also collected on patient demographics and logistical considerations for treatment (i.e., caregiver preferences for therapy, anticipated scheduling or administrative challenges, and specific familial or cultural considerations); however, this information is not incorporated into the scoring process. (See supplementary file for example items and responses.)

Scoring

The Treatment Dosage section assesses symptom severity levels across the following domains: social communication, social interactions, restricted interests, and repetitive behaviors, play skills, adaptive skills, challenging behavior, sensory sensitivities, psychological and medical comorbid concerns, accompanying symptoms (eating or feeding difficulties, sleeping difficulties, etc.) and safety concerns.4 The Treatment Setting section assesses the patient’s abilities and school readiness skills within the following areas: attending to a group activity, sitting with a group of peers, transitioning between activities, following directions, elopement in school or community settings, and the presence of severe or challenging behavior in school or community settings.

Items are presented as multiple-choice questions with each response option corresponding to a different score. As a general example, item response options include descriptions of mild, moderate, and severe symptom presentations, corresponding to varying weights (e.g., mild = 3 points, moderate = 5 points, severe = 10 points). Response options indicating greater skill proficiencies (e.g., engages in spontaneous play) are assigned lower weights than response options indicating greater impairment or skill deficits (e.g., no interest in toys), which are weighted more heavily. Overall, higher scores across items included in the POP-C indicate a higher degree of support required across domains and a greater number of behavioral targets to be addressed in treatment, which result in a higher total score and a higher dosage recommendation. Table 2 illustrates how outcome research studies were broadly utilized in item scoring.

Table 2.

Medical necessity factors and their impact on POP-C score

Select factors that result in a higher score	Select citation(s) supporting scoring
Comorbid intellectual disability	Magiati et al., 2011; Hedvall et al., 2015
Higher autism symptom severity level	Contaldo et al., 2020
Frequent or severe challenging behavior	Hedvall et al., 2015
Absence of functional communication skills	Contaldo et al., 2020
Presence of social avoidance/withdrawal	Robain et al., 2020

The total score on the POP-C is used to generate dosage and therapy setting recommendations within three broad categories:

1. Low Intensity: up to 20 hours with family treatment guidance; or social skills group with family treatment guidance; school or community setting

2. Medium Intensity: 20–30 hours with family treatment guidance; center or home-based setting

3. High Intensity: 30–40 hours per week of direct treatment with family treatment guidance; center or home-based setting

The dosage ranges were purposefully designed to be broad, allowing flexibility for the consideration of additional factors and patient or family contexts that may indicate the need for more (or fewer) treatment hours for a patient. For example, whereas the number of hours spent in a traditional school setting is not a factor included in scoring, a patient’s degree of success learning new skills in that setting may influence whether the assessor recommends a treatment intensity at the high-end of the recommended range (e.g., if the patient has low or no success learning in a traditional school setting) or a treatment intensity at the low-end of the recommended range (e.g., if the patient has emerging success learning in a traditional school setting). The ranges were meant to provide a starting point for practitioners by narrowing the range of hours that is medically necessary based on presenting symptom severity and skill level, wherein they may then use their own clinical judgment to make a more precise treatment intensity recommendation. The ultimate goal of the POP-C is to help highlight the wide variety of individual patient symptoms and skill areas that are important for BCBAs to consider when making dosage decisions.

Pilot Study

Participants and Setting

A total of 77 participants were involved in the present study, including 19 females (25%) and 58 males (75%). Demographic information can be found in Table 3. Patient ages ranged from 2 to 20 years. All participants had a prior diagnosis of ASD and were seeking to enroll in ABA therapy services with the participating organization. Of note, 63% of patients presented with co-morbid diagnoses in various areas (e.g., cognitive impairment, developmental disability, ADHD, anxiety).

Table 3.

Participant demographic information

Characteristics	N (M ± SD)	%
Age
2–4	28 (3 ± 0.7)	26.0
5–7	31 (6 ± 0.8)	40.3
8–10	10 (9 ± 0.6)	13.0
11–13	7 (12 ± 0.9)	9.1
14–19	0	0.0
20	1	1.3
Gender
Female	19	24.7
Male	58	75.3
Diagnosis
Autism	28	36.4
Dual diagnosis	49	63.6

M  Mean, SD Standard Deviation

Materials and Training

In addition to administering the POP-C, several other assessment protocols were individually selected and administered as part of the initial assessment process. In many cases, a single assessment protocol was selected based on individual patient needs, record reviews, and symptom presentations during intake observations, including both norm- and criterion-referenced measures as noted in Table 4. That is, all assessment protocols were not administered across patients, (i.e., those who received the VB-MAPP did not also receive the EFL).

Table 4.

Assessments included in pilot data analysis

Assessment Protocol	Author & Year	Skill Focus
Essential for Living (EFL)	McGreevy & Fry, 2013	Functional Life Skills
Verbal Behavior Milestones Assessment and Placement Program (VB-MAPP)	Sundberg, 2008	Verbal Behavior
Vineland Adaptive Behavior Scales, Third Edition (Vineland-3)	Sparrow et al., 2016	Adaptive and Daily Living Skills

Three BCBAs were selected and trained to administer the POP-C tool as part of the pilot project due to their expertise in making dosage recommendations. These individuals held a specialty designation as “Assessment BCBAs” at the participating organization. They had been provided with extensive training across a battery of initial assessment protocols and had demonstrated a high degree of fidelity administering each of the included protocols. They were exclusively responsible for assessment administration, scoring, interpretation, and initial treatment planning, and were considered to be internal subject-matter experts on assessment. They received ongoing supervision and training from the first author on standardized and criterion referenced assessment for 4 hr monthly, with additional biweekly supervision for 1 hr. During the initial training process, the Assessment BCBAs were provided with a 2-hr training that utilized a behavioral skills training format (Parsons et al., 2012) on how to administer the POP-C tool, including how to operationally define and score specific symptoms, behaviors, and other patient factors with accuracy. Initial applications of the POP-C were completed with support from the first author to ensure integrity. Additional informal integrity checks were completed at least once monthly throughout the 12-month pilot period.

Procedure

Upon completion of initial registration paperwork and after informed consent for assessment had been obtained, initial assessments were scheduled and completed by the Assessment BCBAs. For patients in the pilot group, the Assessment BCBAs were instructed to complete the POP-C electronic form within 24 hr of completing the initial assessment process. Based on feedback from the Assessment BCBAs, the average administration time was 15 min. Upon form submission, POP-C total scores were automatically generated and routed to the first author. It is notable that the Assessment BCBAs completing the forms were kept blind from the score and corresponding dosage and setting recommendations generated by the POP-C. They were instructed to make treatment recommendations as they normally would, based on the information gathered during the assessment, and to send their recommendations to the first author. POP-C scores, dosage, and treatment setting recommendations were collected for study purposes only; recommendations generated by the tool were not utilized to make actual patient dosage recommendations and the use of the tool did not change the trajectory of patient treatment in any way for the pilot group.

Data Analysis and Results

Interrater reliability was measured by comparing the dosage recommendation produced by the POP-C with the independent dosage recommendation made by the Assessment BCBA. Interrater reliability is a concept important to establishing measurement quality. Reliability is the extent to which independent data collectors assign the same score to the same variable (McHugh, 2012). It is essential that behavior analysts select assessments for use with patients that have sufficient evidence regarding validity and reliability (Padilla et al., 2023). Interrater reliability was calculated using a percent agreement method. An agreement was scored if the Assessment BCBA’s dosage recommendation fell within the recommended dosage range produced by the POP-C. The total number of agreements were divided by the sum of agreements and disagreements and multiplied by 100 to produce a percentage. Interrater reliability for the POP-C was acceptable at 88%, with a total of nine disagreements across the 77 patients included in the pilot. In four cases, a lower dosage was recommended by the Assessment BCBA than the recommendation produced by the POP-C and in five cases, a higher dosage was recommended by the Assessment BCBA than the recommendation produced by the POP-C. In addition, we assessed the relationship between the POP-C total score and the total number of hours authorized for treatment as a preliminary measure of predictive validity—that is, the degree to which the POP-C score was predictive of what was ultimately authorized by the patient’s funding source for treatment. Results of the Spearman correlation indicated that there was a statistically significant positive association between the POP-C score and the total number of direct treatment hours authorized, r(48) = .44, p = .002.

We also evaluated the convergent and discriminant validity of the POP-C to provide preliminary information regarding the measure’s overall construct validity. Construct validity refers to the extent to which inferences about constructs are supported by outside evidence and are aligned with the intended use of an assessment (Crocker & Algina, 1986, p. 4). Because “constructs” are hypothetical in nature and not directly observable, it is common to examine correlations between a new measure and older established measures that are hypothesized to measure similar constructs, referred to as convergent validity. In contrast, discriminant validity refers to a lack of correlation between assessments that are not hypothesized to measure the same construct (Malkin et al., 2017).

All statistical analyses were performed using IBM SPSS Statistics (v. 28). A Kolmogorov-Smirnov test was used to test for normality, indicating that POP-C scores did not follow a normal distribution, D(77) = 0.114, p = 0.014. Therefore, Spearman’s rho correlation coefficients were computed to assess the relationship between the POP-C total score, the Vineland-3 composite and all subdomains, and age for all participants. Correlations and significance values were also calculated for the POP-C total score and the VB-MAPP and EFL for a subset of participants for whom these specific criterion-referenced assessments were selected and administered in full. Strong and statistically significant negative correlations were found between the POP-C and Vineland-3 Composite, r(76) = -.65, p = < .001 as well as all Vineland-3 subdomain scores including Communication, r(76) = -.61, p = < .001, Daily Living, r(76) = -.66, p = < .001, Socialization, r(76) = -.55, p = < .001, and Motor Skills, r(76) = -.54, p = < .001. Strong and statistically significant negative correlations were also found between the POP-C and the VB-MAPP, r(38) = -.53, p = .004, and the EFL, r(20) = -.85, p = < .001. That is, as patients’ scores decreased on these measures (indicating greater skill deficits or lower level of independent functioning), patients’ scores increased on the POP-C (indicating a higher treatment dosage recommendation). Figures 1, 2 and 3 provide scatterplots depicting the correlations between the POP-C total score and Vineland-3 Composite score, VB-MAPP score, and EFL score, respectively. A statistically significant correlation was not found between the POP-C total score and age, r(75) = .21, p = 066. These data are summarized in Table 5.

Fig. 1.

POP-C correlation with Vineland-3 composite score

Fig. 2.

POP-C correlation with VB-MAPP score

Fig. 3.

POP-C correlation with EFL score

Table 5.

Relationship between POP-C total score and Vineland-3, VB-MAPP, EFL, and age

	POP-C Total Score		N	Mean Age (years)	Males	Females
	r value	p value
Age	-.21	.066	77	6	58	19
Vineland Composite	-.65	< .001	77	6	58	19
Vineland Communication	-.61	< .001	77	6	58	19
Vineland Daily Living	-.66	< .001	77	6	58	19
Vineland Socialization	-.55	< .001	77	6	58	19
Vineland Motor Skills	-54	< .001	77	6	58	19
VB-MAPP	-.45	.004	39	3.8	32	7
EFL	-.74	< .001	21	7.7	15	7

Note. Bold font indicates statistical significance

The range of scores possible on the POP-C was 0–200. Scores for the pilot group ranged from 12 to 178 (M = 107.92, SD = 41). Figure 4 shows the distribution of scores across participants. The distribution was roughly unimodal, with no outliers, and skewed left indicating a greater number of high scores than low scores on the POP-C. However, ceiling effects were not observed in the current sample. In particular, for the 77 patients assessed, 47% were recommended high intensity services of 30–40 hr per week, 20% were recommended to receive medium intensity services of 20–30 hr per week, and 33% were recommended low intensity services of up to 20 hr per week, social skills groups, or consultation-based services. Table 6. shows the characteristics of patients whose POP-C scores placed them in each of the three treatment intensity categories.

Fig. 4.

Distribution of scores on the POP-C. Note. Closed bars represent POP-C scores corresponding to a low-dose treatment recommendation, striped bars represent scores corresponding to a moderate-dose treatment recommendation, and open bars represent scores corresponding to a high-dose treatment recommendation

Table 6.

Total number and percentage of patient characteristics observed within each treatment intensity category

Characteristics	High Intensity (n = 36)	Moderate Intensity (n = 15)	Low Intensity (n = 26)
Age at Assessment
2–4	15 (41.7%)	8 (53.3%)	5 (19.2%)
5–7	14 (38.9%)	7 (46.7)	10 (38.5%)
8–10	3 (8.3%)	0 (0%)	7 (26.9%)
11–13	3 (8.3%)	0 (0%)	4 (15.4%)
14–19	0 (0%)	0 (0%)	0 (0%)
20	1 (2.8%)	0 (0%)	0 (0%)
Gender
Female	6 (16.7%)	2 (13.3%)	11 (42.3%)
Male	30 (83.3%)	13 (86.7%)	15 (57.7%)
Comorbid Diagnoses
Cognitive Impairment or Developmental Disability (F70–F79, F88)	9 (25.0%)	3 (20.0%)	5 (19.2%)
ADHD (F90)	4 (11.1%)	3 (20.0%)	5 (19.2%)
Anxiety (F41)	7 (19.4%)	0 (0%)	8 (30.8%)
Speech and Language Disorder (F80)	13 (36.1%)	1 (6.7%)	9 (34.6%)
Other*	12 (33.3%)	2 (13.3%)	7 (26.9%)
Skills Assessment Used
VB-MAPP	20 (55.6%)	12 (80.0%)	7 (26.9%)
EFL	13 (36.1%)	3 (20.0%)	7 (26.9%)
Other**	3 (8.3%)	0 (0%)	12 (46.2%)

*Examples include F34.81, R62.5, etc.; **Examples include the Assessment of Basic Language and Learning Skills, Revised (ABLLS-R, Partington, 2010), the Assessment of Functional Living Skills (AFLS; Partington & Mueller, 2013), PEAK Comprehensive Assessment (PCA; Dixon, 2019)

Discussion

The current study describes the development of the POP-C, a tool for training BCBAs to provide ABA treatment dosage recommendations based on patient symptom severity and treatment needs and provides preliminary evidence of the reliability and validity of the measure. In a pilot sample of 77 patients, total score on the POP-C was correlated with both standardized and criterion-referenced measures commonly used by behavior analysts to make medical necessity determinations and dosage recommendations (e.g., Vineland-3, VBMAPP, and EFL). These correlations are encouraging given that generally, patients with lower adaptive functioning and communication skills at baseline necessitate higher dosages of treatment hours to make optimal progress across skill domains. However, they should be interpreted with caution as norm- and criterion-referenced assessments alone are incomplete for determining dosage.

In contrast, no correlation was found between patients’ chronological age and POP-C total score. This provides initial evidence of discriminant validity for the POP-C, as the tool was not designed to provide differentiated recommendations based on patient age. In addition, we also found acceptable interrater reliability (88%) when comparing recommendations produced by the tool with recommendations made by highly trained assessment specialists and demonstrated that the POP-C total score was also significantly correlated with the total direct treatment hours authorized by the patients’ funding source.

Finally, the POP-C yielded a satisfactory distribution of scores across patients in the pilot group (i.e., 47% were recommended high intensity services of 30–40 hr per week, 20% were recommended to receive medium intensity services of 20–30 hr per week, and 33% were recommended low intensity services of up to 20 hr per week, social skills groups, or consultation-based services). This type of spread is important, as the goal of the POP-C is to make individualized recommendations for therapy hours based on a patient’s symptom severity level. As the research on treatment outcomes continues to grow, the idea of 40 hr per week as a blanket recommendation has come under increased scrutiny (see Ostrovsky et al., 2022). The POP-C may be a useful tool for practitioners in how to make appropriately titrated dosage recommendations for their patients based on individual skill levels, characteristics, and symptom severity at baseline.

At present, there is little, if any, structured guidance available in the behavioral literature on how best to synthesize and contextualize individual patient factors when making medical necessity determinations. In the absence of standardized tools and support, BCBAs must rely entirely on their clinical judgment to determine how many hours of therapy are medically necessary for a patient to make optimal progress in treatment. This presents potential risks to quality and consistency, as one’s clinical judgment or competence is often dependent on the extent of their clinical training and practical experience providing services to patients. Just as patients receiving ABA treatment require specific instructional arrangements to be presented over a period of time to reach competency with new skills, practitioners also require time and repeated experiential learning opportunities to develop their skills, becoming more competent practitioners the longer they practice. However, given the current demand for ABA services, we cannot simply wait for newly credentialed BCBAs to accrue years of experience before they are able to fulfill key responsibilities of their role. This is where the use of standardized decision-making tools can be of assistance to novice clinicians. The POP-C adds to a growing number of existing tools and decision-making models available to BCBAs to support the consistent application of high-quality, evidence-based practice in ABA treatment regardless of experience (Marya et al., 2022).

Moreover, the data presented in the current study represent an important and necessary step forward in the tool-development process by establishing preliminary reliability and validity of the POP-C. The importance of utilizing measures that have strong psychometric properties is historically under-represented in behavior analytic literature (Padilla et al., 2023), likely due to the reliance on individualized goals and behavior change measures that do not lend themselves to the standardization efforts typically seen with measures utilized in psychological practice. For example, the VB-MAPP, which is a widely used criterion-referenced assessment of verbal behavior milestones often implemented during the initial assessment process to establish a patient's baseline skill level, has limited published data on reliability or validity (Dixon et al., 2015; Montallana et al., 2019). As the field moves toward broader adoption of standardized measures for evaluating patient outcomes, and stricter requirements for evidence-based medical necessity determinations, it is imperative that clinicians use measures with established psychometric properties to provide ethical and data-informed care. Understanding measurement quality for the assessments used by behavior analysts is not only necessary, but a requirement of our ethical practice.

There are several limitations of the current study that warrant discussion. First, the development of a tool of this nature is inherently challenging. For example, it is unclear whether a patient for whom a recommendation of 20 hr per week of ABA therapy is made, would make even greater progress (or perhaps improve at a quicker rate) if provided with 40 hr instead. This also makes it difficult to establish clear construct validity for the POP-C, as the notion of treatment dosage as a construct is a new concept that has not been defined in previous literature. Although the assessments used to test the validity of the POP-C do not fully capture all the information needed to make a dosage recommendation (and therefore do not comprehensively reflect the construct of interest), they do represent the measures currently used in practice to inform treatment intensity and goals and therefore, the reported correlations provide very preliminary evidence of validity.

Similar challenges are encountered when attempting to determine interrater reliability. Although the BCBAs in the current study were able to leverage their unique training, skill set, and expertise to make dosage recommendations which were later compared to those produced by the POP-C, their clinical judgment is still subject to bias. Furthermore, our definition of “agreement” (i.e., the BCBA’s recommendation falling within the range recommended by the POP-C) provides ample leeway on differences between recommendations. Although the tool was not designed with the purpose of providing an exact dosage recommendation—such that clinical judgment in tailoring the dosage recommendation to the individual patient is removed from the process—the relatively broad ranges represent a limitation of the current reliability data. And finally, although the BCBAs were kept blind to the POP-C score and corresponding recommendation, we cannot say that their recommendations were entirely independent as exposure to the POP-C tool itself may have influenced their subsequent recommendation. Nevertheless, the POP-C may offer some advantages over sole reliance on clinical judgment in that it requires the collection of primarily objective data (not opinions) across a comprehensive set of variables relevant to medical necessity, which may help to mitigate risk of bias.

Finally, it is important to highlight that the POP-C was developed and tested only during the initial assessment process. Despite the utility of the POP-C as a guide for determining initial treatment dosage, it is critical that clinicians continuously engage in progress monitoring and attend to their patient’s response to treatment. For example, much as it is likely true (to an extent) that the more severe one’s symptoms and impairment are, the more hours of intervention they would benefit from, research is limited on outcomes of ABA intervention for individuals with differing severities, etiologies, and symptom presentations of intellectual disability (Ho et al., 2021). The current iteration of the POP-C is not designed to account for the likelihood of a patient being a “nonresponder” to ABA treatment. Furthermore, treatment outcome may be significantly affected by the quality of intervention, irrespective of dosage (e.g., rate of learning opportunities presented, appropriateness of selected procedures, treatment fidelity). Therefore, it is important for clinicians to continuously evaluate the overall quality of treatment hours and consider the patient’s response to treatment in subsequent dosage recommendations.

Due to the lack of empirical research testing tools or decision-making frameworks for making medical necessity determinations, there are many areas for future research to explore. First, more rigorous testing of the POP-C is needed to determine whether it produces recommendations that consistently lead to optimized patient-centered outcomes. Future explorations might examine treatment outcomes for patients from the pilot group who enrolled in therapy at a dosage that was matched by both the Assessment BCBA and the POP-C. Dose-response curve analyses might provide a stronger test of validity if it were to demonstrate that the tool consistently determines the dose of ABA intervention necessary to produce positive outcomes for patients (Calabrese, 2005). Longitudinal studies could further explore patient outcomes between a group for which the POP-C was used to guide dosage recommendation and a group for which the tool was not used. These data may allow for stronger tests of predictive validity by demonstrating that scores from the POP-C accurately predict changes in assessment scores as a function of receiving the recommended hours of ABA intervention.

In addition, collaboration with outside providers is necessary to evaluate if similar outcomes are achieved outside the participating organization. This research might also help to refine the recommendations generated by the tool into more precise treatment-hour ranges (e.g., 5-hr or even 1-hr increments). Independent investigations are particularly important given that the POP-C was designed to support the service options offered by the participating provider organization, which may not reflect the entire range of services available at other agencies, potentially limiting the generality or utility of the tool in other settings.

Evaluating the use of the tool with novice BCBAs is yet another important area for future research. Novice BCBAs were not recruited to test the tool in the current study because within the participating organization, only designated “Assessment BCBAs” conducted initial assessments and made initial dosage recommendations. However, this represents a limitation of the current study that should be addressed in future research by testing the tool with novice BCBAs and comparing their recommendations with those of more experienced BCBAs. Future research might also examine the social validity of the POP-C from the vantage point of multiple stakeholders (e.g., BCBAs, caregivers, payors) and the degree to which administration of the tool and its recommendations are acceptable and appropriate from their perspectives. Finally, as a further test of the utility of the POP-C, it may be interesting to examine how the tool could be used during peer reviews with insurance providers to aid in the justification of medical necessity determinations and dosage recommendations, and whether it allows for more efficient peer review calls or a higher rate of authorization approvals.

ABA treatment dosage may be one of the most important dimensions of intervention outcome. It is surprising that this aspect of practice has not been well-defined, nor have methods been developed for appropriately titrating dosage such that clinicians are able to personalize recommendations so that patients can obtain the intended benefits of treatment while minimizing side effects. The present article contributes to the literature by introducing the POP-C, a tool for determining ABA treatment dosage based on research-supported factors relevant to treatment outcomes. Given the proliferation of newly credentialed practitioners serving individuals with ASD and the current reliance on clinical judgment in making medical necessity determinations, the POP-C provides a much-needed resource for practitioners seeking more structured guidance in making treatment recommendations. The present study also provides data supporting the reliability and validity of the tool and data demonstrating its sensitivity in producing recommendations that span a range of treatment dosages. Considering the paucity of research on this critical aspect of ABA service delivery, the POP-C tool and the supporting data provide an important step forward in ensuring consistently appropriate ABA treatment recommendations are made, irrespective of practitioner tenure.

Supplementary Information

Below is the link to the electronic supplementary material.

Data Availability

The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethical Approval

This study was conducted retrospectively from data obtained for clinical purposes during routine care. All procedures were performed in accordance with the ethical standards of the institution and national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Conflict of Interest

All authors are employed by the organization where the current study took place.