Concurrent Validity and Reliability for the Communication Complexity Scale

Abstract

Purpose

The Communication Complexity Scale (CCS; Brady et al., 2012) was created to fill a void in measures of expressive communication skills in individuals who communicate primarily with presymbolic or early symbolic means. CCS scores reflect expressive communication observed during interactive communication contexts.

Method

Two studies were completed to examine the reliability and validity of the revised CCS scores. Participants in both studies had minimal verbal skills (i.e., produced less than 20 functional words). Study 1 examined interobserver agreement, test–retest reliability, and concurrent validity for 239 participants with intellectual disabilities between the ages of 3–66 years, assessed with the protocol developed at the University of Kansas (KU CCS). CCS scores were compared with scores from the Vineland Adaptive Behavior Scales–Second Edition (Sparrow, Cicchetti, & Balla, 2005) and the Communication Matrix (Rowland & Fried-Oken, 2010). Study 2 examined the reliability and concurrent validity for CCS scores derived from 110 children (chronological age = 3–9) with autism from diverse backgrounds. These children were assessed with the Early Social Communication Scale (Mundy et al., 2003). CCS scores were compared with rates of communication derived from the Early Social Communication Scale.

Results

CCS scores were moderately correlated with scores from existing measures of early communication. In addition, CCS scores from different raters were reliable, and test–retest scores were highly similar.

Conclusions

These findings support the validity and reliability of the CCS when used with individuals across a wide range of ages and with various types of disabilities. The CCS can be used in research and clinical practice to describe extant communication levels in individuals with minimal verbal skills.

The Communication Complexity Scale (CCS; Brady et al., 2012) is a scale that describes expressive communication in individuals who communicate primarily with nonsymbolic means. It was developed to address a void in existing instruments available for measuring early communication behaviors, such as gestures, vocalizations, and eye gaze, and it remains one of the few measures available that can show variation within prelinguistic communication. The focus of this particular study is on demonstrating that results from the CCS are (a) reliable when applied to assessments of individuals with intellectual disabilities (IDs) across a wide range of ages, (b) reliable when applied to the same individual in a relatively short period of time, and (c) correlated with existing measures of early communication. In addition, we evaluated the reliability and validity of CCS scores derived from two different interactive communication assessments—the CCS Scripted Assessment and the Early Social Communication Scales (Mundy et al., 2003).

The CCS describes observed communication behaviors according to a developmental continuum of prelinguistic and early linguistic communicative behaviors. The scale encompasses three levels of early communication—preintentional, intentional nonsymbolic, and beginning symbolic communication. Scores of 1–5 are used for preintentional communication, such as demonstrating basic awareness to environmental changes, or a single focus of attention on an object or person. The important transition to intentional communicative behavior occurs at a score of 6. Scores of 6–10 indicate intentional nonsymbolic communication, where an individual uses gestures and/or vocalizations to demonstrate coordinated attention between objects and people and an expectation for a partner's response. Scores of 11 and 12 correspond to symbolic communication, such as purposeful use of words, signs, or symbols. Within each category, higher scores reflect increased complexity on the basis of the addition of potentially communicative behaviors (PCBs; Keen, Woodyatt, & Sigafoos, 2002; Sigafoos et al., 2000). PCBs include vocalizations and gestures that may be used in communication. For example, the difference between a score of “6” and a score of “9” on the CCS reflects the addition of PCBs. A 6 indicates triadic eye gaze, for example, shifting eye gaze between a referent and communication partner and back again, whereas a 9 indicates a triadic eye gaze with the addition of another behavior, such as a gesture or vocalization. Similar differences are reflected throughout the 12-point scale (see Table 1).

Table 1.

CCS scores.

No.	Definition	Communication level
0	No response
1	Alerting – a change in behavior, or stops doing a behavior	Preintentional
2	Single orientation only – on an object, event, or person; can be communicated through vision, body orientation, or other means	Preintentional
3	Single orientation only and a PCB1	Preintentional
4	Single orientation only + more than 1 PCB	Preintentional
5	Dual orientation – shift in focus between a person and an object, between a person and an event using vision, body orientation, and so on (without PCB)	Preintentional
6	Triadic orientation (e.g., eye gaze or touch from object to person and back)	Intentional nonsymbolic
7	Dual orientation + 1 PCB (e.g., dual focus + gesture)	Intentional nonsymbolic
8	Dual orientation + 2 or more PCB (e.g., dual focus + gesture + vocalization, and switch closure)	Intentional nonsymbolic
9	Triadic orientation + 1 PCB (e.g., triadic + vocalization)	Intentional nonsymbolic
10	Triadic orientation plus more than 1 PCB (e.g., triadic plus vocalization and differential switch closure)	Intentional nonsymbolic
11	One-word verbalization, sign, or AAC symbol selection	Intentional symbolic
12	Multi-word verbalization, sign, or AAC symbol selection	Intentional symbolic

Note. Copyright © 2017 by The University of Kansas. Printed with permission. CCS = Communication Complexity Scale; PCB = potentially communicative behavior; AAC = augmentative and alternative communication.

The CCS scale has been revised since it was first published (Brady et al., 2012), and the revised version is the subject of the current study. The revision differs from the original in the following ways. The original CCS included a score for scanning between objects, and the current version does not include this behavior. We also renumbered the scale. Triadic eye gaze changed from a score of 7a on the previous version to a score of 6 on the current version. Dual orientation plus one or more PCBs changed from a score of 7b to 7. Finally, we separated the previous score for dual orientation plus one or more PCBs into two scores—one for dual orientation plus one PCB and one for dual orientation plus more than one PCB.

The current version of the CCS also describes communication functions for intentional communication acts, whereas the previous version did not include information about communication functions. This is an important addition, given that differences in the functions of early communication acts have been associated with particular types of IDs (Camaioni, Perucchini, Muratori, Parrinii, & Cesari, 2003; Legerstee & Fisher, 2008; Wetherby, Watt, Morgan, & Shumway, 2007), as well as severity of disorder (McLean, McLean, Brady, & Etter, 1991; Ogletree, Wetherby, & Westling, 1992).

The CCS is a scale, and it can be applied to communication behaviors observed in response to various interactive communication contexts. Its intended use is for research and practice aimed at more precise measurement of early communication. For example, scores from the CCS could impact identification of treatment goals, measurement of responses to various interventions, and identification of phenotypic characteristics of a particular disorder. However, it is first necessary to demonstrate that the scale has acceptable psychometric properties. Toward this end, we investigated the reliability and validity of the scale when used with two different assessment protocols.

For Study 1, CCS scores were derived from activities administered in two versions of interactive assessment protocols designed at The University of Kansas. One protocol was designed for younger participants (younger than 16 years) and, the other, for older participants (16 years and older). We designed the activities in each protocol to be age appropriate for their respective groups. In Study 2, scores were derived from the Early Social Communication Scales (Mundy et al., 2003). Each of these assessment protocols is described further in the Methods section. Importantly, each protocol provides scripted opportunities for initiated communication by early communicators. For example, one activity provided a remote control car for the participant and another for the experimenter. The participant's remote control was sabotaged so that it did not work, setting up an opportunity for the participant to protest or request help using gestures, vocalizations, or symbolic communication.

In a previous study, we showed that the CCS assessment protocol developed at the University of Kansas (KU CCS) could be administered with fidelity and scored reliably with young children (Brady et al., 2012). We also showed that the scores from the CCS were comparable to another widely used measure of early communication, the Communication Matrix (Rowland & Fried-Oken, 2010). Specifically, individuals were categorized similarly on both the CCS and the Communication Matrix as preintentional, intentional but presymbolic, or early symbolic communicators. Perhaps, most importantly, scores from the CCS demonstrated a range of expressive communication abilities in individuals with severe disabilities who showed floor effects on standardized measures, such as the Preschool Language Scale (Zimmerman, Steiner, & Evatt Pond, 2003) or the Peabody Picture Vocabulary Test (PPVT; Dunn & Dunn, 2007).

In the current studies, we further evaluated the utility of the revised CCS by examining results from a larger and more age-diverse sample (3–66 years) and by comparing scores derived from three different assessment protocols (KU CCS child protocol, KU CCS adult protocol, Early Social Communication Scale [ESCS; Mundy et al., 2003]) to additional measures. In Study 1, we compared CCS scores to the widely used and standardized Vineland Adaptive Behavior Scales–Second Edition (Vineland II; Sparrow, Cicchetti, & Balla, 2005) and scores from the Communication Matrix. In Study 2, we compared CCS scores to communication rate data from an assessment of early communication that is commonly used in experimental studies of children with minimal verbal skills—the ESCS (Mundy et al., 2003).

While we hypothesized similar scores between the CCS and comparison measures, we also expected the correlations to be moderate because the CCS uniquely reflects the complexity of prelinguistic communication, and it is a direct observation measure—that is, scores are derived by direct observation of a participant rather than asking informants about their communication. In contrast, the Vineland II communication scale contains some prelinguistic items, but it does not specifically measure differences in prelinguistic communication complexity, and it relies on caregiver report rather than on direct observation. The Communication Matrix measures differences in prelinguistic communication complexity, but it also relies on caregiver report. Although the ESCS is a direct observation measure, typical scores are frequencies of communication behaviors rather than complexity of prelinguistic communicative behaviors.

The following research questions were addressed in Study 1:

1. How reliable are scores from the CCS, demonstrated by interobserver agreement between scores derived from a wide age range of 3–66 years?

2. How reliable are scores for the same person within a short period of time (test–retest)?

3. How do scores from the CCS compare to scores from two established early communication measures: the Vineland II Expressive Communication scale and the Communication Matrix?

The following research questions were addressed in Study 2:

1. How reliable are CCS scores derived from the ESCS assessment protocol administered to young children with autism spectrum disorder (ASD) from diverse backgrounds?

2. How do CCS scores compare to frequency scores derived from the ESCS in young children with ASD from diverse backgrounds?

Study 1

Method

Ethics, Consent, and Permissions

All methods, including participant-informed consent procedures, were approved by the Human Subjects committee at The University of Kansas–Lawrence under approval number 20790. Consent was obtained from parents or guardians, and assent was obtained from each participant.

Participants

Participants had minimal verbal skills defined as less than 20 functional words and/or signs. All participants were able to hold their head steady and upright, sit with support, and physically interact with the toys provided during the assessment. Hearing and vision were within normal limits or corrected to be within normal limits for each participant. School records and parental report were relied upon to confirm participants' number of words, motor abilities, vision, and hearing. However, if a participant produced more than 20 different words (in any modality) during our assessment, they were not included in our study.

Two hundred and thirty-nine individuals from metropolitan and rural areas in the midwest were recruited by directly contacting school districts and facilities that provide services to adult clients, posting on the CCS website, and posting on local websites for families of children with Down syndrome and families of children with ASD. English was the primary language spoken in all participant homes. The age range of the participants was 3–66 years, and 38% were female (62% were male). Twenty-seven participants were African American, four were American Indian/Alaska natives, seven were Asian, one was Hawaiian/Pacific Islander, seven did not report their race, 21 reported more than one race, and the remaining 172 were White. Twenty-six participants were Hispanic, with half identifying as White and half as more than one race. Thus, this sample was diverse in terms of age but not very diverse in terms of ethnicity.

Fifty-five percent of participants were between 3 and 16 years of age, and we refer to these as the child participants. Nearly all of the child participants lived at home with their parents, whereas many of the adults lived in homes with caregivers other than their parents. The caregiver report measures used in this study (Communication Matrix and Vineland II) were obtained from parents (n = 150) or caregivers (n = 89) who lived with or frequently interacted with the participants at the time they provided information.

Measures

The KU CCS Assessment Protocol, Communication Matrix, and Vineland II were administered to each participant in Study 1 or their caregiver within 2 weeks of each other. The following sections describe how scores were obtained from each of these measures.

CCS

The CCS (Brady et al., 2012) describes early communication skills of individuals with severe intellectual and/or developmental disabilities on the basis of a 12-point scale. For Study 1, the CCS was used with scripted videotaped assessments of communication opportunities created by Brady et al. (referred to here as the KU CCS protocols). Research assistants administered a series of 12 play-based activities that were designed to elicit communication for behavior regulation (BR) or joint attention (JA). Administration took approximately 20–35 min for each assessment. We administered two slightly different versions of the assessment activities to reflect preferences and age appropriateness. Items were similar across the two versions. For example, Version A presented a child's book with altered pages, and Version B presented a magazine with altered pages. Both tasks were designed to see if the participant would notice and point out the altered pages (e.g., some were upside down, scribbled on, or torn).

Upon completion of scripted assessments, research assistants watched assessment video recordings and assigned a code for the highest communicative act for each activity. Scores of 0 and 1 describe no response and alerting behavior, respectively. Scores of 2–5 describe preintentional communicative acts, such as orienting to an activity or person and nonword vocalizations. Scores of 6–10 describe intentional nonsymbolic communicative acts, such as triadic eye gaze and intentional gestures, such as pointing with accompanying eye gaze shifts. Scores of 11 and 12 describe intentional symbolic communication, such as words spoken verbally or produced using sign or other augmentative communication. Table 1 presents the entire scale along with abbreviated definitions.

Scores of 6 or higher convey intentional communication, and for those scores, it is possible to further assign a communicative function. On the basis of past research (Bates, Benigni, Bretherton, Camaioni, & Volterra, 1979; Brady, Marquis, Fleming, & McLean, 2004), we differentiated between the functions of BR, JA, or response to question. BR includes requests and protests. An example of BR is handing a container that is difficult to open to a communication partner and waiting for the partner to open it. JA refers to social commenting, such as pointing at an unusual toy to share with a communication partner. Response to question refers to responses to questions issued by the examiner, such as “Do you need help?” It should be noted, however, that questions are discouraged during administration of the KU CCS protocol.

Optimal Scores

After each activity in the videotaped protocol was scored, we determined the overall optimal score by computing the average of the three highest scores from the 12 activities. For example, if the three highest scores were an 11 for Activity 1, a 10 for Activity 7, and another 10 for Activity 8, the participant's optimal score would be 10.33. Optimal scores were determined because the interaction is relatively short (approximately 30 min), and we have found substantial intrasubject variability in responses to different items on the basis of individual preferences. Thus, cumulative or mean scores could reflect interest in activities rather than in communication skills.

Further, we determined the optimal score within functions (i.e., optimal BR and optimal JA) by averaging the highest three scores produced for BR and JA, respectively. We computed scores within function because reporting differences across functions may be important for research and clinical practice with particular groups. For example, individuals with severe IDs have been reported to have more impaired communication in JA than in BR (Brady, McLean, McLean, & Johnston, 1995). Recall that only scores of 6 and above reflected intentional communication and, hence, were scored for function. If a participant did not have three scores of 6 or higher for a particular function, they did not receive an optimal function score. Most participants ended up with three scores from the CCS: overall optimal score, optimal BR score, and optimal JA score.

Communication Matrix

The Communication Matrix (Rowland & Fried-Oken, 2010) is a communication skills assessment that describes seven levels of expressive communication, from early preintentional behaviors to formal language systems. It was designed primarily for use with children with severe or multiple disabilities, including cognitive, sensory, and motor impairments (Rowland & Fried-Oken, 2010). Caregivers answer questions about the individual's ability to communicate four different functions: rejecting items and activities, obtaining items and activities, engaging in social interaction, and seeking and providing information.

For this study, research assistants administered the Communication Matrix to participants' caregivers by interview, either in person or over the phone, and entered responses onto an online portal. When caregivers indicated that an individual communicated a specific function, they also specified what behavior the individual used to express that message. For example, did the participant use body movements, vocal sounds, facial expressions, visual behavior, simple gestures, conventional gestures/vocalizations, concrete symbols, abstract symbols, or language to express the designated message? Each behavior that the individual used was then labeled as emerging (if it rarely occurs) or mastered (if it frequently occurs). We used the highest level emerging scores in our analyses because these scores seemed most analogous to the optimal scores derived from the CCS protocol during a relatively short observation.

Vineland II

The Vineland II (Sparrow et al., 2005) was administered through an interview conducted over the telephone. The Vineland II can be used with persons across a wide range of disability, including individuals with minimal verbal skills, and has been normed extensively across seven different disability groups. It is currently one of the most widely used diagnostic evaluation and program planning measures for people with ID. Composite raw scores for the Expressive Communication subtest were used in the current study.

Results

CCS Interrater Reliability

Eighty-one assessments (24%) were independently scored by two trained raters and compared for reliability calculations. Reliability assessments were selected using a computerized random number generator. The overall kappa score across all the scripted opportunities presented was .83. Eighty-five percent of the scores were identical, and 93% were within 1 point of each other.

In addition to examining scores for each opportunity separately, we were also interested in rater agreement at the participant level. We assessed interrater reliability for the most common outcome score, the overall optimal score. Of the 81 videos coded by two independent observers, 54 (66.67%) had identical overall optimal scores from both independent raters, and 76 (93.83%) had scores that were within 1 point of each other. Spearman rho for scores from the primary and reliability optimal overall scores was .984.

An important consideration is the extent to which the CCS scores are reliable across three broad categories of communication levels. Recall that CCS scores of 0–12 map onto three broader communication levels: preintentional, intentional nonsymbolic, and symbolic. Participants with optimal scores of 5.32 or lower are considered to be communicating at the preintentional level. Participants with scores in the 5.33–10.32 range are considered to be communicating at the intentional nonsymbolic level, and participants with scores of 10.33 and greater are considered to be communicating at the symbolic level. The rationale for these cut-points is that we wanted the broader communication level to reflect the optimal nature of the score (e.g., participants who had an average score above a 5.32 communicated a least once at the intentional nonsymbolic level or higher). When the optimal scores were categorized according to communication level (preintentional, intentional nonsymbolic, and symbolic), the independent raters agreed on the communication level for all but three participants, producing a kappa of .93.

The following score disagreements were noted for the three cases of disagreement across category. In one case, the primary coder's three highest scores were 7, 7, and 3, yielding an optimal score of 5.67, whereas the reliability coder's scores were 7, 3, and 3, yielding an optimal score of 4.33. In the second case, the primary coder's highest three scores were 7, 3, and 7, yielding an optimal score of 5.67, whereas the reliability coder scored a 7, 3, and 4, yielding an optimal score of 4.67. In the third case, the primary scores were 11, 10, and 10 for an optimal score of 10.33, and the reliability scores were 11, 10, and 9, for an optimal score of 10.00. Hence, the optimal scores were close—extremely close in the third case—yet outside of our established categories.

Test–Retest Reliability

Eighteen of the participants, aged 3–55 years were re-administered a scripted interaction protocol about 2 weeks following the initial administration (range = 10–28 days). Retest participants were randomly selected. CCS overall optimal scores on the two occasions were correlated, .84, p < .001. We further examined scores to determine if they remained in the same communication category (preintentional, intentional nonsymbolic, and symbolic). Nine retest participants were adults (mean age = 26 years, range = 16–55), and all nine were classified into the same communication category at both time points. Seven of the nine child participants were also categorized in the same category at both time points. Of the two participants who did not score in the same category on both administrations, one scored a 6.33 (intentional nonsymbolic) at their first assessment and 4.67 (preintentional) at reassessment, and the second scored 8.00 (intentional nonsymbolic) at the first assessment and 10.33 (symbolic) at reassessment. Differences across assessments could be associated with interest levels and motivation.

Concurrent Validity

We assessed concurrent validity by comparing CCS scores to scores from two widely used parent report measures of early communication: the Vineland II (Expressive subscale scores) and the Communication Matrix. We hypothesized moderate correlations between the CCS, Communication Matrix, and Vineland II Expressive subscale scores because the CCS is based on observed responses over a short period of time, whereas both the Vineland II and Communication Matrix report typical behaviors over a wide sample of contexts on the basis of informant report. In addition, we divided the sample into younger (< 16 years) and older (≥ 16 years) age groups in order to determine if there were any differences across ages.

Total Sample Comparison

Correlations between overall optimal scores on the CCS and comparison measures for the entire sample are presented in Table 2. As can be seen, correlations with the Vineland II Expressive scale and the Communication Matrix were significant and moderate, as predicted. In terms of differences across functions, correlations between the caregiver report measures and optimal JA and optimal BR scores were also statistically significant. Optimal JA scores were somewhat more strongly related to the other language outcomes than were the optimal BR scores.

Table 2.

Study 1 results comparing CCS scores to Communication Matrix and Vineland II scores for the total sample and by age group.

CCS scores	Total sample		CA less than 16		CA 16 or greater
	Matrix emerging	Vineland expressive	Matrix emerging	Vineland expressive	Matrix emerging	Vineland expressive
Overall optimal	.35**	.47**	.28**	.42**	.41**	.52**
	n = 224	n = 231	n = 120	n = 123	n = 104	n = 108
Optimal JA	.44**	.50**	.41**	.36**	.46**	.62**
	n = 108	n = 112	n = 61	n = 63	n = 47	n = 49
Optimal BR	.17*	.31**	.14	.20	.24	.42**
	n = 149	n = 153	n = 120	n = 92	n = 59	n = 61

Note. CCS = Communication Complexity Scale; CA = chronological age; JA = joint attention; BR = behavior regulation.

^* p < .05.

^** p < .01.

Comparisons by Age Group

We examined the associations between CCS optimal scores and other communication measures by age group in order to examine the appropriateness of the CCS for older and younger participants. We decided to split the sample into two age groups: Participants who were less than 16 years old were placed in the younger age group, and those 16 years old or over were placed in the older age group. As noted earlier, different materials were used across these two age groups. In addition, most participants younger than 16 years were still receiving active communication programming at school and/or through private clinics according to parent reports. Most participants 16 years old or older were not receiving direct intervention aimed at improving communication. In addition, many of the older participants were living with nonparent caregivers and attending nonschool educational or vocational programs.

In order to better understand the relationships between the three CCS scores (overall optimal, optimal JA, and optimal BR) and other measures of early communication, we assessed the correlations of each CCS score with those scores derived from the Vineland II and the Communication Matrix. CCS overall optimal and CCS optimal JA scores were significantly correlated with the Vineland II Expressive and Communication Matrix scores for both age groups. CCS optimal BR score was significantly correlated with the Vineland II Expressive score for the older group only (see Table 2).

The significant, moderate correlations between CCS scores and Vineland II Expressive scores and Communication Matrix scores provide evidence for concurrent validity. The size of the correlations indicates that the measures are not redundant given that their magnitude is not close to 1.00.

Study 2

Method

Ethics, Consent, and Permissions

All methods, including participant-informed consent procedures, were approved by the Human Subjects committee at the University of California, Los Angeles under numbers 11-000855 and 10-001758. Consent was obtained from parents or guardians, and assent was obtained from each participant.

Participants

Data were obtained from 110 participants who were part of ongoing intervention studies for children with ASD conducted at the University of California, Los Angeles. The age range was 3 to 9 years, and 13% were female (87% were male). Thirty-five were identified as Hispanic. Six participants were African American, one was American Indian/Alaska Native, 26 were Asian, two were Hawaiian/Pacific Islander, eight selected more than one race, and 32 did not report their race. The remaining 35 were White. Thus, we were able to obtain information about using the CCS with children from ethnically diverse backgrounds.

Measures

The ESCS was administered to each participant in Study 2. The following sections describe the ESCS and how scores were obtained from this measure.

ESCS

Participants were assessed with the ESCS, a structured observational assessment designed to measure social communication skills in young children (Mundy et al., 2003). The ESCS has been used to assess change in social communication skills in various studies involving children with developmental disabilities, specifically ASDs (e.g., Kasari, Freeman, & Paparella, 2006; Remington et al., 2007; Yoder & Stone, 2006).

The ESCS creates opportunities for social communication between a child and an adult. Administration lasts approximately 20 min. The procedure involves an examiner presenting a series of standardized toys and objects selected to elicit communicative interactions between the child and the examiner. As is the case with the KU CCS protocols, some of these materials are designed to provide opportunities for BR acts. For the current study, we coded responses to seven BR tasks that closely resembled tasks in the KU CCS protocol: windup toy 1, windup toy 2, windup toy 3, toy in jar 1, toy in jar 2, mechanical toy, and balloon. We also coded responses to six tasks more likely to evoke JA communication: car, hat, comb, ball, glasses, and book. Each task began with the experimenter asking “What do you want to do next?” while motioning toward the entire set of toys that were presented within view but out of reach of the child. Responses to the first seven of these general requests and the specific tasks were coded.

The ESCS was video-taped for later scoring. It was scored twice: first, to obtain rates of initiating joint attention (IJA) and initiating behavior regulation (IBR); and second, to obtain CCS scores. IJA behaviors include child eye contact, pointing, giving, showing, and spoken language used to initiate shared attention to the objects or event. IBR behaviors include eye contact, reaching, giving, pointing, and spoken language used to elicit assistance in obtaining an object or object-related event. Each instance of IJA or IBR was recorded, and these frequency counts were converted to rates of IJA and IBR by dividing the total frequencies of IJA by the assessment duration. The rate measures were used in analyses in order to account for the varying length of assessments.

The ESCS structured observation videos were scored a second time at a later date using the CCS. The coders for the CCS were different from the coders who obtained frequencies of IJA and IBR. Each CCS coder was trained to be reliable with the first author. Once the coders' scores for level (0–12) and function were in agreement for at least 75% tasks for three consecutive videos, the coder was considered to be reliable. As previously described, CCS scores were obtained by first identifying the highest scoring behaviors within each task and, then, averaging the highest three overall scores and the highest three scores within each function (JA and BR). It should be noted that we coded a total of 20 responses in the ESCS as opposed to 12 in the KU CCS protocol. However, the procedure to obtain the optimal score was the same for both assessments. As was the case for Study 1, most participants ended up with three scores: overall optimal score, optimal BR score, and optimal JA score.

Results

Reliability

Seventy-eight assessments were independently scored by two trained raters and compared for reliability calculations. Reliability assessments were selected using a computerized random number generator. The overall kappa score across all the scripted opportunities presented was .66. Seventy percent of the scores were identical, and 85% were within 1 point of each other.

In addition to examining scores for each opportunity separately, we were also interested in rater agreement at the participant level. We assessed interrater reliability for the most common outcome score, the overall optimal score. Of the 78 videos coded by two independent observers, 36 (46%) had identical overall optimal scores from both independent raters, and 69 (88%) had scores that were within 1 point of each other. Spearman rho for scores from the primary and reliability optimal overall scores was .930.

When the optimal scores were categorized according to communication level (preintentional, intentional nonsymbolic, and symbolic), the independent raters agreed on the communication level for all but 10 of the scores, producing a kappa of .750.

Concurrent Validity

Participants in Study 2 were assessed with the ESCS protocol, and videotapes of these assessments were scored in two ways. We compared the frequency scores typically analyzed in research studies (e.g., Kasari, Paparella, Freeman, & Jahromi, 2008) to CCS scores from the exact same observations. Specifically, we compared rates of IJA and rates of IBR derived from the ESCS to CCS scores.

Comparison to ESCS Scores

Significant correlations were obtained between the CCS overall optimal score and rate of IJA per minute (r = .334, p < .01) and rate of IBR per minute (r = .340, p < .01). However, the correlation between the CCS function scores and ESCS scores were nonsignificant. The correlation for optimal BR and ESCS rate of IBR per minute was .07, whereas the correlation between the CCS optimal JA and ESCS rate of IJA was −.11.

Discussion

The current results extend earlier findings on the reliability and concurrent validity of the CCS. We demonstrated that CCS scores obtained from different assessment protocols and from different raters are reliable and that test–retest scores are highly similar. Further, we were able to demonstrate moderate correlations between optimal CCS scores and existing measures of early communication. Importantly, these results were obtained across a wide age range of participants with various types of disabilities, extending the psychometric support for the measure to older participants.

One goal in creating the CCS was to develop a system for quantifying the communication skills for those individuals who scored at or close to the bottom of the score range on other communication measures. The CCS is a strengths-based assessment designed to describe how an individual can communicate rather than focus on deficits. Strength-based assessment is a paradigm that seeks to identify an individual's strengths and skills and, thereby, increase motivation to increase functional skill use across contexts (Climie & Henley, 2016). Identifying the communication skills of someone who does not speak can lead to positive expectations about how the individual may communicate in the future. Further, scores from the CCS can highlight the contexts in which the person communicated, and this information can be used to evaluate communication observed in everyday contexts. For example, we identified communication acts that were complex and intentional in response to scripted opportunities to communicate. Teachers, family members, and other communication partners can use this information to provide similarly motivating opportunities in classrooms and at home.

Focusing on communication strengths can also improve interactions between team members, including family members (Cosden, Koegel, Koegel, Greenwell, & Klein, 2006). Often in our research, we hear families report that they have never received information about how their family member can communication prelinguistically. Such information may provide hope and a more positive outlook about their family member's communication skills.

Traditional, deficits-based assessments of communication in individuals with minimal verbal skills have presented numerous challenges to researchers and clinicians (Kasari, Brady, Lord, & Tager-Flusberg, 2013; Plesa Skwerer, Jordan, Brukilacchio, & Tager-Flusberg, 2016). For example, many individuals with minimal verbal skills are described as “untestable” on assessments, such as the PPVT (Dunn & Dunn, 2007). Of the participants in Study 1, 125 were unable to complete the training portion of the PPVT and did not receive a score on the PPVT. However, these same participants received overall optimal CCS scores ranging from 2 to 12, with a mean of 8.6 and a mode of 11. Forty-four of these 125 participants demonstrated some symbolic communication, despite not being able to complete the training section of the PPVT and scoring more than 2 SDs below the mean on the Expressive subtest of the Vineland II. An additional 71 of the 125 demonstrated intentional nonsymbolic communication. In fact, only 10 of these participants scored in the preintentional range on the CCS. Thus, the CCS may provide richer and more valuable data about communication levels for individuals who are not testable with traditional assessments. These data could contribute to understanding differences in participants who are minimally verbal that are associated with differential successes in particular interventions. Additionally, more fine-grained analyses, such as those available from the CCS, may lead to identification of relative strengths or weaknesses within particular diagnostic groups.

Limitations to the Current Study

Our sample was a convenience sample of interested parties. Results may not be representative of the entire population of people with minimal verbal communication skills. While we recruited without regard to gender, we ended up with a predominance of males, even in excess of the reported male dominance in autism (Werling & Geschwind, 2013). Therefore, oversampling females in future research would ensure that results pertain to females with minimal verbal skills. Additional adaptations would be needed to use our assessment protocol with individuals who have severe motor or sensory impairments; thus, our results are limited to individuals with motor and sensory skills adequate to interact with materials and gesture. The data were collected by trained members of our research team, so additional work needs to be done demonstrating reliability of scoring when assessed and scored by clinicians and other research teams.

One issue we examined more closely concerned the function scores (JA and BR) derived from the CCS and the ESCS in order to determine why they were not more closely associated. Recall that scores of 5 and below on the CCS do not receive function scores because there typically is not enough information to infer a function. However, ESCS coders indicated that some behaviors not meeting our threshold for intentional communication would be assigned a function in the ESCS. For example, reaching toward an object without looking at the examiner would be scored as a communication act with a BR function on the ESCS, whereas this same behavior would be scored as a 3 and would not be assigned a function score on the CCS. This discrepancy would yield different summary scores across the two measures and seems likely to account for the differences.

It should be noted that, in CCS coding, we combined communication acts that were requests with those of protests under the heading of BR. We did this because it is often the case that participants reject one item in order to access a different toy. Thus, a single act can function to reject and request at the same time. In addition, the term behavior regulation has been used to refer to both requesting and rejecting/protesting in developmental literature (Bates et al., 1979) and in studies of early communication in children with autism (Calloway, Myles, & Earles, 1999; Watson, 2013). Although this is in line with past research, the practice prevents separate analyses of requests versus protests, and this could limit some further analyses.

Future Directions

In addition to quantifying current levels of communication, it is important to show that our measures are sensitive to changes over time, particularly those changes associated with treatment effects. Future investigations are needed to compare CCS scores at various points in a treatment protocol to determine if changes are detected with the CCS, even if changes cannot be detected using standard assessments.

In addition, different scores derived from the CCS could be evaluated. For example, modal scores may be useful to report for some purposes, such as comparison of communication in everyday contexts at school, home, and work. Calculating scores on the basis of real-life activities instead of scripted interactions may further provide a different picture about someone's communication that would serve to complement the information derived from scripted assessments.

In order to promote use of the CCS across other research labs and clinical practices (including schools), we are examining different techniques for training others to use the CCS scoring system. Our goals are to ensure that the training is accessible to interested teams and individuals and can be delivered in a timely and cost-effective manner while maintaining accuracy of administration and reliability of scoring. Currently, we are evaluating web-based training systems in terms of usability.

Conclusions

In conclusion, the current study lends further evidence in support of the use of the CCS as a measure of expressive communication in individuals who communicate primarily with prelinguistic means. It describes the current level of communication observed in an interactive assessment, such as the KU CCS assessment protocols or the ESCS. CCS scores may be particularly useful in research studies focused on children with disabilities and minimal verbal skills because there are so few existing measures that have been validated with a large sample. In clinical practice, CCS scores may be useful in describing current levels of communication and monitoring changes. Thus, the CCS fills a need within research and clinical practice.

Funding Statement

This research was supported by grants R01 HD076903 and U54 HD090216 from the National Institutes of Health and R324A160072 from the United States Department of Education, Institute of Educational Science.