When You Can’t R.I.O.T., R.I.O.: Tele-assessment for School Psychologists

Abstract

The acronym R.I.O.T., record review, interview, observation, and test, is a well-known tool for conceptualizing a comprehensive assessment. With COVID-19 and the need to provide school psychological services virtually, it is important to reconsider R.I.O.T. in light of the limitations of virtual assessment. We describe the limitations of virtual assessment and argue that in spite of these barriers, the first three elements of R.I.O.T., record review, interviews, and observations, when used systematically, can provide useful comprehensive assessment data. Specific recommendations are provided for implementing assessment virtually.

The definition of a comprehensive special education evaluation in School Psychology references laws, regulations, ethics, and how our professional literature describes good practice (Hass and Carriere 2014). One common thread is that school psychologists should gather information from a variety of sources using a variety of methods. The acronym R.I.O.T. (Leung 1993) describes one approach to conceptualizing the use of a variety of data gathering methods. These include Reviewing records, Interviewing key informants, Observing students, and administering Tests.

Leung argues that each method of gathering data has limitations and strengths. A critical implication of this is that school psychologists should be cautious of “doubling down” on data collected using one method with data from the same process. Although, in some cases, interviews or observations, for example, more is better, up to a point. For standardized tests and behavior rating scales, psychometric error, differences in the definition of a construct, and differences in the norm group make using multiple tests or scales highly problematic. At the same time, reviewing records, observing children, and interviewing stakeholders, each has both unique contributions but also limitations.

Leung (1993) recommends that we compensate for the limitations of one approach to data gathering by cross-validating the findings with data gathered using other methods. For example, the results from a test might be cross-validated with data from an interview or classroom observation, or both. In this way, we gather comprehensive data and meet the legal mandate to use a variety of evaluation tools and approaches and to not rely on any single source of data when making high-stakes decisions about students (Individuals With Disabilities Education Act 2004).

Tele-assessment in Extraordinary Circumstances

During extraordinary circumstances, such as the current global pandemic, face-to-face interactions with students are challenging, if possible, at all. Currently, schools are closed because of the coronavirus disease 2019 (COVID-19) pandemic and when and how they will re-open is unclear. It is important to remember that although the COVID-19 may be unprecedented in scope for the last several decades, in the twentieth century, schools have closed because of the Spanish Flu and polio. Also, regional closures have occurred for extended periods following Hurricane Katerina.

Currently, conventional face-to-face evaluations are not practical and may not be viable for an indeterminate time. This creates both a legal and practical dilemma for school districts and school psychologists. Although the United States Department of Education’s Office of Civil Rights has advised that evaluations that do not require face-to-face interactions may occur remotely while schools are closed, they have not provided guidance regarding how to decide if a face-to-face assessment is needed or required (O.C.R. 2020).

Because of this lack of guidance and uncertainty as to what restrictions will be in place once schools open, Local Education Agencies (L.E.A.) and individual school psychologists are faced with the prospect of doing some, if not all, of their evaluations, remotely rather than face-to-face. Given that there is little current evidence to support the use of standardized tests remotely (Hiramoto 2020; Farmer et al. 2020). Because of this, it is critical that school psychologists revisit R.I.O.T. and thoughtfully consider what can be done using record reviews, interviews, and observations to gather useful and valid data if remote evaluations are needed.

First, it is crucial to understand that conducting an evaluation remotely is dramatically different than completing an in-person evaluation. However, remote evaluations have occurred with varying levels of success and acceptance. For example, the College Board has administered the S.A.T. and Advanced Placement exams online for some time. The medical field, which had been moving toward utilizing telephone and video technology to conduct office visits (which typically have an element of assessment), has now moved rapidly to holding virtual office appointments because of COVID-19. Even in school psychology before COVID-19, it was not uncommon for school psychologists to conduct aspects of their evaluations remotely. For example, calling parents to interview them or sending parents or teachers questionnaires to complete via an online system. Despite these examples, large-scale use of tele-psychoeducational evaluations with K-12 students was not seriously considered until COVID-19, and few school psychologists have had systematic training in conducting evaluations remotely.

Remote evaluations also require a variety of resources, including appropriate technology in the home (web camera, devices, microphone, reliable WiFi), the availability of adults in the household who can assist with technology, and a sufficiently quiet and private space for conducting assessments. The dilemma school psychologists face is that in many communities, the necessary level of technology or even a quiet area is not available to many families. The U.S. Department of Education had discussed concern for the gap in access to digital resources before COVID-19 (KewalRamani et al. 2018). Now that it appears that these digital resources will be even more critical, the lack of resources exacerbates the inequities we grapple with in providing services to working-class and low-income families (Garcia et al. 2020; Herold 2020). Given the overlap in our society between socioeconomic status and race, we also run the risk of worsening some of the racial disparities found in special education.

Also, in homes where adults will often be observers of different assessment processes, it will be essential to explain to parents, whose inclination is to help their children with tasks that frustrate them or they find difficult, that the most accurate results come when children can respond without adult help or coaching. It might be useful to adapt assessment plans to contain phrasing like the following:

I understand that the procedures used in teleassessment require a quiet location where my child can work with the examiner without interruption from family members. I also understand that the goal of this evaluation is to gather accurate information about children’s independent functioning. This means that the results will be less accurate and potentially invalid if I assist them or comment on their performance while being assessed. (Leung and Hass 2020, p. 15)

Using R.I.O. Rather Than R.I.O.T. in Tele-assessment

In addition to these general issues, this paper discusses how aspects of the R.I.O.T. model, R.I.O., can be applied in conducting remote evaluations. We put forward these ideas as points to consider if it becomes necessary to perform evaluations remotely; however, we would argue that many of these suggestions are also essential to consider when conducting face-to-face evaluations. One caveat is that most of what we have to say is intended for working with students who can verbally communicate and can attend to someone communicating via a computer or other device. Of course, limited verbal abilities and problems with attention can be problematic when conducting face-to-face evaluations, but our ability to manage them and facilitate the assessment process is limited when we are not physically present.

Assessment Using Records (R)

Documentation of a student’s academic, health, and medical history is an essential source of data (Leung 1993; Hass and Carriere 2014). Records such as teacher comments in report cards, grades, group or district educational testing, or health records can be instrumental in establishing patterns of behavior over time that suggest the presence or absence of an academic or social-emotional problem. The usefulness of this historical data when accessed remotely depends on the robustness of a school district’s student information system and whether it can be easily accessed by school psychologists working remotely.

Historical records will not necessarily provide information regarding students’ current functioning, especially if the student has been out of school for weeks or months. School psychologists will have to determine to what extend the available historical information informs the student’s current functioning. At the same time, if records do not provide accurate information about students’ current functioning, they may still offer insight into patterns over time. Using records to understand trends over time is especially crucial in establishing the presence of disabilities such as intellectual disabilities, autism, attention-deficit hyperactivity disorder, and specific learning disabilities that have a developmental feature (Hass 2018).

Assessment Using Interviews (I)

Interviews of stakeholders about their perceptions of a student’s performance were an essential source of data pre-COVID-19. Busse and Beaver describe interviewing as the most common method used to gather information from individuals (Beaver and Busse 2000). Despite their importance, interviews often appear to be less utilized by school psychologists than standardized tests or rating scales (Hass 2018). In the context of tele-assessment, this source of data becomes critical. Potentially, parents, caretakers, teachers, support personnel, and students can be interviewed effectively via video platforms. As stated above, school psychologists currently interview parents over the phone, potentially giving the addition of video some advantage in establishing rapport with parents or children and observing children during the evaluation.

When interacting with family members, offering a clear explanation of the purpose and structure for the interview is essential to establishing a collaborative relationship as well as gathering accurate data. When interviews have a clear purpose, this focuses both the interviewer and the interviewee, making the interview more efficient and useful. When interviews are done via teleconferencing, explaining the goal of an interview might take longer than it does in person, and it will be essential to allow more time for clarification and to answer questions. In some cases, it will be helpful to send a written document ahead of time with topics or questions.

An introduction such as: “I would like to ask you some questions about your child so that I can have a better understanding of how you see things and what your concerns are.” Please do not hesitate to add any thoughts that come to mind even if I didn’t ask about it” can be used to set the stage for the interview and promote cooperation. When interviewing adults or students, it is often useful to start with an open-ended conversational prompt such as “tell me what Johnny is like as a student in our class” or “tell me about your concerns for Johnny at school” rather than a direct question. These open-ended conversation starters allow interviewees to express their perspectives in their own words, which makes it more likely that the assessor understands the interviewee’s concerns. This strategy also demonstrates respect for the interviewee’s viewpoint, which promotes a more collaborative relationship.

An open-ended interview starter can be followed by more specific questions that can focus on topics relevant to the evaluation. These might include:

• Interviewing to gather background or contextual information

• Interviewing for strengths and resources

• Interviewing to establish next steps and goals

• Interviewing to establish an accurate diagnosis or classification (Hass 2018)

If interviewing someone more comfortable being interviewed in a language other than English, an interpreter will typically be needed. Here, the introductory remarks mentioned above become even more important as they should include a description of the translation process. Also, similarly, informing an interpreter about the context of the evaluation and the goals of the interview before the interview are important to gathering quality information and establishing a working relationship with a parent or caretaker (Rhodes et al. 2005). Debriefing afterward to understand the interpreter’s observations and thoughts about the interview is also highly recommended.

Standardized behavior rating scales can be used in conjunction with interviews. Because many publishers of rating scales have systems that allow them to be completed online, this is one aspect of tele-assessment that is potentially the same as typical evaluations. The caveat is the issue discussed above of the availability of adequate technology and technological knowledge on the part of families to complete rating scales online using these systems. Often school psychologists send rating scales home to be completed by hand. This process will also not be significantly different. The recommendation, whether conducting face-to-face evaluations or tele-assessments, is that school psychologists contact the parent via phone or video to explain the purpose and structure of what is being sent home.

Given the constraints of time and energy, especially in tele-assessments, a broad-spectrum rating scale such as the Behavior Assessment for Children-Third Edition (BASC-3; Reynolds and Kamphaus 2015) is likely to be more useful than a narrow domain-specific rating scale. Broad-spectrum rating scales allow the gathering of data about a student’s functioning in several areas relatively quickly. It is important to remember that rating scales are not diagnostic but provide information regarding what areas of functioning need follow-up.

Rating scales assess children’s perceptions in a specific social context, and in the context of homeschooling, these become our de facto observation data. Although there are many things about the current situation that seems atypical, this does not necessarily make rating scales invalid. Instead, it makes it even more critical that the results be interpreted in context. It also means that, in the spirit of R.I.O.T., rating scales’ results should also be interpreted in the context of other data. One of the best ways to cross-validate the results of a rating scale is with a follow-up interview. For example, when interviewing a parent or teacher about the results of a rating scale, the interview might start with a prompt like “on that questionnaire you filled out, you rated Johnny very high in anxiety. For example, you said he almost always worries. Can you tell me a little more about that? What do you see that tells you he worries all the time?” (Hass 2018). This process allows a parent or teacher to elaborate on their response and provide useful contextual information that helps examiners better understand their ratings.

It is also important to keep in mind that teacher information about a student’s performance in the classroom gathered from an interview or rating scale will be retrospective. While data regarding typical classroom performance via direct observation will often not be available, it may be possible to gather useful observational information from teachers regarding the student’s behavior during the delivery of online instruction. Possible questions include (these and the other interview questions discussed in the paper can be found in the appendix):

• Is the student attending online class sessions? How is attendance?

• How is the amount of work turned in compared with your expectations and other students during online instruction?

• How is the work quality compared to grade-level expectations and other students during online instruction?

• How would you characterize the student’s engagement during online class sessions (e.g., attentive on-screen, participating with questions & answers, etc.)? Any unusual behaviors? (Leung and Hass 2020, p. 14)

Interviewing to Assess Mental Health

Changes in our day-to-day lives rendered by COVID-19 have increased the stress on parents and students and are likely to exacerbate pre-existing mental health conditions. Given this, for the foreseeable future, all students who are assessed should be screened for mental health challenges, including suicide. For all students remotely evaluated, it is vital to know the person’s location, including address, at the start of the session. It will also be critical to have emergency contact information for the student (Center for Practice Innovations 2017).

The following interview questions can be adapted to parents, caregivers, or students of different ages. The questions start with what is called a typical day interview (Goldston and Whiteley 1967).

1. What is a typical day for you like now? Tell me what you do first thing when you get up? Next? And then what…[the the goal is to move from morning to bedtime]

2. Have you been feeling down or cranky lately?

3. What are you doing now that you enjoy, or think is fun? What have you stopped doing that you use to enjoy?

4. Have you ever thought your life was not worth living or thought about hurting yourself in some way?

5. How is your appetite? What kinds of things do you like to eat? Have you gained or lost weight lately?

6. How do you sleep? About how much do you sleep each night?

7. Do you often have trouble paying attention to details or keeping your mind on what you are doing?

8. Do you worry a lot? More than other kids, you know? If yes, ask: Do you worry as often as every day or every other day? What do you worry about?

If, in the process of interviewing a student regarding social and emotional functioning, you suspect suicidality, there are various protocols, including many developed by local educational agencies. A straightforward version is the Columbia Suicide Severity Rating Scale (Posner et al. 2011).

1. Have you wished you were dead or wished you could go to sleep and not wake up?

2. Have you had any thoughts about killing yourself?

3. Have you thought about how you might do this?

4. Have you had any intention of acting on these thoughts of killing yourself, as opposed to you have the thoughts, but you definitely would not act on them?

5. Have you started to work out or worked out the details of how to kill yourself? Do you intend to carry out this plan?

6. Have you done anything, started to do anything, or prepared to do anything to end your life?

Including Strengths

Lastly, tele-assessments, like face-to-face assessments, should address student’s competencies and strengths. Saleebey (2008) discusses questions that help name and elaborate on strengths. Among these are three that are especially important in the context of COVID-19 including (1) perspective questions, (2) change questions, (3) meaning questions, (4) survival questions, (5) support questions, (6) possibility questions, (7) exception questions, and (8) esteem questions (p. 73). Survival questions shift a conversation to how someone has coped with adversity rather than the adversity itself. For example, “Given everything you have had to deal with, how do you make it to school?”

Support questions access perceptions of social support. Who does a student look to for help? Who can they count on? Exception questions focus on what is happening when a problem is absent or diminished. An example might be, “What is different about the days when you don’t’ get mad at the teacher?” Possibility questions inquire about hopes, goals, and dreams. Possibility questions can be as simple as, “what do you like to do?” They can also include questions that focus on the future, such as, “If things work out and go well, what will you be doing in five years?” Lastly, esteem questions focus on current strengths and can be worded to bring in the perspective of someone else, “What would your best friend say they like about you?”

In addition to interviews, well-validated rating scales or questionnaires such as the Social-Emotional Health Survey (SEHS; Furlong et al. 2013, 2014) can be useful in gathering data about students’ strengths. The SEHS assesses several aspects of positive youth development, including positive beliefs or confidence in self, a sense of trust in others, a sense of emotional competence, and feeling engaged in daily living (Furlong et al. 2018). It is available for elementary (Furlong et al. 2013) and secondary (Furlong et al. 2014) age students and can be accessed via an app on smartphones.

Assessment Using Observation (O)

Although traditional classroom observations are not available when students are receiving their instructions virtually, some observations such as those typically part of a mental status assessment, such as affect, mood, and problem-solving style, may be available. However, even if possible, this data would have to be interpreted with caution and cross-validated with other data sources. Also, given the limitations of a video format, other essential data such as non-verbal behaviors, e.g., nervous tapping and limited eye contact, may also not be available. As we noted above, in many cases, it will be more useful to gather retrospective observational information from teachers and parents via interviews.

Conclusion

Although new and innovative assessment methods may be developed, a traditional evaluation that includes the Test in R.I.O.T. is currently not practical regardless of the modifications made. Rather than force the use of tests in ways that they were not standardized for, we propose R.I.O. as an alternative. School psychologists who depend on test scores as their primary source of data will need to become more comfortable with the qualitative data available to them via R.I.O., which depends on considering the weight of evidence and clinical judgment.

Given this, it is essential to consider if gathering data via a review of records, interviews, and limited observations are legally and ethically adequate. Legally, we are required to use a variety of evaluation tools and approaches to gather functional and relevant data (Individuals With Disabilities Education Act 2004). The term “variety” is not defined in law, but professional guidance such as Levitt and Merell’s (2009) rule of two. The rule of two suggests that a finding or conclusion in an evaluation can be sufficiently confirmed with data gathered from two methods, two informants, and in two settings. The application of the rule of two continues to be possible using R.I.O. in the context of virtual evaluations. For example, difficulties with planning and organizing can be confirmed by a combination of parent and teacher behavior rating scales and interviews of parents, teachers, and the student. Problems with reading could be identified by past grades and group test scores, work samples, observations of the student reading, and interviews of the teacher, parent, and student. These each easily meet the standard of data gathered using two methods, from two people and in two settings.

The term “relevant” means the data we gather should be directly useful to stakeholders in helping the student we have assessed (Hass and Carriere 2014). In the Federal Register, the term functional defines functional as “…skills or activities that are not considered academic or related to a child’s academic achievement. Instead, ‘functional’ is often used in the context of routine activities of everyday living” (Federal Register /Vol. 71, No. 156 /Monday, August 14, 2006, Rules and Regulations, p. 4661). These are statements about the kind and quality of the data we gather in assessments, not about how we collect that data. Thus, there is no reason to believe that relevant and functional data cannot be gathered remotely via strategies such as R.I.O.

Core to the NASP code of ethics (2020) is the notion of beneficence or responsible caring. Responsible caring obligates us to act to benefit others. School psychologists accomplish this by using current knowledge from psychology and education to provide effective help. School psychologists are also responsible for helping their clients, children, parents, and professional educators make informed choices. Given this, school psychologists must be transparent about what we propose to do and what its limitations. We must also be both flexible and modest in our claims about children. Both ethically and legally, we are responsible for making good faith efforts to meet students’ needs. Evaluation data is always incomplete, and although data from a tele-assessment may appear more so than typical, the crucial ethical question is: Is this data sufficient to act beneficently on behalf of our clients? Although the circumstances of tele-assessment are dramatically different, this stance of beneficence, humility, and appreciation of human complexity is similar to how we should approach all evaluations regardless of the circumstances.

Biographies

Michael Hass

is a Professor of Scholarly Practice in the Attallah College of Educational Studies at Chapman University. He is also a Visiting Professor at Vietnam National University, University of Education, Hanoi, Vietnam. Professor Hass is author of Interviewing for Assessment: A practical guide for School Counselors and School Psychologists, and co-author, with Dr. Jeanne Anne Carriere, of Writing Useful, Accessible, and Legally Defensible Psychoeducational Reports and Interviewing for Assessment: A Guide for School Psychologists and School Counselors. In 2016, he earned the California Association of School Psychologists’ Sandra Goff Memorial Award for exemplary contributions to the profession of School Psychology in California and the nation.

Brian Leung

Ph.D. is the program director of the LMU Los Angeles school psychology program. He has practiced as a school psychologist in both suburban and urban school districts. Dr. Leung was the president of the California Association of School Psychologists from 2018 -2019.

Appendix

Interview Questions for Use in Tele-assessments

Questions regarding participation in online education

1. Is the student attending online class sessions? How is attendance?

2. How is the amount of work turned in compared with your expectations and other students during online instruction?

3. How is the work quality compared to grade-level expectations and other students during online instruction?

4. How would you characterize the student’s engagement during online class sessions

Questions regarding mental health status

• 5.What is a typical day for you like now? Tell me what you do first thing when you get up? Next? And then what…[the goal is to move from morning to bedtime]
• 6Have you been feeling down or cranky lately?
• 7What are you doing now that you enjoy, or think is fun? What have you stopped doing that you use to enjoy?
• 8Have you ever thought your life was not worth living or thought about hurting yourself in some way?
• 9How is your appetite? What kinds of things do you like to eat? Have you gained or lost weight lately?
• 10How do you sleep? About how much do you sleep each night?
• 11Do you often have trouble paying attention to details or keeping your mind on what you are doing?
• 12Do you worry a lot? More than other kids, you know? If yes, ask: Do you worry as often as every day or every other day? What do you worry about?

Questions regarding suicidality from Columbia Suicide Severity Rating Scale (Posner et al. 2011)

• 8.Have you wished you were dead or wished you could go to sleep and not wake up?
• 9Have you actually had any thoughts about killing yourself?
• 10Have you thought about how you might do this?
• 11Have you had any intention of acting on these thoughts of killing yourself, as opposed to you have the thoughts, but you definitely would not act on them?
• 12Have you started to work out or worked out the details of how to kill yourself? Do you intend to carry out this plan?
• 13Have you done anything, started to do anything, or prepared to do anything to end your life?

Compliance with Ethical Standards

This research leading to this paper did not involve gathering data from humans or animals; thus, no informed consent was required.

Conflict of Interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.