Rapid Online Assessment of Reading (ROAR): Evaluation of an Online Tool for Screening Reading Skills in a Developmental-Behavioral Pediatrics Clinic

Abstract

Objective:

Reading difficulties are highly prevalent and frequently co-occur with other neurodevelopmental/behavioral conditions. It is difficult to assess reading routinely in pediatric clinical practice due to time and resource constraints. Rapid Online Assessment of Reading (ROAR) is an objective, gamified assessment that children take in a web-browser without adult supervision. This study’s purpose was to evaluate ROAR as a screening tool for reading difficulties in a clinical setting.

Method:

A convenience sample of 6–14 year old children, attending an in-person or telehealth visit in a Developmental-Behavioral Pediatrics (DBP) clinic participated. Children took ROAR and completed the Woodcock-Johnson IV Letter-Word Identification (LWID) and Word Attack (WA). Basic Reading Skills (BRS), a standardized aggregate score of LWID and WA, was used as the gold standard assessment. The strength of association between standard scores on ROAR and BRS was calculated. BRS scores < 90 (bottom quartile) were classified as poor readers. Receiver Operating Characteristic (ROC) curve analysis was used to assess the quality of ROAR as a screening test.

Results:

A sample of 41 children, 78% boys, mean age 9.5 years (SD 2.0 years), completed the study. The correlation of ROAR standard score with BRS was r = 0.66, p<0.001. ROC curve analysis with ROAR scores accurately classified poor readers with an area under the curve (AUC) of 0.90.

Conclusion:

ROAR is a useful objective screening tool to identify children at high risk for reading difficulties. Assessment of the tool during a busy clinic was challenging and a larger replication is warranted.

BACKGROUND:

Reading disorders (e.g., dyslexia) affect 5–12% of the general population^1–3. Children with reading disorders are at increased risk for difficulties across academic domains, including writing and math, and for broader struggles in school generally^1,2. Reading disorders often co-occur with other neurodevelopmental and neurobehavioral conditions; the prevalence of reading disorders is estimated to be 25–40% in children with attention deficit-hyperactivity disorder (ADHD)^4,5, approximately 30% in children with autism⁶ and 10% in children with anxiety⁷. Reading difficulties also are more prevalent among children with chronic disease, children born prematurely and those with under-stimulating home reading environments^8,9. A quick and valid screening tool that accurately identifies children at high risk for reading disorders would be useful for many pediatric clinics, including subspecialty and primary care clinics, to recognize which children should undergo a thorough, formalized reading assessment.

The Rapid Online Assessment of Reading (ROAR) is an objective, online reading screening tool, developed for the general population of school-aged children, 6–18 years old¹⁰. The tool uses a two-alternative, forced choice, time-limited lexical decision task (LDT) to assess reading skills¹⁰. The LDT has been gamified to be engaging and easy to navigate for school-aged children and adolescents. The child is automatically led through a series of increasingly difficult written words or pronounceable non-words (pseudowords) that appear on the computer screen. Their task is to indicate if the stimulus on screen is a valid word or a pseudoword. The full version takes 15 minutes to complete. The test has been formatted to be delivered in a web-browser, requiring no adult involvement once the child has been enrolled and has entered the site. Thus, ROAR can be self-administered online from any computer without the presence of a parent, teacher, psychologist, or other adult - it does not require any specialized software to be installed. The results are relayed to an online database where they are scored automatically in real time. ROAR returns a raw score (total items correct) and a standard score (mean of 100 and standard deviation of 15 calculated from age-based norms) and both reflect accuracy. Future ROAR infrastructure will allow results to automatically be entered into an electronic medical record or be emailed to the provider or family, but these features were not available for this study.

ROAR has proven to be an accurate approximation of standard reading measures in children between ages 6 and 18 years with reading profiles ranging from severely impaired to exceptional in community and school settings¹⁰. In the initial validation study, ROAR performance had high correlation (r = 0.91) with raw scores on Woodcock-Johnson IV Letter Word Identification (LWID), a standardized measures of reading ability¹⁰. ROAR has also proven to be highly reliable (r = 0.97) across test administrations¹⁰.

Developmental-Behavioral Pediatrics (DBP) subspecialty clinics evaluate children with neurodevelopmental and neurobehavioral conditions such ADHD, autism, anxiety, and other learning differences. Because of the high co-occurrence between reading disorders and these neurodevelopmental/behavioral conditions, the DBP clinic is an ideal location to identify children with reading disorders. Additionally, the creators of this study sought to evaluate ROAR in a clinical population because the patients are likely to have a wide range of academic abilities and co-existing conditions that affect school functioning. Such children may also have difficulties performing well on ROAR because of inattention, assessment anxiety, and/or other challenges common to the neurodivergent population. Because ROAR is gamified, simple, and short, we were hopeful it would prove reliable and valid in this clinical population.

We hypothesized that standard scores on the ROAR would be strongly correlated with standard scores on a gold-standard test, the Basic Reading Skills (BRS) aggregate score from the Woodcock-Johnson IV Tests of Achievement. The BRS combines scores from the Letter-Word Identification Test (LWID) and Word Attack (WA) to assess single word reading and phonological decoding¹¹. If ROAR can be shown to correlate with established standardized reading assessments in a clinical population, then it may prove useful as an objective screening tool that could be completed at home or in a clinic to accurately identify those children who are at high risk of concurrent reading disorders.

METHODS:

Study Design

The study is a prospective cohort design comparing ROAR as a screening test versus a gold standard measure of reading ability.

Participants

A convenience sample of participants (n=41) were recruited between August 2021 and August 2022 in the DBP clinic at Stanford Medicine Children’s Health, part of the Stanford Medicine in Palo Alto, CA (Table 1). Participants in this study represented a mix of new and established patients returning for follow-up care. Children in this sample had a variety of neurodevelopmental and neurobehavioral conditions such as ADHD, autism, anxiety, language disorders and learning disabilities, as outlined in Table 1. Some children may have underlying conditions that predispose them to neurodevelopmental or neurobehavioral differences, such as preterm birth or genetic differences. In this sample, patients did not include children with complex medical comorbidity.

Table 1:

Characteristics of Patient Cohort Aged 6–14 years (n=41)

Agea (years)	Min 6.5, Max 14.2, Mean 9.5, Std Dev 2.0
Male, n (%)	33 (78.6%)
Insurance type, n (%)
Public	13 (31.0%)
Private	29 (69.0%)
Special Education Services:
No IEP/504	20 (48.8%)
IEP in place	15 (36.6%)
IEP evaluation in process	3 (7.3%)
504 in place	1 (2.4%)
Other/unknown	2 (4.9%)b
Diagnoses per most recent medical encounter:
Single diagnosis	14 (34.1%)
Two diagnoses	18 (43.9%)
Three diagnoses	7 (17.1%)
Symptom level diagnosis provided only	2 (4.9%)
Diagnosis (total)
ADHD, any type	28 (68.3%)
Anxiety	17 (41.5%)
Learning Disability	10 (24.4%)
Dyslexia	7 (17.1%)
Dysgraphia	2 (4.9%)
Dyscalculia	1 (2.4%)
Autism	9 (22%)
Speech language disorder	3 (7.3%)
Depression	2 (4.9%)
Oppositional Defiant Disorder	1 (2.4%)
Disruptive mood dysregulation disorder (DMDD)	1 (2.4%)
Diagnoses (multiple)
ADHD and Anxiety	9 (22.0%)
ADHD and Learning Disability	6 (14.6%)
ADHD and Autism	5 (12.2%)
ADHD and Speech Language disorder	2 (4.9%)
Anxiety and Autism	4 (9.8%)
Anxiety and Depression	2 (4.9%)
English is primary language	37 (90.2%)c

^aAge in years when the child completed ROAR. Mean displayed as continuous variable.

^bOne patient had a student study team evaluation but no IEP as of yet. Another patient’s mother was unsure what if any special education resources were being provided to her child.

^cFour parents did not complete the parent survey

Recruitment

Potential study participants were identified by direct referral to the research team by their DBP clinician or were identified from the clinic schedule and provided with a study flyer. Inclusion criteria were as follows: the child’s parent needed to read English to complete the consent; the child was 6 years of age or older; the caregiver confirmed at recruitment that the child could at least read simple sight words (intellectual disability was not independently a disqualifying condition). The child also needed documented English proficiency, defined as having either English as a first language or having completed at least two years of schooling in English if a language other than English was spoken at home. Patients and families that were interested in participation in the study were given the invitation flyer that contained an internet link to an online dashboard where the parent provided written consent, HIPAA authorizations, and a brief parent questionnaire. The child provided study assent and completed ROAR. All study procedures, consent documents and recruitment materials were presented to and approved by Stanford University IRB.

Measures

ROAR is a two-alternative, forced choice, lexical decision task delivered in a web-browser. Before beginning ROAR, the participant is led through instructions explaining how to complete the assessment, with practice and feedback to ensure understanding. In addition to written instructions, each page has a voice recording of the instructions that can be heard through the computer’s speakers. The ROAR instructions explained that the participant has entered the world of Lexicality and the goal is to reach a gate that connects participants to their own world, Earth. The participant’s helper, the Scout, explains that to find the gate, the participant needs to tell the difference between the magical language of Lexicality, a set of pronounceable non-words (pseudowords), and words in English. The stimulus word flashes briefly (350ms) on the computer screen and the participant uses the arrow keys on the computer (left or right) to indicate if the word is real or magical (Figure 1a). The participant completes up to 3 practice trials, with corrections and additional practice items before the participant can continue. The participant must answer each practice trial correctly before they can advance to the actual assessment. The full test version consists of 252 word or pseudoword prompts split up into four sections with breaks offered in between sections. The set of prompts was determined by preliminary studies using Item Response Theory¹⁰. ROAR back-end programming calculates a total correct score and measures how long between word/pseudoword presentation and when a left or right arrow key was struck. When the participant answers correctly, a pleasant chime sound is produced and when the participant answers incorrectly a dissonant thud sound is produced. There is no time limit for participants to choose the right or left arrow key, but an answer is required for the game to proceed. Participants are kept engaged during ROAR by collecting coins - the participant gets 10 gold coins for each 10 correct answers (Figure 1b). In addition, for each section completed an animated character joins the participant’s journey. This study used the same version of ROAR introduced in the original paper. Subsequent versions, using a computer adaptive testing algorithm for more efficient assessment, have since been released¹⁰ which take about 5 minutes to complete.

Figure 1:

The image on the left (1a) shows an example of a pseudoword and the prompts on the screen for the participant to choose the left or right arrow key. In this case the participant would use the left arrow key to indicate that “hom” is a pseudoword. The image on the right (1b) is animated and displayed to the child after collecting 10 coins. The child is nearly at the end of ROAR testing and has three companions that have joined the adventure.

The Woodcock-Johnson IV Tests of Achievement Letter-Word Identification (LWID) and Word-Attack (WA)¹⁰ were administered by a member of the research team. The Basic Reading Skills (BRS) score is an aggregate measure that combines scores of LWID and WA. The BRS was the gold standard assessments of single word reading in this study. The LWID, WA, and BRS each have a median reliability of 0.92, 0.90, and 0.95 respectively in the 5 to 18 age range¹¹.

Most of the participants (n=32) completed the LWID and WA in person. A few (n=9) completed the assessments via video teleconference, which were done during telehealth visits or when the child had not been able to complete all study procedures while in-person at the clinic visit. The definitive testing took about 10 minutes in total. The research team did not have knowledge of the ROAR results prior to administering the gold standard assessments.

The study team did not provide feedback to the families directly regarding the results of ROAR or the gold standard testing with the Woodcock Johnson IV. We shared the results of the Woodcock Johnson IV with the primary DBP clinician and flagged results that were suggestive of poor reading skills.

Parent Questionnaire

A parent questionnaire was used to confirm patient demographic information and school services in place for the child, if any.

Chart Review

The research staff performed a chart review on the 41 participants that completed the study. The information collected in chart review included: age, sex, top three diagnoses associated with the clinical encounter at time of recruitment, presence or absence of school supports such as individualized education plan (IEP) or 504 plan, and insurance type.

Data Analysis

Descriptive statistics were tabulated to evaluate and categorize the patient population. The degree of association between the ROAR standard score and standard scores on the LWID, WA and BRS were calculated using Pearson moment correlation. The degree of association was assessed using partial correlations with potential covariates, including the child’s sex, co-existing diagnosis, and age. To evaluate ROAR as a screening test for reading ability, we performed classification analysis with creating a receiver operating characteristic (ROC) curve and assessed area under the curve (AUC). We defined participants as poor readers if their BRS score fell in the lowest quartile of scores.

RESULTS:

Participants that completed the study were between the ages of 6 and 14 years. Table 1 summarizes characteristics of the patient cohort. The mean age of the participants was 9.5 years; the majority were male (78.6%) and had private insurance (69%). The percent who had no school support in place such as an IEP or 504 plan was 48.8%. Most participants had more than one medical diagnosis associated with the clinical encounter at which they were recruited. The four most common diagnoses were ADHD, any type (68.3%), anxiety (41.5%), learning disability (24.4%) and autism (22%). Most participants learned English as their first language (90.2%). Given that we were collecting a convenience sample, we did not carefully track the percent of patients who declined participation. Within our sample the minimum value (Min), maximum value (Max), mean (M) and standard deviation (SD) of the standard scores for the different assessment measures were as follows: WA (Min 54, Max 145, M=100.8, SD=18.9), LWID (Min 40, Max 141, M=95.1, SD=21.1), BRS (Min 40, Max 143, M=97.4, SD=20.4), and ROAR (Min 70, Max 160.5, M=104.0, SD=20.6). ROAR standard scores were normally distributed.

Correlation

ROAR raw score was strongly correlated with raw scores on the Woodcock Johnson IV Tests of Achievement: LWID (r = 0.76, p < 0.001, CI [0.59, 0.87]) (Figure 2a) and WA (r = 0.73, p < 0.001, CI [0.55, 0.85]) (Figure 2b). ROAR standard scores were also significantly correlated with the Woodcock Johnson test of Achievement standard scores BRS (r = 0.66, p < 0.001, CI [0.44, 0.80]) (Figure 3), LWID (r = 0.65, p < 0.001, CI [0.42, 0.80]) and WA (r = 0.63, p < 0.001, CI [0.40, 0.79]). Scatter plots revealed three participants that were identified as outliers with a standardized residual that was larger than 2 (in absolute value). One participant was recruited during a new patient visit; this child had a poor BRS score and poor ROAR standard score and was identified as likely having intellectual disability during the clinical visit. The second outlier did comparatively less well on ROAR than BRS; difficulty with sustaining attention after a long clinical visit likely contributed to lower ROAR score, although the child’s performance remained in the high average ROAR score. The third outlier performed in the average range on LWID and WA in clinic then completed the ROAR, performing above the average range; the parent shared that the child was somewhat inattentive in the clinic testing and at home was given a distraction-free environment to work on the ROAR where he performed better than on the standardized measures. Removal of these three outliers did not substantially change the correlation between standard score on ROAR and BRS (r = 0.68, p < 0.001, CI [0.46, 0.82])

Figure 2:

Scatter plot 2a shows each participant’s LWID raw score versus their raw score on ROAR; Pearson correlations coefficient (r = 0.76). Scatter plot 2b shows each participant’s WA raw score versus their raw score on ROAR; Pearson correlations coefficient (R = 0.73). Scatter plot 2c shows each participant’s BRS standard score versus their standard score on ROAR; Pearson correlations coefficient (r = 0.66). In 2c the triangles indicate the participants that were identified as outliers with a standardized residual that was larger than 2 (in absolute value).

Figure 3:

ROC curve of participants’ standard score on ROAR in relation to Woodcock Johnson IV Test of Achievement Basic Reading Score (BRS). Participants were classified as good readers versus poor readers based on a cutoff standard score of 90 on the BRS. Area under the curve (AUC) = 0.90 (CI 0.81 to 1.00).

ROC curve

To evaluate ROAR for use as a screening test to classify individuals with reading difficulties or not, we created a receiver operating characteristic (ROC) curve and assessed area under the curve (AUC). To construct the ROC curve, we set a classification threshold for poor readers as a standard score on the BRS of less than 90. With this distinction, 26.8% (approximately the lowest quartile) of participants in our clinical sample were categorized as poor readers. The ROC curve was created by plotting true positive rate versus the false positive rate. The curve created has an area under the curve (AUC) of 0.90 (CI 0.81 to 1.00) (Figure 3). The coordinates on the ROC curve were then analyzed to determine the score on ROAR with the optimal sensitivity and specificity of distinguishing between good and poor readers. A standard score of 91.22 or greater on ROAR has a sensitivity of 90.0 and specificity of 81.8 for predicting a BRS score greater than 90.

DISCUSSION:

In this study, we found a strong correlation between raw and standard scores on ROAR and standard scores on the Woodcock Johnson LWID, WA and BRS. The area under the ROC curve showed that ROAR was an excellent measure to identify children at highest risk of having poor reading skills. These findings suggest that the ROAR is a useful screening tool for identifying children in a pediatric general or subspecialty clinic who are at high risk for reading difficulty or reading disorder, including children with co-existing neurodevelopmental/neurobehavioral conditions. ROAR is designed to be objective, fast, and completely automated with the goal of lifting the burden of screening from clinical staff and not relying on subjective and potentially biased parent-report measures. Because of ROAR’s ability to be delivered via an online browser, it provides flexibility to be administered virtually anywhere at any stage of the clinical assessment.

Many states require schools to screen children for dyslexia and other reading disorders. However, at this time, 10 states, including California, do not¹². Early screening and intervention are critical to reducing the prevalence and severity of reading disorders¹³. Using an objective, validated screening measure in clinical settings may be helpful for children who are not screened at school, for children whom the school screening is at odds with their parents’ impressions, or for children whose medical or neurodevelopmental/neurobehavioral profile raises challenges for other screening approaches. In addition, if a child has difficulty with reading after a negative state-administered screening test, an objective screening tool might trigger consideration that the child was a false negative on the original screening test. Implementing ROAR in clinics and sharing the results could improve the communication between clinic, home, and school about a child’s reading status. ROAR is already being implemented within selected schools.

As with any screening measure, best practice is that a positive result triggers an assessment. ROAR has been designed to be a screening test to determine whether resources should be devoted to a more detailed assessment of a child. In the case of a positive result on the ROAR, the reading assessment could take place in a medical setting or an educational setting, depending on the details of the case and the resources of the community.

One limitation of this study is the small sample size. Research staff had anticipated being able to recruit more participants for the study within the timeline than were ultimately recruited. The staff found that participants largely declined participation in the study because they did not want to add extra time to their clinical visit. There were occasions when patients and families consented to the study but were unable to complete either the ROAR or the gold standard assessment before they had to leave clinic. If participants left the clinic without completing all aspects of the study, it was difficult to entice them to finish the procedures despite emailing them with reminders several times. To add more confidence in using ROAR as a screening tool in a clinical population, it should be assessed in a larger population. The benefit of a larger population would be to replicate these findings and to assess how ROAR performs as a screening tool in clinical sub-populations, which we could not accomplish confidently in the present study. In addition, subsequent studies could use the new version of the ROAR, which uses a computer adaptive testing algorithm, reducing the time required to generate accurate results.

Another limitation was that this study was also completed in only one setting, a single DBP clinic. The advantage was that it was easy for the research team to obtain the standard scores on the ROAR from the test developers. In the future, this screening tool should be evaluated across various subspecialty clinics, including those that see children with complex medical conditions. Additional assessment of the tool would indicate if ROAR could be a valid screening tool appropriate for universal use in general pediatric clinics, for all patients. Before the instrument could be used broadly, the infrastructure for delivering scores to clinicians and/or families would need to be further developed.

Another opportunity to consider in future research with a larger sample is to investigate why discrepancies exist between ROAR score and Woodcock Johnson scores. The correlation between ROAR scores and Woodcock Johnson scores was lower in this DBP clinical study than the correlation shown in the prior general population study (r = 0.76 current study versus r = 0.91 in the prior study). A larger sample would be required to determine the extent to which this reflects specific characteristics of the clinical sample versus the influence of outliers in a small sample. For example, one of the outliers had an average Woodcock Johnson score for age and an above average ROAR score. One possible explanation is that this participant’s inattention in a new clinical setting made the child perform below their true ability level when they were completing one-on-one, individually administered assessments with the research team. In this scenario, ROAR performed in a distraction-free environment might, in fact, be a more accurate estimate of true ability than a standardized clinical measure. Another possibility is that, in certain cases, the ROAR taps into different underlying skills than the Woodcock Johnson IV and might, therefore, miss some cases of reading difficulties that would have been detected by an individually administered assessment. Another possibility to be aware of is that an outlier might have received help from a parent or sibling; with an unmonitored assessment, this possibility must be taken into consideration. At present, ROAR development team is making efforts to write algorithms to flag these cases.

CONCLUSION:

We found strong correlations between ROAR scores and a gold standard measure of reading ability and a high area under the curve in ROC analysis, providing strong preliminary support for the use of ROAR as a screening tool to accurately identify those children in a pediatric medical clinic as likely having reading difficulties. Future research in larger samples is necessary to fully understand the strengths and weaknesses of ROAR in relation to conventional, individually administered assessments and how differences between ROAR and conventional assessment vary by clinical condition. Further studies should consider how to implement ROAR in relation to other approaches to reading screening, such as use of parent surveys. In addition, this study provides a strong proof-of-concept for how medical clinics can leverage new technology to improve efficiency in conducting comprehensive evaluations of young children at high risk for reading disorders.