How moderation affects remote psychophysical tasks with children

Abstract

The increasing use of remote platforms for auditory research necessitates more in-depth evaluation of assessment protocols, especially when working with children. This work investigates the influence of the presence of a moderator on remote audiovisual speech perception studies, by assessing how moderation impacts children’s understanding and performance of the psychophysical tasks as well as their attention on these tasks. In sum, both moderated and unmoderated methods can reliably assess audiovisual speech perception benefits. However, regardless of similar error patterns between both studies, unmoderated online studies with children are prone to more general attention lapses as suggested by higher overall error rates.

1. Introduction

Internet-based research is not new. The dawn of internet ushered the first wave of remote data collection [1]. Initially, online surveys gained popularity [2], followed by increased interest for online assessments of psychology-related topics with therapeutic purposes [3]. Other disciplines remained more attached to in-person assessments. For auditory-related research, the recent COVID-19 pandemic accelerated interest in online research [4].

Remote research can be beneficial when executed thoughtfully. It allows data collection of large sample sizes [5] in real-world environments while potentially overcoming geographical limitations, with opportunities for increased ecological validity and generalizability. Participating from the comfort of the individual’s home can reduce stress and costs [1], as well as enhance opportunities for longitudinal or training studies [6].

Nonetheless, conducting research online is not without challenges – calibration of stimuli can be difficult and attrition is often a concern for remote research [1]. There is even more hesitance when adapting studies with children for remote testing, perhaps due to the limited peer-reviewed methodological reports benchmarking the success of these remote adaptations in developmental research protocols. Behavioral research involving child participants requires age-specific considerations related to task design [7], attention span [8], immature auditory attention skills [9], cognitive load [10], language development [11], and general perceptual development [12]. Limited face-to-face interaction, minimal relationship building with a remote researcher, the abstractness of online rewards, and the limited opportunity for physical breaks in combination with longer screen times makes it even more complex to control all these aspects when moving developmental research online.

The presence of a researcher moderating a study can also influence the results. For example, a researcher can unintentionally or unconsciously present unwanted verbal or non-verbal cues, or even create stress for the participants that lowers performance just by their presence. This phenomenon, known as the researcher effect, can be easily reduced in remote studies [5]. However, limited presence also confines aspects the researchers can control. For example, readily assessing engagement of the participant or understanding of the task becomes more complicated when the researcher is not present. Therefore, when designing a developmental study with remote data collection, the degree of moderation may need to be carefully considered. This degree of moderation can range from completely unmoderated (i.e., no researcher is present and all the instructions and guidance are provided by computer prompts), to completely moderated (i.e., the researcher is constantly present via a video-conferencing tool giving the instructions as they would for in-person research and guiding the participants through the tasks), to anything in between.

Unmoderated studies can increase across-subjects consistency and remove potential experimental bias due to the instructions being delivered in a fully automated fashion. The data collection phase is also less work-intensive and can be easily scaled to a greater number of subjects. However, it can increase the number of invalid data points, especially if the child does not sufficiently understand the task. Additionally, the absence of researcher supervision can increase technical errors (e.g., corrupted recording files; [13]), and in general it does not create space to address questions or concerns of the participants and their caregivers [14]. In contrast, moderation allows for improved task understanding, and for immediate experimental adjustments to better suit the testing environment. Children often reside in sensory stimulating rooms that can be distracting during data collection. The presence of a remote researcher can anticipate and mitigate these problems. However, especially when working with young children who struggle with unfamiliar surroundings, introducing a stranger (i.e., the moderator), even online, can be intimidating and potentially interfere with the validity of data collection [5]. Additionally, not only the child’s but also the caregiver’s behavior can unwillingly or unconsciously change by the presence of a researcher. Moderated or unmoderated remote studies therefore might introduce different additional confounds to the original research question that warrants care in the interpretation of the study results, and especially comparing across studies.

The current work quantifies how moderation can affect remote psychophysical tasks with children. Gijbels et al. [14, 16] completed two highly similar remote audiovisual speech perception studies in children (age 6–7.5 years)—one moderated and one unmoderated—to address how children use audiovisual speech perception cues to improve their speech understanding in noise. Specifically, the paradigm was designed to answer the question: “How does it benefit children when visual cues (mouth movements) are added to a degraded auditory speech signal (speech-in-noise)?” Both studies [14, 16] looked at improvement of speech recognition scores, and whether the use of speech-specific visual articulatory cues was necessary to accomplish this audiovisual speech benefit. Cognitive and linguistic influences were minimized by developing an age-appropriate task that requires very little cognitive load, and all vocabulary used in these studies should be acquired by age 3 – 5 years [15]. The details of the specific research questions in both studies, as described in Gijbels et al. [16,17], varied but are not relevant for the further analysis described here.

Both studies included measures to look at different aspects of attention. This allows the researchers to investigate how attentional factors influence these perceptual skills and whether this is different based on the level of moderation. Attention skills are generally immature for children between 6–7.5 years old [9] and the level of attention varies depending on how appropriately designed these tasks are for the age group [7]. By monitoring both their general attention level (i.e., the task engagement within every trial) as well as their cross-modal attention (i.e., the children’s ability to attend to the visual modality while they are performing this speech task), different aspects of attention influenced by the researcher being present for this age group can be studied.

In sum, this work aims to address three questions when comparing moderated and unmoderated studies:

• What is the influence of moderation on performance (audio-only, visual-only, and audiovisual) of an online speech perception task with children?

• Does moderation influence general attention for children performing an online task?

• Does moderation influence cross-modal attention for children performing an online task?

2. Methods

A description of the study design is provided below, highlighting the differences between the moderated and unmoderated studies. For a more detailed explanation of participants, stimuli, recordings, procedures, and analysis methods we refer to [16].

2.1. Participants

All participants were children, with reported typical speech, language, auditory, and visual (normal or corrected to normal) development. They were recruited through a pre-existing database at the Institute for Learning & Brain Sciences, or through the Communication Studies Participant Pool at the University of Washington. Parents of all participants provided written informed consent under a protocol that was approved by the University of Washington’s Institutional Review Board.

2.1.1. Moderated study

The participants (N= 37) of this remote moderated audiovisual speech perception study [16], were a subset of the 48 children who participated in a previous study at the University of Washington [18]. This subset of participants (M = 15, F = 22) had English as a first language, were in first grade and were between 6.29 and 7.36 years of age (mean = 6.74 years, SD = 0.27), at time of participation.

2.1.2. Unmoderated study

For the purpose of comparing both moderated and unmoderated studies we selected a subgroup of 46 (M = 23, F = 23) children with a similar age range (6.04 – 7.44 years, mean = 6.73, SD = 0.44) from a group of 161 English-speaking children (M=79, F=82), age 4 to 15, who participated in the remote unmoderated audiovisual speech perception study [17].

2.2. Stimuli and materials

The psychophysical tasks used in both moderated and unmoderated studies measured auditory, visual, and audiovisual speech perception. All target words were one-syllable consonant-vowel-consonant (CVC) words from the Lexical Neighborhood Test (LNT; [19]) recorded by a female native English speaker [15], embedded in speech-weighted noise, presented at a comfortable suprathreshold level (for more details: [16]).

We used a one interval four alternative-forced-choice (1I-4AFC) task, consisting of the CVC word (1I), followed by four images (4AFC; presented 2 × 2): the target (e.g., sun) and three carefully chosen foils: (a) a “minimal pair” foil, which differed in first or last consonant from the target word (e.g., run), (b) a “vowel” foil, which shared only the same vowel with the target word (e.g., gum), and (c) a “random” foil, which had no shared phonemes with the target word (e.g., pink). This “random” foil was added to measure general attention as all children should receive sufficient information from the audio-only trials to not pick this answer option. All the minimal pairs were auditory minimal pairs; thus, they are distinctively different acoustically but not necessarily visually (i.e., articulatory movement of the mouth).

Audiovisual trials were a combination of the auditory recording of the target word, with the accompanying video of the speaker producing the word. Audio-only trials were auditory recordings of the target word in noise paired with a still image of the same female speaker with a neutral expression, and visual-only trials were essentially silent videos. Speech-weighted noise was present in all three modalities. There was no visual noise present in any modality.

2.2.1. Moderated study

All participants completed two speech-in-noise recognition sessions, which differed by their target stimulus set sizes (i.e., 25 vs 10). As the small stimuli set was a subset of the large set and our analysis [16] yielded no distinction between these two sessions, we will only discuss the combined results. The stimuli were tested in two noise levels, −2 dB signal-to-noise ratio (SNR) and −8 dB SNR.

2.2.2. Unmoderated study

The participants completed one speech-in-noise recognition session, with 20 different spoken target words. In this unmoderated study, the target words were presented in three different noise levels: −5 dB, −8 dB, and −11dB SNR. These levels were chosen based on the expanded age range for this study and analysis of the moderated study [16].

Although the two studies were not identical and consisted of different conditions (i.e., noise levels and set sizes), to accurately compare both studies we only used trials from both studies that were similar. This means we limited our comparison for all further analysis and discussion to (congruent) audiovisual speech trials and audio-only trials that were tested in −8 dB SNR. The visual-only trials were very similar in both studies. The number of trials presented in each study for each condition slightly differed (i.e., 170 vs 200 trials).

2.3. Procedures

In both study types, children were seated at a table facing the computer screen, and a caregiver was asked to sit next to the child participant during testing. Depending on each child’s computer fluency as assessed during the training session, answers were collected via mouse clicks either completed directly by the participant or by the parent when the participant pointed to the screen. Before testing and training, participants were instructed to set the computer’s audio output to a comfortable level based on a repeating speech fragment of a cartoon character, and auditory stimuli were presented in free field through the computer’s speakers. Participants completed a training session consisting of two phases (for more details: [16]). In phase 1, all children would be exposed to all used words (target and foils) and in phase 2 children would complete some example trials of the actual task.

Three trial types (audio-only, visual-only and audiovisual) were assessed at the different SNRs, in random presentation order. There was no time limit to respond, and no feedback was provided. Participants were offered a break after every block, and they could continue when ready. On average task completion would take up to 20–30 minutes, depending on the duration of breaks and response times.

A picture of a cartoon character was introduced with a double function; i) it was a fun motivator for the participants throughout the studies and ii) served as a catch-trial stimulus to measure cross-modal attention throughout the experiment. In these catch-trials, the cartoon character would randomly pop up on the screen and children had to respond verbally (in the moderated study) or manually by pressing an additional button on the screen (in the unmoderated study), to confirm that they noticed the cartoon and were still visually engaged to the task. For these trials no response to the presented speech stimuli from the children was measured.

2.3.1. Moderated study

The moderated study was created via a free online study builder. The tasks were emailed to the caregiver, and they were directed to load the tasks on their computer directly. The study was moderated via a videoconferencing session that was simultaneously set up to guide caregivers through the experiment (e.g., loading the task file, collecting the data, and sending the output file back), and to make notes of any unforeseen circumstances (e.g., internet issues, sibling intervening). The computer screen was shared with the moderator. The child’s understanding of the training was assessed with the moderator to assure all words were part of the child’s vocabulary (phase 1) and the child had an understanding of the task setup itself (phase 2). At the beginning of the training the caregiver was informed by the moderator about their role during the experiment.

Children completed 200 trials, broken up in a longer (110) and shorter (90) session, with a total of 10 blocks of trials. Randomly dispersed throughout all trials, the catch-trial cartoon character appeared 20 times in total. Participants could take breaks between blocks in every session, but critically the moderator had the opportunity to suggest additional breaks in the middle of trial blocks when they noticed interest in the task declined.

2.3.2. Unmoderated study

The unmoderated study was created via the same free online study builder; however, the experiment was instead hosted via an online platform created to independently run remote studies. Parents would navigate to the experiment via a web browser link and responses were automatically stored online. In this unmoderated study, parents were provided with clear instructions via email with no researchers present during the assessment of the task. Like the moderated study there were two training phases; however, image understanding from phase 1 was not explicitly assessed. Children completed 160 trials, broken up into 8 blocks. The remaining 10 trials were catch-trials.

3. Analysis, Results and Discussion

3.1. Moderation and Overall Task Performance

The first step was to validate that our findings in both remote studies (moderated and unmoderated) are comparable to results obtained in other audiovisual speech perception tasks in children obtained in traditional in-person settings. Accordingly, these children were expected to i) score above chance level even on the most difficult visual-only trials [20] and ii) to perform best in the audiovisual trials, followed by audio-only then visual-only modality. Performance was analyzed in all modalities with a one-tailed t-test against performance at chance level (i.e., > 25% in these 1I-4AFC tasks), for both the moderated and unmoderated studies, with Bonferroni correction for each modality per study. Results (See Table 1) show that children scored significantly above chance level for all modalities in both studies (all p< .001) and that they performed best in the audiovisual trials, followed by the audio-only and visual-only trials (Fig. 1)

TABLE I.

One-tailed t-test of auditory, visual, and audiovisual modality for both the moderated and unmoderated task, against a 25% chance performance.

Study	Modality	Mean score (% correct)	t-value	p-value
Moderated	Audiovisual	84	61.30	<.001
Unmoderated	Audiovisual	79	37.53	<.001
Moderated	Auditory	81	44.10	<.001
Unmoderated	Auditory	71	36.95	<.001
Moderated	Visual	44	7.82	<.001
Unmoderated	Visual	38	4.79	<.001

FIG. 1.

Boxplots of performance (in % correct) per modality (Audio-only (−8 dB SNR), Audiovisual (−8 dB SNR), Visual-only) per study (Moderated, Unmoderated). The red dotted line indicates chance performance (25%). Differences between studies are indicated with p-values. Thick horizontal lines represent medians, gray dots represent means, boxes represent interquartile ranges, and whiskers represent the range, excluding outliers. Individual scores are visualized as small black dots.

Children this age are expected to perform significantly better in audiovisual trials than in audio-only trials [21]. These audiovisual speech perception benefits can be expected for monosyllabic words presented in moderate background noise (e.g., −8 dB SNR; [21]). A one-tailed t-test shows that for both moderated (μ=3.54, t₃₆ = 3.11, p = 0.001) and unmoderated studies (μ=7.74, t₄₅ = 4.62, p < 0.001), there are significant audiovisual speech perception benefits. Results so far are in line with findings of other in-person studies [20,21], indicating the validity of both moderated and unmoderated online studies with young children.

Next, we explored how moderation influenced the online performance of these studies with children. As visualized in Figure 1, the mean performance is lower in the unmoderated study for every modality. Two-tailed t-tests comparing both studies per modality show significant differences for the audio-only (t_79.80 = 5.37, p< .001) and audiovisual (t_75.60 = 3.08, p= .003) trials, but not for the visual-only trials (t_80.99 = 1.63, p > .10). A Type III ANOVA of our linear mixed model (Model: Performance ~ modality * study + (1|subject)), describes a significant main effect of modality (F_175.44 = 361.38, p< .001), a significant main effect of study (F_200.75 = 17.14, p < .001), but no significant interaction effect (F_175.44 = 0.96, p > .10). Therefore, we conclude that performance is overall lower in the unmoderated study, but the relationship between modalities is not significantly different for the moderated or unmoderated studies.

Overall, we find that children in the moderated study tend to do better in all modalities (Fig. 1). This difference in performance between both studies is significant in both the auditory and audiovisual modalities. This suggests that a moderator’s presence significantly influences the study results [5], where children perform better when a moderator is present. However, the audiovisual speech perception benefit — a difference score in performance (Audiovisual speech perception benefit = [(AV-A)/(1-A)]), and our a priori measure of interest—is not significantly different for the moderated or unmoderated studies. This is the result from a similar decline in audio-only and audiovisual performance in the unmoderated study and suggests that whether moderation influences remote studies depends on the chosen outcome measure (i.e., performance vs. difference in performance).

3.2. Moderation and General Attention

Analyzing the errors made in both moderated and unmoderated studies provides an indication of general attention during the online studies. Both moderated and unmoderated studies use the same 1I-4AFC tasks. As explained in the Methods section, a random foil word was chosen as one of the four alternatives to monitor the participant’s general attention. Given the moderate SNR level used (−8 dB), it would be expected for both the audio-only and the audiovisual trials that children receive sufficient auditory information to avoid choosing the random foil. However, when working with children on studies consisting of 170 to 200 trials, limited attention span [8] resulting in some random responses is not surprising. Since all participants responded above chance level, we interpret that children would pick the random answer only due to a lapse of general attention, and not due to them not understanding the task. For both moderated and unmoderated studies, we find that, as expected, when errors are made in the audio-only or audiovisual trials, most of them are closest to the target option (minimal pair errors), followed by vowel errors and only a limited number of random errors (Fig. 2).

FIG. 2.

Boxplots of the percentage of errors compared to the total amount of errors made, for all types of error options (Minimal pair, Vowel, Random) by study (Moderated or Unmoderated). Thick horizontal lines represent medians, gray dots represent means, boxes represent interquartile ranges, and whiskers represent the range, excluding outliers. Individual scores are visualized as small black dots.

To address whether these moments of inattention are significantly different when assessing a moderated or unmoderated online study with children, the percentage of errors are compared to the total errors. There are limitations in just comparing the number of errors itself, since children in the unmoderated study made significantly more errors than in the moderated study. An ANOVA of the linear model (Model: Percentage of errors to total errors ~ study * error type) was used to quantify the main effects of study and error type, and to see if an interaction effect is present indicating a relationship between the error types and moderation used in the study. There is a main effect of error type (F_1,234 = 617.59, p < .001), with having significantly more minimal pair errors than vowel errors (t₂₃₄ = 18.74, p < .001) and significantly more vowel errors than random errors (t₂₃₄ = −4.49, p < .001; see Fig. 2). There is no main effect of study, and no interaction effect. This suggests that although children made overall more errors in the unmoderated study, when looking at the types of errors compared to the total errors made, there was no difference in error patterns between the two studies. More specifically, when we look at the random errors, the mean performance in the moderated and unmoderated studies were the same (~6%).

Thus, when looking at absolute error rate, children made more random errors associated with a lapse of general attention in the unmoderated study, but despite this lower performance, the error pattern stays the same indicating that general attention is not different when assessing an online audiovisual speech perception study that is moderated or unmoderated.

3.3. Moderation and Cross-modal Attention

Our third measure of interest is cross-modal attention. Cross-modal attention is described by measuring how often (in %) a child finds the visual catch-trial throughout the task. Although children were trained to find this catch-trial in phase 2 of the training for both studies, there was a difference in task setup. There are some intrinsic differences between the two studies: the number of catch-trials (20 vs 10), in how children reported the catch-trial (verbally vs manually) between studies, and the moderated study allowed for a repetition of instructions because there were two sessions, whereas the unmoderated study has just one.

Descriptive statistics of these catch-trial counts show that there is a discrepancy in mean performance between the two studies (μ_mod=95.68%, μ_unmod=67.39%). As shown by a two-tailed t-test, the mean score of the catch-trial count is significantly lower in the unmoderated study (t_46.72 = 4.45, p < .001). Interestingly, no participant finds less than 80% of the catch-trials in the moderated study, whereas a fifth of the participants (N=9/46) do not find any catch-trials (0%) in the unmoderated study, and six participants find 50% or less. This discrepancy suggests that it might be more meaningful to compare the medians (Q2_mod = 95%, Q2_unmod = 90%) than means (μ_mod=95.68%, μ_unmod=67.39%). Comparing the median catch-trial counts (by using a Wilcoxon rank sum test) in both studies indicates no statistically significant difference (W = 653, p > 0.05), and shows there is a big discrepancy in catch-trial performance in the unmoderated study (either scoring very low, or very high).

We explored further into this group of children (N = 14) who had low catch-trial counts (< 50%). By comparing the performance on the task itself (audio-only, visual-only and audiovisual trials) of the children who have a catch-trial count <50% and >50% in the unmoderated study we did not find any significant differences in performance in any of the modalities (audio-only: t_19.45 = −0.57, p > .10; visual-only: t_11.27 = 0.73, p > .10; audiovisual: t_17.47 = 0.23, p > .10).

From both moderated and unmoderated studies, we conclude that the cross-modal attention measure is not sufficiently sensitive. In the moderated study, the catch-trial count was too high to correlate with performance, and in the unmoderated study the catch-trial count did not correlate with performance on the speech perception task at all. This is surprising since visual task-irrelevant catch-trials are often used to check attention in-person studies [18]. Adding extra focus on the utility of these catch-trials as well as on task understanding for children in online studies in future work could clarify these findings.

4. Conclusion

The interest in remote studies is burgeoning in the auditory field, but validity measures of remote studies remain scarce. To limit confounds such as attention span, cognitive load and general perceptual development, remote moderation has been suggested especially when working with children [14]. The current work evaluates the influence of moderation on i) remote audiovisual speech perception studies, ii) potential differences in attention, and iii) task understanding in moderated and unmoderated psychophysical tasks with children.

First, we conclude that both moderated and unmoderated remote audiovisual speech perception studies can be valid. Nonetheless, there is a significant influence on performance from the virtual presence of a researcher. Selecting the outcome measure as a difference score, rather than absolute performance could limit moderation effects. Second, a higher number of general attention errors are found in the unmoderated study, but they do not influence the observed error pattern. Therefore, both studies provide us similar information regarding the use of auditory cues to make decisions about the presented stimuli. Finally, cross-modal attention measures might not be as sensitive when assessed online as for in-person studies, and the presence of a moderator is important for task understanding. This points to the need to find other valid ways to assess task comprehension as there is no room for asking questions or re-iterating instructions in unmoderated studies.