Visuals versus textual scales: Optimizing reliability and user experience in observational assessment of parent-child interaction

Abstract

Objective

Assessment of parent-child interaction by practitioners is of great importance but hindered by a lack of instruments that withstand the constraints daily practice places on usage. Visuals may offer an alternative format. Visualizations were tested on reliability, accuracy, and feasibility in observational assessment of parent-child interaction, as alternatives for textual rating scales.

Methods

In Study 1, 95 students rated parent-child interactions with a video or text anchor scale, and in Study 2, 217 professionals rated the same interactions with a decision tree including visual components or a text anchor scale.

Results

Students using the video anchor scale were less reliable and accurate, slower, and had a less positive user experience than students using the text anchor scale. Professionals using the decision tree did not differ in reliability and were comparable in user experience with professionals using the text anchor scale. Rater accuracy showed similar dependency on quality of parental behavior for both scales: ratings were less accurate when the quality of the parent-child interaction was low, and more accurate when the quality was high. However, professionals were less accurate and slower in using the decision tree than the text anchor scale.

Conclusion

With a first iteration of a decision tree performing the same to or only slightly worse, efforts to further develop decision trees might be worthwhile.

Innovation

These nonintuitive findings underscore the value of experimental testing in assessment design in daily practice.

Highlights

• •Practitioners lack feasible instruments for assessment of parent-child interaction.
• •New formats for observational assessment scales are needed in daily practice.
• •A decision tree performed like a text scale, while video based scales did worse.
• •Further developing decision trees might build feasible alternatives to text scales.
• •Experimental testing in assessment design for daily practice can be of value.

1. Introduction

The relevance of parent-child interaction qualities for predicting and understanding developmental outcomes of children is well established (e.g., [33,44,55]). In identifying needs, risks, and protective factors in parent-child interaction, as well as in providing support strategies for this interaction, attachment theory and research has been of great importance. Among the insights from attachment theory and research are three principles for practitioners to use as they seek to support children from high risk families: 1) respecting the child's need for familiar, non-abusive caregivers; 2) preserving the value of continuity in good-enough care; and 3) capitalizing on networks of attachment relationships [17]. In light of the first two principles, parental sensitivity emerged as a central aspect in parent-child interaction. Parental sensitivity refers to the parent's ability to notice child signals, interpret these signals correctly, and respond to these signals promptly and appropriately [2]. It was found predictive of a broad range of positive and negative outcomes, like emotion regulation and cognitive abilities [47], attachment security [34], and internalizing and externalizing behaviors [12]. As a consequence, parental sensitivity has been endorsed by the World Health Organization (WHO) which recommends nurturing care for all young children and providing parents with the necessary support to enhance their sensitivity [60]. Understanding needs for such support as well as risks and resilience within parent-child interaction requires that practitioners can reliably and validly assess the quality of the interaction and, more specifically, parental sensitivity.

In providing practitioners with instruments for assessment of parent-child interaction, the need has been recognized for instruments that withstand the constraints daily practice places on usage [35,49]. Observational instruments provide for reliable and valid assessments of parent-child interaction [25,50,61], whereas self-report measures of parenting behavior have shown weak validity (e.g., [24]). However, despite the broad variety of observational instruments with sufficient psychometric qualities available (see for reviews [19,32,36]), observational instruments for assessment of parent-child interaction have mostly been developed for use in research contexts and by highly trained professionals in clinical practice [19,52,54]. These instruments have been regarded as unfeasible for use in daily practice, due to their time-consuming and costly nature, and high level of training required [13,23,51,52]. It is yet unclear how minimally feasible observational instruments for use in daily practice may be designed. Attempts to develop observational instruments aimed for broad implementation have thus far yielded insufficient evidence for reliability and validity (e.g., [3,15,59]). If the field is to modify existing observational instruments rather than develop new ones, as suggested by Aspland and Gardner [4], Lotzin et al. [32], and Mesman and Emmen [36], altering their format may be an avenue forward.

Traditionally, observational instruments for assessment of parent-child interaction rely on textual representation of focal constructs [4,9,32]. However, reading or memorizing assessment manuals may not be the most efficient, effective, and user-friendly way in which valid and reliable assessment may be practiced. For example, Baker et al. [5] tested an alternative format by providing students with a joystick to rate parent-child interaction during their observation. However, single raters were insufficiently reliable on all three parental behavior constructs. On two constructs, acceptable reliability and validity of parent-child interaction ratings were obtained by averaging scores of six to eight raters, which would be highly unpractical in daily clinical use. Another potential alternative rating scale for observational assessment of parent-child interaction in daily practice may be offered by visualizations.

1.1. Visualization

In search for alternative formats for existing textual observational instruments for assessment of parent-child interaction, studies in education show comprehension and learning can be facilitated by using visual components (e.g., [14,20,37,45,56]). Visuals are suggested to serve a diverse range of functions, such as to activate knowledge, assist comprehension, concretize abstract concepts, and reinforce memory [42]. As Dewan [14] describes, pictures have several advantages over words. For instance, pictures require less effort to recognize and process and are easier to recall. Furthermore, static pictures provide for permanent information, facilitate comparison between graphics, and lessen information overload, whereas animations enable processing of dynamic information [8].

The effectiveness of visual components can be understood from several theoretical perspectives. For example, Dual Coding Theory (DCT) highlights distinct modes of processing and storing of verbal and visual information in linguistic and visual memory representations [11,40,41,48]. Because information is stored in different representations, there are also multiple ways of accessing it. Visualization makes information more concrete, which makes it easier to remember than more abstract information [56]. Furthermore, the Visual Argument Hypothesis [58] states that graphical displays are more effective in communicating complex content than text, because graphical displays are less demanding on cognitive capacities such as working memory [46,56,58].

In their meta-analysis on reading comprehension, Guo et al. [20] concluded that graphics in text aided comprehension. No difference in effectiveness was found between pictures, flow diagrams, and pictorial diagrams. Garcia-Retamero and Cokely [18] concluded on the basis of their systematic review that visual aids tend to improve understanding of health and psychological risks and informed decision making, regardless of skill level. In a meta-analysis on problem solving performance, adding pictures or replacing text with pictures had a small to medium effect on response accuracy, but not on response time [27]. Focusing on concept maps, Nesbit and Adesope [37] found studying concept maps in postsecondary education had a small to medium effect on knowledge retention and transfer.

Comparing dynamic and static visualizations in their meta-analysis, Höffler and Leutner [26] found a small to medium effect of instructional animations over static pictures in learning outcomes. Moreover, highly realistic animation, such as animation based on video, had a strong effect. Following this meta-analysis, Berney and Bétrancourt [8] made a comparison of 140 pair-wise animated versus static visualizations in multimedia instructional material. Focusing on expository animations, they found a small effect of animation over static graphics in learning. Animation was more effective than static graphics for gaining conceptual knowledge and for understanding and applying this knowledge. Within the animations, a small effect on learning was found for iconic visualizations, among which photo realistic pictures, and a large effect when the animation did not include additional textual information.

In sum, a substantial evidence base supports visuals as an aid to a wide range of cognitive processes. The extent to which visuals harbor unused potential for facilitating assessment of parent-child interaction is yet unknown, as no previous study has focused on incorporating visual components in observational instruments for use in daily practice.

1.2. This study

The general aim of this study was to test visual enhancements of a traditional, textual observational instrument for assessment of parent-child interaction by practitioners. Moreover, to aid use of observational assessment of parent-child interaction in daily practice, we aimed for a positive user experience and time-efficiency. To test the hypothesis that visual anchor scales provide for a higher interrater reliability, a better accuracy and user experience, and a quicker assessment time than text anchor scales, we conducted two consecutive studies comparing visual formats to a traditional textual format.

2.1. Study 1

In Study 1 we compared a traditional text anchor scale with a video anchor scale. Using a non-interactive, representational, and highly-realistic animated format might maximize effectiveness [26]. Also, this format might aid feasibility by providing for a good user experience, a quick assessment, and a means of communication with families, because video-based descriptions of parental sensitivity might be easier to understand than textual descriptions. In interventions focusing on parental sensitivity, video instruction and feedback are important working ingredients for enhancing parental sensitivity [6,7].

In Study 1, we hypothesized that a video-based anchor scale as alternative format would provide for a quicker, more reliable and accurate observational assessment of parental sensitivity with a better user experience than a traditional, text anchor scale.

2.2. Methods

2.2.1. Participants

Participants were students of Psychology or Educational Sciences at the Vrije Universiteit Amsterdam. Recruitment mainly took place through their online study platform, and the study was promoted during lectures. First year undergraduates received credits for participating in psychological research—an obligatory part of their education. Second, third, premaster or master year students received 10 euros. Ethical approval was granted by the Scientific and Ethical Review Board of the faculty, file number VCWE-2017-134.

Of 95 students, 64 (67.4 %) were first year undergraduates, and 31 (32.6 %) were in their second, third, premaster or master year. Eighty-six (90.5 %) were female. Participants were between 17 and 47 years old with a mean of 21.3 (SD = 4.9) years.

2.2.2. Procedure

The study was part of a research project aiming to develop an observational instrument for assessment of parental sensitivity for use in daily practice. The project team consisted of scientists, professionals working in daily practice, policy makers, and an expert by experience, i.e., a parent who was, at that moment, receiving child welfare services. Ideally, professionals working in daily practice would have been recruited as participants. However, the limited time available for professionals, the burden placed on professionals, and feasibility of all studies incorporated in the research project were major resource constraints. For these reasons, the project team decided to include students as a test population in the first trial study.

Recruitment and participation took place from December 4th 2017 till January 12th 2018. To join the study, students reserved a time slot, received information on the study, and signed an informed consent form and a confidentiality agreement. Students used a computer and headphones in a private space at the university. Although the study was promoted throughout the data collection, the desired sample size was not reached after six weeks of data collection due to a lack of students willing to participate.

To randomize the participants to two conditions, the software in which the research task was built randomly appointed participants to one of two conditions—a textual and a video condition—by a ratio of 1:1. Forty-eight students were randomized to the text condition, and 47 in the video condition. One student did not complete the study, because of computer failure. Ninety-four students were included in the primary analyses.

Participants answered questions about demographics, read a Dutch translation of the National Institute of Child Health and Human Development Study of Early Child Care and Youth Development (NICHD SECCYD) Parental Sensitivity (PS) rating scale [39], and answered three quiz questions about the text they read. The quiz questions are provided in the Supplemental Materials. To avoid interaction between the scale format factor and the particular training, no further training component was provided. Subsequently, participants were instructed on rating parental sensitivity. Each participant received the same set of target parent-child interaction recordings for rating, containing six videos of 5 min each showing a mother with her 24-month old child. Participants observed each target video and then rated parental sensitivity according to either a Dutch translation of the original text anchor PS-scale or a video anchor scale, depending on the condition they were randomized in. Lastly, participants answered questions about their user experience in rating parental sensitivity with the text or video anchor scale.

The videos with parent-child interactions were derived from the Generation² longitudinal cohort study (see [29]). This study (2009–present) aims to examine the development of parenting and mental health. From eleven different videos ranging from very low parental sensitivity to very high parental sensitivity a selection of five minutes was made. Videos were coded with the PS scale by two expert coders who reached full agreement on six videos. These six videos showed five minutes of the Three-boxes assessment [38], in which the mother and child were given three stacked boxes with toys to open in a set order from the top box down and to play with during 15 min.

2.2.3. Measures

2.2.3.1. Parental sensitivity

Parental sensitivity was measured with the National Institute of Child Health and Human Development Study of Early Child Care and Youth Development (NICHD SECCYD) Parental Sensitivity (PS) text anchor rating scale [39]. This scale focuses on aspects of parental sensitivity, like child- or parent-centered interactions, well-timed and paced responses to the child or the lack thereof, and granting of control and autonomy versus intrusive interactions. The scale consists of a 7-point Likert scale ranging from Very Low Sensitivity to Very High Sensitivity. The adapted visual anchor scale was comprised out of seven short video clips depicting each NICHD SECCYD PS rating scale anchor and showing a variety of parental behaviors that are sensitive or insensitive. A description of each video clip is provided in the Supplemental Materials. The video clips were selected from the Generation² longitudinal cohort study video materials by an expert coder (first author) of the PS scale.

2.2.3.2. User experience

User experience was measured through an adaptation of the Dutch version of the User Experience Questionnaire (UEQ; [31]). The UEQ is a 26-item questionnaire, leading to a total score, and scores on six subscales: Attractiveness (6 items), Efficiency (4 items), Perspicuity (4 items), Dependability (4 items), Stimulation (4 items), and Novelty (4 items). Items are comprised of two extremes, for example “not understandable” versus “understandable”, “inefficient” versus “efficient”, and “inferior” versus “valuable”.

In the current study, seven items were excluded, because these did not apply to the research task, for instance “not secure” versus “secure”. The 19 items remaining were used to calculate an overall user experience score. Further, only three of six scales could be used, that is Efficiency, Perspicuity, and Stimulation. Out of the other three scales items were missing. The Efficiency scale measures the extent to which using the scale is without unnecessary effort, for instance by the item “inefficient” versus “efficient”. The Perspicuity scale measures ease of learning to apply the scale, for instance by the item “confusing” versus “clear”. The Stimulation scale measures the extent to which using the instrument is exciting and motivating, for instance by the item “not interesting” versus “interesting”. The original 7-point Likert scale was transformed into a Visual Analogue Scale (VAS) ranging from −50 to 50 to provide for more variation within the scale anchors. Participants could slide a pointer from the middle position to any given point on a line in between the two extremes. Scoring was non-visible for participants. Overall user experience score was calculated by dividing the sum of all questions by 19. Each scale score was calculated by dividing the sum of the four items by 4.

2.2.4. Data analyses

Data analyses were performed using IBM SPSS Statistics 25. First, to estimate interrater reliability, intraclass correlation—type two-way random effects, absolute agreement, single rater [28]—was calculated between participant ratings and the expert ratings. Intraclass correlations were added into the main dataset per participant to enable further comparison between the text and video condition. Second, to estimate rater accuracy the mean absolute difference score between participants and the expert rating was calculated. Third, to limit the effect of prolonged time measurement because of computer problems—i.e., in some instances the computer took a very long time loading the next video that was to be rated—assessment time was winsorized per video. This meant that the time of identified outliers was set to the time of the highest non-outlier. Fourth, one-way analysis of variance (ANOVA) was used to test the difference between the textual and video condition in mean intraclass correlation, rater accuracy, user experience, and assessment time.

2.2.5. Power analyses

An a priori power analysis and a sensitivity analysis were performed in the power analysis software G*Power [16]. The a priori power analysis showed a desired sample size of 64 participants per condition for an alpha of .05, power of .80 and an effect size of d = 0.50—based on small to large effect sizes for dynamic and highly realistic animation [26], and small to medium effect sizes for response accuracy [27]. The sensitivity analysis showed the study was sufficiently powered to detect an effect size of d = 0.58 for the obtained sample size of 48 for the text condition and 47 for the video condition with an alpha of .05 and power of .80.

2.3. Results

Table 1 shows the means, standard deviations, and ANOVAs of the text anchor scale versus the video anchor scale for intraclass correlation, absolute difference score with the expert rating, overall user experience, efficiency, perspicuity, stimulation, and assessment time. All but one one-way ANOVAs were statistically significant (p < .05), and showed that the text anchor scale provided for a higher interrater reliability, a higher rater accuracy, a better overall user experience, a higher efficiency, and perspicuity in using the scale, and a quicker assessment. Only stimulation in using the scale was not statistically different between the text anchor scale and the video anchor scale.

Table 1.

Means, standard deviations, and one-way analyses of variance of textual versus video anchor scales in study 1.

	text		video		F(1, 92)	p	d
	M	SD	M	SD
Interrater reliability	.77	.18	.67	.25	6.12	.015	0.51
Difference with expert	1.05	0.47	1.29	0.54	5.19	.025	0.46
User experience
Overall	17.31	11.65	8.16	15.83	10.19	.002	0.67
Efficiency	15.81	13.15	−0.80	18.95	24.40	<.001	1.03
Perspicuity	19.03	16.10	7.03	17.87	11.69	<.001	0.70
Stimulation	19.46	13.35	13.43	17.32	3.58	.062	0.41
Time	36.39	4.98	38.79	3.88	6.73	.011	0.55

Note. Homogeneity of variances could not be assumed for Interrater reliability, therefore Welch F(1, 82.98), and Overall User experience, Welch F(1, 84.54) were computed.

2.4. Discussion

In Study 1 the NICHD SECCYD Parental Sensitivity text anchor rating scale was compared with video-based descriptions of this rating scales anchors. Results showed that students aided with video anchors performed worse than those aided with text anchors. Video anchors led to ratings that were less reliable and accurate, took more time, and resulted in a less positive overall user experience, efficiency, and perspicuity. Thus, in assessment of parental sensitivity, no support was found for our hypothesis that video clips depicting scale anchors provide for a higher interrater reliability, a better accuracy and user experience, and a quicker assessment time than a text anchor scale.

With parental sensitivity being a complex, multifaceted construct [36], the provision of one exemplar of parental sensitivity per scale anchor might have led students to look for the exact behavior portrayed in the video clip. More specifically, Mesman and Emmen [36] emphasize Ainsworth's original sensitivity scale, that requires making one global assessment, rather than evaluating specific maternal behaviors that contribute to the sensitivity construct. Video clips might have tapped into such specific maternal behaviors when parental sensitivity was low as well as when it was high, impeding the formation of the global assessment. This might have resulted in a lack of generalizability—video anchors might not have been representational [26]. Because previous studies on enhancement of learning [26], and processing of dynamic information [8] did find effects of animation versus static pictures, further research on video clips might yield positive results in assessment of more concrete and specific behaviors in parent-child interaction, for example parental warmth [57] and harsh parenting [43].

In conclusion, in this form video anchors do not provide for an effective alternative format for traditional textual scales for assessment of parental sensitivity. A visual format needs to aid translation of general aspects of parent-child interaction to unique observations. Therefore, in Study 2 an alternative visual format potentially more fitting to these requirements was tested.

3.1. Study 2

In Study 2 we compared the NICHD SECCYD PS text anchor rating scale with a decision tree including visual components for internal and leaf nodes depicting the NICHD SECCYD PS rating scale. Decision trees can be regarded as an interpretation visualization. These type of visualizations combine elements of representational visuals—that provide concreteness to abstract concepts—, and organizational visuals—that categorize information [42]. In their meta-analysis on clinical judgement versus statistical prediction in mental health, Ægisdóttir et al. [1] described decision trees as a tool to improve accuracy in assessments made by practitioners. In addition, visual internal and leaf nodes might not only aid comprehension, but also enable explanation to parents, thus increasing its suitability for supporting families.

In Study 2, we hypothesized that a decision tree with visual components leading to scale anchors as alternative format would provide for a quicker, more reliable and accurate assessment of parental sensitivity with a better user experience than a traditional, text anchor scale. Also, we aimed to gain insight in the influence of the degree of parental sensitivity on accuracy of assessments, and to examine equivalence between the two text condition groups in Studies 1 and 2.

3.2. Methods

3.2.1. Participants

Participants were professionals from Jeugdbescherming Regio Amsterdam (Youth Protection Amsterdam Area). All professionals of Youth Protection Amsterdam Area who directly worked with families were asked to participate. Professionals who participated, received a snack coupon for the canteen. As in Study 1, ethical approval was granted by the Scientific and Ethical Review Board of the VU, file number VCWE-2017-134.

Of the 217 professionals participating, 184 (84.8 %) were case managers including 8 interns, 22 (10.1 %) were psychologists including 5 interns and 11 (5.1 %) were team managers. One hundred fifty-four (71.0 %) professionals had a university college degree, and 63 (29.0 %) a master's degree. One hundred eighty-three (84.3 %) professionals were female. Professionals were between 18 and 65 years old with a mean of 36.2 (SD = 10.5) years. Their years of work experience varied between 0 and 38 years with a mean of 6.7 (SD = 7.4) years.

3.2.2. Procedure

Results of the first study were discussed in the focus group meetings of the project team of the research project. Next, the project team decided to include professionals instead of students in the second study to increase the informational value of the findings for the next step in the project, to develop and test an assessment tool for practice.

Recruitment and participation took place from March 5th 2018 till May 3th 2018. Professionals were given information about the purpose and design of the study. If they chose to participate, they signed an informed consent form and a confidentiality agreement. In the meeting room, they individually participated in the research task on a laptop provided by the researchers. Because data was collected during team meetings, all professionals were present, except those on leave or otherwise absent. If professionals did not want to participate, they could leave the meeting room or work in silence on their computer. Most professionals chose to participate, leading to a higher number of participants recruited than would have been necessary according to the a priori power analysis.

The following procedure, including randomization, test conditions, instruction, and format, was identical to the one used in Study 1. One hundred ten professionals were randomized to the text condition, and 107 to the decision tree condition. Five professionals did not complete the study, because of computer failure or an acute situation in their work. Two hundred twelve professionals were included in the primary analyses.

3.2.3. Measures

3.2.3.1. Parental sensitivity

In the textual condition parental sensitivity was measured identical to measurement in Study 1. In the decision tree condition professionals received a decision tree with illustrations depicting the different constructs from the original textual scale for internal and leaf nodes leading to the seven scale anchors from the original textual scale. Internal nodes are elements within the tree that contain information, and from where one or more other branches extend; leaf nodes are elements in the last layer of the decision tree from where no more branches extend, and that contain information depicting a class label or, in this study, scale anchor. In a maximum of three steps, users of the decision tree were led to a scale anchor. The decision tree was constructed by an expert coder (first author) of the PS scale in collaboration with two other expert coders (co-authors) using online illustration software and PowerPoint. Fig. S1 shows the decision tree.

3.2.3.2. User experience

User experience was measured identical to measurement in Study 1.

3.2.4. Data analyses

Data analyses were identical to analyses in Study 1. In addition, influence of the degree of parental sensitivity on rater accuracy was assessed through a mixed between-within subjects ANOVA, with the text condition versus the decision tree condition as between-subjects, and rater accuracy on the degree of parental sensitivity as within-subjects effect. After results were in, it was decided to test for equivalence for results on the ANOVAs that were nonsignificant, in order to establish a better understanding of possible similarity of the two conditions. Equivalence was examined using a two one-sided tests (TOST) procedure with the Equivalence_Tests_TOSTER Excel-spreadsheet provided by Lakens [30]. Using the previously performed sensitivity power analysis, the smallest effect size of interest (SESOI) was set at d = 0.38 [30].

Also, additional data analyses were performed to test for equivalence between the two text condition groups in Studies 1 and 2. Again, a two one-sided tests (TOST) procedure was applied using the Equivalence_Tests_TOSTER Excel-spreadsheet [30]. Using the sensitivity power analysis in G*Power [16] to determine the SESOI [30], the SESOI was set at d = 0.49, based on the obtained sample size of the text condition in Study 1 of 48 and in Study 2 of 110, an alpha of .05 and a power of .80.

3.2.5. Power analyses

The a priori power analysis was identical to the one used in Study 1, thus showing a desired sample size of 64 participants per condition. The sensitivity analysis showed the study was sufficiently powered to detect an effect size of d = 0.38 for the obtained sample size of 110 for the text condition and 107 for the decision tree condition with an alpha of .05 and power of .80.

3.3. Results

Table 2 shows the means, standard deviations, and ANOVAs of the text anchor scale versus the decision tree anchor scale for intraclass correlation, absolute difference score with the expert rating, overall user experience, efficiency, perspicuity, stimulation, and assessment time. One-way ANOVAs for absolute difference score with the expert rating and time were statistically significant (p < .01), and showed that the text anchor scale provided for a higher rater accuracy, and a quicker assessment. One-way ANOVAs for intraclass correlation and all user experience measures did not show significant effects, thus, equivalence was examined for these outcomes. The TOST procedure showed that equivalence of the text anchor scale and the decision tree anchor scale was not statistically significant in intraclass correlation (t(214.88) = −0.91, p = .182) and in perspicuity (t(207.41) = −1.26, p = .104). Overall user experience (t(213.60) = 2.62, p = .012), efficiency (t(212.56) = 2.42, p = .008), and stimulation (t(214.58) = 2.05, p = .021) were statistically equivalent, suggesting overall user experience, and efficiency and stimulation in using the scale were comparable for the text and the decision tree anchor scale. Interrater reliability, and perspicuity in using the scale were neither statistically different nor statistically equivalent for the text and the decision tree anchor scale.

Table 2.

Means, standard deviations, and one-way analyses of variance of textual versus decision tree anchor scales in study 2.

	text		decision tree		F(1, 210)	p	d
	M	SD	M	SD
Interrater reliability	.76	.20	.71	.19	2.45	.119	0.20
Difference with expert	1.07	0.47	1.25	0.44	8.19	.005	0.41
User experience
Overall	12.11	12.97	13.08	13.68	0.28	.596	0.00
Efficiency	11.72	13.27	12.43	14.37	0.14	.710	0.00
Perspicuity	13.29	14.29	10.03	16.86	2.32	.129	0.20
Stimulation	14.28	17.43	16.00	16.22	0.55	.458	0.00
Time	35.52	2.63	41.70	3.58	203.42	<.001	2.00

Note. Homogeneity of variance could not be assumed for Time, therefore Welch F(1, 186.71) was computed.

Fig. 1 shows a mixed between-within subjects ANOVA for rater accuracy. There was no significant interaction effect between condition and degree of parental sensitivity (Wilks' Lambda = .96, F(5, 206) = 1.66, p = .15, ${\eta }_p^2=.04$). A higher absolute difference with the expert rating was found for both scales when parental sensitivity was low whereas a lower absolute difference with the expert rating was found for both scales when parental sensitivity was high (multivariate tests: F(5, 206) = 9.81; p = .000; ${\eta }_p^2=.19$). Further, there was a significant difference in rater accuracy between the text and decision tree condition (between groups: F(1,210) = 8.19; p = .005; ${\eta }_p^2=.04$).

Fig. 1.

Rater accuracy versus degree of parental sensitivity.

Note. Multivariate tests: F(5, 206) = 9.81; p = .000; ${\eta }_p^2=.19$. Between groups: F(1, 210) = 8.19; p = .005; ${\eta }_p^2=.04$.

3.3.1. Equivalence between text condition groups

The TOST procedure showed students in the text condition of Study 1 and professionals in the text condition of Study 2 were statistically equivalent in interrater reliability (t(98.90) = −2.57, p = .006) and rater accuracy (t(89.61) = 2.58, p = .006). Students in the text condition of Study 1 and professionals in the text condition of Study 2 were not statistically equivalent in overall user experience (t(99.09) = −0.39, p = .348), efficiency (t(90.36) = −1.03, p = .152), perspicuity (t(80.75) = −0.63, p = .266), and stimulation (t(115.40) = −0.94, p = .174) in using the text anchor scale, and assessment time (t(56.19) = −1.43, p = .08).

3.4. Discussion

In Study 2 the NICHD SECCYD PS text anchor rating scale was compared with a decision tree including visual components for internal and leaf nodes leading to this rating scale's anchors. Results showed that professionals aided with a decision tree anchor scale performed the same to slightly worse than those aided with text anchors, in contrast with our hypothesis. The decision tree anchor scale led to ratings that did not statistically differ in reliability with the text anchor scale—both scales showed sufficient reliability [10]—and in perspicuity in using the scale. Even more so, overall user experience, efficiency, and stimulation in using the decision tree anchor scale were statistically comparable with using the text anchor scale. Both the decision tree anchor scale and the text anchor scale led to less accurate ratings when parental sensitivity was low, and more accurate ratings when parental sensitivity was high. However, the decision tree anchors led to ratings that were less accurate, and cost more time.

In comparing findings with previous studies, previous studies showed that graphics and flow diagrams aid comprehension [20], concept maps promote knowledge retention [37], and visual components strengthened recognition and process [14], and lessened information overload [8]. Although a text scale might be more specific, these features of the decision tree might have aided assessment of a complex construct like parental sensitivity, contributing to the finding on good reliability that did not differ from the text anchor scale.

However, rater accuracy was higher for the text scale than the decision tree with visual components. Research suggesting decision trees can improve accuracy has not compared different formats of instruments [1]. Nevertheless, our findings were not in line with the effect of adding or replacing text with pictures found by Hu et al. [27], and, if decision trees are further developed for assessment of parent-child interaction, special focus should be placed upon accuracy, for instance through accompanying training. Further, Hu et al. [27] found no effect in response time; this study showed assessment with the traditional textual scale was quicker than with the decision tree. The format of the research task might have caused this, because the decision tree required several steps and decisions, while using the textual scale required only one decision in one step.

Thus, the decision tree anchor scale did not outperform a traditional, textual scale, yet findings showed good interrater reliability and a positive user experience.

4. General discussion and conclusion

4.1. Discussion

The studies reported aimed to test visual alternatives for traditional textual scales. In both studies visual scales did not aid observational assessment of parent-child interaction better than the traditional, textual scale. Moreover, in Study 1 video clips did not lead to reliable and accurate assessments, nor to a positive experience of users in comparison with the textual scale. However, in Study 2 the decision tree performed the same to slightly worse than the textual scale. These findings are notable. The traditional, textual scale—the NICHD SECCYD Parental Sensitivity rating scale—used in both studies has been widely used and showed its worth in assessment of parental sensitivity in research contexts [36], whilst the decision tree was developed for this study and tested therein for the first time. It is therefore noteworthy that interrater reliability did not differ between the decision tree and the textual scale, user experience was comparable for both scales, and rater accuracy showed the same dependency on the degree of parental sensitivity. Moreover, even though the text scale performed sufficiently in both studies—for instance by showing reliability—, results are based on scores of the group, and not on individual participants. Characteristics precluding use of instruments like the NICHD SECCYD rating scales still hinder their use in daily practice—such instruments are time-consuming and costly, and require a high level of training [13,23,51,52]—and attempts to develop textual scales for assessment of parent-child interaction in daily practice have not been successful thus far (e.g., [3,15,59]). However, the need for observational instruments for assessment of parent-child interaction that can be reliably used by practitioners is high. For example, in a study by Hammarlund et al. [22] the vast majority of a group of child protection workers expressed their need for forming an opinion of attachment quality—a construct related to parental sensitivity assessed with textual scale formats [53]—as part of investigations concerning children's needs and parents' caregiving capacities, and as a basis for decisions to place the child in out-of-home care. Moreover, most child protection workers formed opinions about attachment quality, while the vast majority did not use systematic assessment instruments. Thus, with the importance to move towards including assessment of parent-child interaction in daily practice and the findings of Study 2, efforts to further develop the decision tree might be worthwhile. Results of such efforts should be tested on reliability and validity of that decision tree. Also, further research should examine feasibility of using alternative formats in working with families.

Furthermore, in Study 2 both anchor scales led to less accurate ratings when parental sensitivity was low, and more accurate ratings when parental sensitivity was high. This differentiation in rater accuracy depending on the degree of parental sensitivity was consistent with findings of Wilson et al. [59], who found high levels of agreement among health visitors in ratings of positive parental interactions and low levels of agreement in ratings of a mix of positive and negative parental interactions. In line with this, Elmer et al. [15] found health visitors to rate certain aspects of low parental sensitivity higher than other aspects of low parental sensitivity. Perhaps professionals tend to focus on positive aspects when they try to encourage families and keep their working alliance intact [21]. Further development of observational instruments for assessment of parent-child interaction in daily practice and training in such instruments therefore needs to grapple with differential rating accuracy. Overestimating low quality parent-child interactions will result in instruments not identifying families in need of support, and not facilitating adequate decision making.

Further, comparison of Study 1 to 2 showed that the study populations were similar in important aspects, i.e., interrater reliability and rater accuracy. Thus, findings of Study 1 and 2 can be compared on these outcomes. However, students and professionals were not comparable in their user experience and the time it took them to use the text anchor scale. The lack of equivalence between groups of participants in both studies on these outcomes is a limitation.

4.2. Innovation

Although existing text-based instruments for assessment of parent-child interaction have mostly been considered unfeasible for use in daily practice (e.g., [23,51,52]), innovative approaches to instrument development formats have rarely been a focus in research. In translating science to practice, alternative formats to traditional textual instruments are presented in this study by using visualizations, in particular video clips and a decision tree with visual components for leaf and internal nodes. Developing and testing alternative assessment designs could improve the feasibility and effectiveness of a wide range of instruments that practitioners across fields currently cannot use or use effectively. Also, as an alternative design to traditional textual instruments, the visual format in this study provides a promising basis for tools that aid practitioners in working transparently and engaging with their clients. Realizing this potential will require steps for further development and testing for reliability and validity in daily practice.

4.3. Conclusion

In conclusion, a first iteration of a decision tree performed similar to or only slightly worse than its well-developed yet unfeasible textual counterpart. Conversely, whilst visuals aid a wide range of cognitive processes (e.g., [8,18,20]), video clips depicting scale anchors clearly did not enhance observational assessment of parental sensitivity. With unsuccessful attempts to enhance textual scales towards including assessment of parent-child interaction in daily practice thus far, efforts to further develop decision trees for including observational assessment of parent-child interaction in daily practice appear worthwhile.

CRediT authorship contribution statement

Mirte L. Forrer: Writing – original draft, Visualization, Resources, Project administration, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Carlo Schuengel: Writing – review & editing, Supervision, Resources, Methodology, Conceptualization. Mirjam Oosterman: Writing – review & editing, Supervision, Resources, Methodology, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

Quiz Questions and Visual Anchor Scales