Comparing Two Cut-off Based Criteria While Investigating the Discriminatory Characteristics of a Tablet-Based Dyscalculia Screening Battery for 5–9 Age Group

Abstract

Introduction:

Math skills are essential in academic, occupational, and scientific areas as well as in daily life activities. There are two existing models in screening dyscalculia: IQ-Achievement Discrepancy Model (DM), and Response to Instruction (RTI) model. In these models, two different cut-off based criteria (being 1 or 2 standard deviation below within their age groups, and being in the lower 5–10% group) are mainly used. The main purpose of this study is to compare these two different cut-off criteria, based on three sub-tests of Tablet-PC Based Dyscalculia Screening Battery (TAB-DSB) (Canonic Dot Counting-CDC, Symbolic Number Comparison-SNC and Mental Number Line-MNL). It is expected to show which criteria would yield the best discrimination in differentiating students who have dyscalculia tendency from the rest of the students in three sub-tests.

Methods:

The participants of the study included 316 volunteer 1st, 2nd, and 3rd year elementary school students.

Results and Conclusion:

The results indicated that CDC, SNC, and MNL (MNL1, MNL2) tasks failed to discriminate the groups when the standard deviation rule (below, above, and within 1 standard deviation) was taken into consideration. On the other hand, these tasks were found to be effective in discriminating the groups when the lower 10% and the upper groups were compared.

INTRODUCTION

Math skills are essential in academic, occupational and scientific areas as well as in daily life activities. Nevertheless, many children experience math learning difficulties at school. Dyscalculia is defined as a specific learning difficulty which negatively affects the learning of number related concepts and procedures (1). While some researchers report that 5% of school-aged children have dyscalculia (2, 3); some point out that this number varies between 6% and 14% depending on the tools that measure dyscalculia (4).

Although there is no single reason for it, developmental dyscalculia has certain characteristics: when compared with their peers, dyscalculic children experience difficulty in calculating numbers, understanding concepts related to number, and numbers themselves. Although they have normal intelligence and normal academic achievement in other fields, these children fail in arithmetic (and/or fall behind their peers at least two years), and apply more primitive arithmetic strategies.

The success of any effective instructional intervention for dyscalculic individuals depends heavily on determining the basic reasons underlying the issue. Therefore, contemporary research put a lot of emphasis on core skills, early screening, and basic number competencies (5). Some of those research investigating the underlying reasons of developmental dyscalculia suggest that humans have a basic core cognition system, and the number module which helps humans learn mathematics is among this core system (6). Known also as number sense, this module is considered as the base for numbers as in the form of exact and approximate. The insufficiency and/or misconnection between numbers, and their symbolic representations are also shown to be another major reason behind the math learning difficulty.

There are two models in screening dyscalculia: IQ-Achievement Discrepancy Model (DM), and Response to Instruction (RTI) model (1). DM differentiates dyscalculic children from those who have poor performance in general, based on intellectual abilities, and math achievement test. In RTI Model, it is important to follow if the child learns mathematical concepts and operations in the same speed with his peers. The child is being observed throughout the whole education period determined between a specific time in this model.

Both of these models have limitations in various points. DM is considered to be inadequate in that intelligence tests are sensitive to socio-economic status, hence those with lower intelligence scores tend to experience learning difficulties; in addition, the greater part of intelligent tests includes mathematical skills (4, 7, 8), the average scores in math achievement tests could be misleading, and/or standardized tests are limited in what they measure (6, 9). Another disadvantage is that using this approach could lead to being late in diagnosis of dyscalculia (8) because early diagnosis is important in intervention of the problem before it gets worse. In addition to this, DM approach is not informationally efficient in terms of intervention because it does not provide any contribution about how to cope with this disorder (10). Although RTI model is more dominant approach with respect to DM for both practitioners and researchers, some important issues related with this approach should be taken into consideration (10). Like the DM approach, the RTI approach is problematic in validity of classification (10). The cut-off point for differentiating a dyscalculic child from other students is unclear. Moreover, during implementation of intervention or measurement of response, validated tools and materials are needed (11). To conclude, the varieties in traditional dyscalculia screening tools, and their criteria which these tools are based upon make it difficult to reach a theoretical framework; therefore, there is a need to explore new tools and/or more objective methods to determine those who have tendency to learning difficulties.

In those models mentioned above, two different cut-off based criteria (for detailed reviews, see references 1 and 12) are used. One of them is to apply the mean score of 1 to 3 standard deviation below within their age groups (13, 14). The other is to determine the lower 5–10% group (15), or 30% group (16). In the present study, two of these cut-off approaches were compared: one is being 1 standard deviation below within their age groups (mean of CPT score), and the second is the lower 10% group (according to CPT score) cut-off criteria (sequentially; 14, 15) independent from their intelligence.

In the field of learning disorders, studies related with dyscalculia are pretty scarce. Furthermore, besides dyscalculia related studies, there is no study reported regarding the comparison of two models in Turkey. The main purpose of this study is to compare two different cut-off criteria based on three sub-tests of Tablet-PC Based Dyscalculia Screening Battery (TAB-DSB) (Canonic Dot Counting, Symbolic Number Comparison, and Mental Number Line) in Turkish school age children. In other words, it is expected to show which cut-off based criterion would yield the best discriminatory characteristic in differentiating the students who have tendency to dyscalculia in three sub-tests. Secondly, by reporting the results from a country where no studies reported yet, it was aimed to contribute to the cross-cultural understanding of dyscalculia for further studies.

METHODS

Participants

The participants of the study included 316 voluntary 1st, 2nd, and 3rd year elementary school pupils, at ages between 5–9 (X_=7.26 Std=1.03), among which were 158 females and 158 males. All participants were within normal IQ ranges for their ages. The intelligence levels of participants were measured by Raven Standard Progressive Matrices Test (RSPM) (17). According to this test’s results, those at the middle range group had a mean score of 30.37 (8.19); those at the higher group had 41.70 (6.58). All participants and their parents were informed about the study, and asked for their consent. Students were recruited randomly from 12 public-funded schools in different socio-economic level representative regions in a metropolitan city. The ethical permissions were sought from the university ethical council where the first author was affiliated with. Based on school records, students who have diagnosed general learning difficulty, reading disorders, and ADHD were excluded from the study. Students with lower IQ (102 students, 32.3%) were also excluded. Threshold for lower IQ was determined depending on the Turkish school age norm study of RSPM conducted by Düzen, Şahin, Raven J, and Raven CJ (18). Teachers’ evaluations about the school performances of students were obtained.

Data Collection Tools

Calculation Performance Test (CPT): CPT was developed by De Vos (19) and adapted into Turkish by Olkun, Can, and Yeşilpınar (20). CPT consists of basic arithmetic operations (addition, subtraction, multiplication, and division). At the first grade, the test includes 20 items of addition only; for second graders, there are 80 items, of which are 40 additions and 40 subtractions; for the third graders, there are a total of 200 items, of which are 40 additions, 40 subtractions, 40 multiplications, 40 divisions, and 40 mixed arithmetic operations. KR-20 reliability co-efficiency of the test was 0.95 with time constrained administration, and 0.98 at the absence of a time constraint.

Raven Standard Progressive Matrices Test (RSPM): RSPM, developed by Raven et al. (17), is a neuropsychological test which measures academic success, cognitive processing speed, visuospatial perception, analytical reasoning, and clear-thinking ability. It offers insight about someone’s capacity to observe, solve problems, and learn. In addition to general intellectual ability score (g score), RSPM is also known to be used to measure one’s visual-spatial ability (17, 21). The test was standardized for Turkish school-age population (18), and consists of 5 sets with 12 items in each set with a total of 60 items, increasing from simple to difficult levels. In each set, participants were required to identify relevant figures, determine the characteristics of each figure based on the given relations, develop a systematic inquiry approach, and establish a new thinking mindset. In each task, participants were provided a figure with a missing part, and were asked to find the best piece to complete the figure among the provided choices. There are 6 choices in each of the first two sets; the following three sets include 8 choices each. There is no time constraint to complete the test. In this study, the completion time was between 35–40 minutes. The test can be administered either in a group, or individually. This general intellectual ability test requires pervasive functions of brain regions (17, 21).

Tablet PC-based Dyscalculia Screening Battery (TAB-DSB) Sub-Tests: TAB-DSB was developed to screen the tendencies toward dyscalculia for elementary school students (22). The TAB-DSB contains three sub-tests, which were reported as the best to discriminate dyscalculia. These sub-tests are: Canonical Dot Counting (CDC), Symbolic Number Comparisons (SNC), and Mental Number Line (MNL) which had two sub-tasks and named as MNL1 and MNL2. The reported KR-20 reliability coefficients for CDC, SNC, MNL1, and MNL2 were significant at moderate level (respectively; 0.69; 0.79; 0.75; 0.72).

Content validity was assessed based on expert opinions in the fields of mathematics education, special education, and educational psychology. In order to ensure criterion validity, a correlation analysis was run between Math Achievement Test (23), and CDC, SNC, MNL1 and MNL2 scores. The results were significant, negative, and at moderate level (respectively; 0.356, p<0.000;-0.449, p<0.000; -0.276, p<0.000; -0.531, p<0.000). A partial correlation analysis was run, and it was found that CDC, SNC, and MNLtotal (summation of MNL1 and MNL2 scores) were effective in predicting Math Achievement Test scores (respectively; -0.222, p<0.014; -0.220, p<0.015; -0.356, p<0.000) (22). In all tests (CDC, SNC, MNL1, MNL2), black, white, and blue colors were used, the font in numbers was Calibri consistently. Before proceeding to the test, a practice session was presented in each sub-test. Sample items from the sub-tests are provided in Figure 1.

Figure 1a-c.

Sample items from the three sub-tests CDC (Canonic Dot Counting) (a), SNC (Symbolic Number Comparison) (b), MNL2 (Mental Number Line 1–100) (c).

Canonic Dot Counting (CDC): This task required both perceptual and conceptual subitizing (24). Dots ranging from 3 to 9 were arranged into dice or domino like patterns. Students are requested to enter their responses by touching a number ordered left to right from 1 to 9. Before starting to the task, they were instructed to touch on the correct number below the screen compatible with the number of dots that exist. There are 21 items in this test (Table 1).

Table 1.

Number of Items in each sub-tests used in TAB-DSB

Sub-Tests		Grade
		1 # items	2 # items	3 # items
1	Canonic Dot Counting (CDC)	21	21	21
2	Symbolic Number Comparison (SNC)	36	36	36
3	Mental Number Line (MNL)
	0–10 (MNL1)	9	9	9
	0–100 (MNL2)	24	24	24
Total		90	90	90

Symbolic Number Comparison (SNC): This sub-test consisted of Arabic number comparison tasks arranged in accordance with numerical Stroop paradigm. Numbers from 3 to 9 were arranged in a pseudo random order. Students were asked to enter their answer by touching the numerically larger number. No physical comparison tasks were included. Only numerical comparison tasks with a distance of 1 and 2 were asked. The numbers to be compared in the test were arranged in three different forms as congruent (5–7), neutral (5–7), and incongruent (5–7). Congruent form means that numerically large number is presented as perceptually large in compatible with its value. On the other hand, incongruent form means that numerically large number is presented as perceptually small in contrast with its value while numerically small number is presented vice versa. There were totally 36 items, 12 congruent, 12 neutral, and 12 incongruent in this test. Correct answers were equally distributed on both sides. Before starting to the task, students were instructed to touch on the numerically larger number on the screen.

Mental Number Line (MNL): Mental Number Line (MNL), was consisted of number placement tasks. A typical number line is a horizontal or vertical line with zero on the left end, and 10 (MNL1), 100 (MNL2), on the other end. Students are requested to place the numbers shown one at a time on the number line by drawing a hash mark on the number line. In experiments, number lines were placed horizontally on the screen. With this test, students would be able to touch the relative place of numbers, and move their finger on the screen to adjust the finer place. When touched, a vertical short line appeared on the horizontal number line, and moved as the students moved their fingers. Before starting to the task, students were instructed to mark the possible place of the number appeared above the number line by dragging the vertical line to the appropriate place. No timing was recorded for this test. Only the absolute values of the difference between the estimation, and numbers to be estimated were recorded in number to position tasks. There were 9 items in 0–10 (MNL1), and 24 items in 0–100 (MNL2) sub-tests (Table 1). Number of items of these tests was determined by taking into consideration of class level, and syllabus content generated based on levels (see 25 for detailed information). The sample items from each of the three sub-tests were given in Figure 1.

Procedure

Ethical Committee Report of the study was taken from the University of Ankara. The administration of the tests was realized via tablet-PC by nine research assistants who were trained in administering the sub-tests in a silent room in three separate sessions for CPT, RSPM, and TAB-DSB sub-tests. TAB-DSB (CDC, SNC, MNL1, MNL2) sub-tests were administered individually; whereas, CPT and RSPM tests were administered in a group. Three sub-tests were randomized in administration, and took about 30 minutes to complete. In these tests, the higher one score means that the more errors she/he made or the more time she/he spent on the test. Since the correlation was found to be negative between math achievement scores and these four sub-tests, this would merely imply that the higher one scored in the math achievement score, the less time she/he took to complete the test and the less errors she/he made (for details, see Section of Data Collection Tools).

RESULTS

Descriptive statistics about the participants were summarized in Table 2.

Table 2.

Means, standard deviations, and percentages regarding the participants’ demographic characteristics (N=316)

Age (year)	Gender	Grade	School District
X=7.26 (1.03)	50% (158) Male 50% (158) Female	34.5% Grade 1 31.0% Grade 2 34.5% Grade 3	26.6% Çankaya* 22.5% Mamak* 25.6% Yenimahalle* 25.3% Etimesgut*

^*School districts represent different socio-economic levels in the same metropolitan city.

In order to test whether CPT scores across first, second, and the third grades were normally distributed, Kolmogorov-Simirnov test was run and the scores were as follows at each grade respectively; D (109)=0.069, p=0.200; D (98)=0.073, p=0.200; D (109)=0.075, p=0.156). The duration for administering the CPT was 1 minute for the first grade, 2 minutes for the second, and 5 minutes for the third (20).

CDC and SNC scores were transformed into Inverse Efficiency Scores (IES), which are calculated by dividing the time spent (ms) on each item into the percentage of correct items (26, 27). For MNL (MNL1, MNL2), absolute values of the difference between the estimation and numbers to be estimated were calculated, and it is preferred to use for the following inferential analyses because previous researches showed that measures other than absolute values (like latency and MNLtotal) was not efficient to use, and have lower discriminatory power (22, 28–30). Before running the inferential analyses, normality tests were run by looking at the skewness and kurtosis scores for two separate criteria. First, data were not found to be normally distributed across low, medium, and high levels; whereas, when the upper and lower 10% distribution was considered, the data were found to be normally distributed (Table 3). The number of participants for 1 standard deviation upper and lower group was limited to 214 students because students with lower IQ (102 students) were excluded from the sample.

Table 3.

Descriptive statistics for normality test

	N	Minimum	Maximum	Mean	Std. Deviation	Skewness		Kurtosis
							Std. Error		Std. Error
SNCIES	214	32770.29	148567.35	54009.86	16768.71	2.136	0.166	6.904	0.331
CDCIES	214	36564.00	212499.00	71268.93	22928.09	2.023	0.166	8.132	0.331
MNL1	214	2.10	58.60	11.80	8.34	1.720	0.166	4.631	0.331
MNL2	214	65.40	1184.40	302.73	174.82	1.542	0.166	3.310	0.331

SNC_IES, Symbolic Number Comparison; CDC_IES, Canonic Dot Counting; MNL1, Mental Number Line 1-10; MNL2, Mental Number Line 1-100.

Log transformations were also done to ensure normal distribution for the below, above, and normal grouping; yet, normality could not be ensured. Kolmogorov Smirnov test results were significant at p=0.001 level. Therefore, based on CPT scores, participants were grouped into three based on 1 standard deviation below, above, and the middle (CPT mean values for 1 Std below group=33.46, above group=51.37, and the middle group=44.92). Then, nonparametric analysis Kruskal-Wallis test was run. Descriptive statistics about these three groups were presented in Table 4.

Table 4.

Mean rank of students below, above and normal ranged according to Z distribution (for 1 standard deviation in CPT)

Mean and Standard Deviations of Sub-Test Scores
n	CDCIES	SNCIES	MNL1	MNL2
13	136.23	101.92	110.88	124.27
156	102.59	106.21	104.06	103.19
45	116.22	113.57	118.46	117.61

SNC_IES, Symbolic Number Comparison; CDC_IES, Canonic Dot Counting; MNL1, Mental Number Line 1–10; MNL2, Mental Number Line 1–100.

In order to see whether these three groups differ in terms of their scores in these three sub-tests, one way Kruskal-Wallis analysis was run and the results indicated no significant differences for each sub-test (for CDCIES H(2)=4.673, p=0.097; for SNCIES H(2)=0.605, p=0.739; for MNL1 H(2)=1.930, p=0.38; for MNL2 H(2)=2.91, p=0.233).

Another analysis was run based on the other criterion, which was the lower 10%, and upper groups (CPT mean values for 10% below group = 31.16, and upper group = 46.18). The mean and standard deviations for this analysis are presented in Table 5.

Table 5.

The standard deviations of lower and upper groups for three sub-tests of TAB-DSB

	Group	N	Mean	Std. Deviation
CDCIES	Lower 10%	32	87540.71	26290.48
	Upper 90%	284	72276.40	23244.91
SNCIES	Lower 10%	32	66137.74	20465.68
	Upper 90%	284	54255.88	16992.82
MNL1	Lower 10%	32	15.90	9.03
	Upper 90%	284	12.34	8.74
MNL2	Lower 10%	32	473.20	254.76
	Upper 90%	284	318.18	188.23

SNC_IES, Symbolic Number Comparison; CDC_IES, Canonic Dot Counting; MNL1, Mental Number Line 1–10; MNL2, Mental Number Line 1–100.

In order to compare the TAB-DSB sub-test scores of lower 10% and upper groups which were divided based on the CPT scores, the t-test analysis was performed. The results are presented in Table 6, and indicated that there is a significant difference between the groups (tCDC (314)=3.47, p=0.001; tSNC (314)=3.67, p=0.000; tMNL1 (314)=2.17, p=0.03; tMNL2 (314)=2.17, p=0.002).

Table 6.

Comparison results of lower and upper groups for three sub-tests of TAB-DSB

Group	N	CDCIES	SNCIES	MNL1	MNL2
Lower 10%	32	t (314)=3.47	t (314)=3.67	t (314)=2.17	t (314)=3.34
Upper 90%	284	p=0.001	p=0.000	p=0.03	p=0.002

SNC_IES, Symbolic Number Comparison; CDC_IES, Canonic Dot Counting; MNL1, Mental Number Line 1–10; MNL2, Mental Number Line 1–100.

In order to see which graders differ in terms of their TAB-DSB scores, further analysis was conducted indicating that the difference is only significant at CDC, MNL1, and MNL2 scores for the groups at the first grade (significant marginally; p=0.051); yet, no difference was observed at SNC scores (p=0.123). For the other grades, no significant differences were observed.

DISCUSSION

It is important to develop and utilize objective tools to screen dyscalculia at schools, especially in Turkey, since there is no epidemiological study conducted nationwide yet. In this study, the discriminatory nature of CDC, SNC, and MNL (MNL1, MNL2) tasks on two cut-off based criteria was explored: the first grouping was based on standard deviation, and the other was based on percentages. The results indicated that CDC, SNC, and MNL (MNL1, MNL2) tasks failed to discriminate the groups when the standard deviation criteria (below/above, and within 1 standard deviation) was taken into consideration. On the other hand, these sub-tests were effective to discriminate the groups when the lower 10% and the upper groups was compared. Further analyses to explore whether grade level differences existed also showed that CDC, MNL1, and MNL2 tasks were discriminatory at the first grade (marginally significant at p=0.051), whereas no significant differences were observed for the other grades.

The results of this study suggest that the same cognitive tasks (CDC, SNC, MNL1, MNL2) yield different results when the discrimination power is sought. In other words, the same cognitive tasks could discriminate tendency to dyscalculia based on one cut-off criterion; whereas they would fail if based on another cut-off criterion. Therefore, it can be concluded that the criteria of 10% below and upper grouping based on CPT scores is more effective in discriminating dyscalculia tendency. Yet, it should also be noted that the only significant difference was for the first graders; therefore, there would be a need for further studies to develop other cognitive tasks to discriminate dyscalculia tendency for other graders.

The mental representation of numbers could be either exact or approximate. One could experience a difficulty in learning about numbers if there is disorder in any or both of these systems. The CDC task measures about the exact number system, and requires learners to subitize in a fast and correct way. This task became a discriminatory task to determine the lower 10% at the first grade.

The MNL task with two sub-tests (MNL1 and MNL2) requires learners to guess the approximate magnitude of numbers, and position of each given number on a hypothetical number line (0–10 for MNL1, and 0–100 for MNL2) appropriately. This task is also found to discriminate the lower 10% at the first grade.

The discriminatory nature of MNL task (MNL1, MNL2), on the one hand, might indicate a problem in one’s approximate number system, which posits a transformation of a given symbolic number into an analog representation of a magnitude; on the other hand, it might also indicate a problem in multitude, and attributing quantitative meaning to symbols (31). Yet, since students did not experience difficulty in SNC tasks, which queried the relationship between symbolic and quantitative magnitude, the latter argument is weakened.

The findings related to MNL tasks (MNL1, MNL2), and their discriminatory nature supports the existing research (32) in that the accuracy in approximation is correlated with math achievement; hence, it could be used to screen math learning difficulties. The finding why the task was more effectively valid for first graders could be explained by the guesses from smaller children that show logarithmic function in distribution compared to older children that show more linear function in their guesses, as suggested by Booth and Siegler (32).

The results related to the discriminatory nature of CDC and MNL tasks (MNL1, MNL2) confirm the existing hypotheses claiming that children with dyscalculic tendencies have problems with approximate and exact number systems (33, 34).

The findings of this study showed that SNC task, which measures children’s perceptual understanding of magnitude and their inferences from symbols, was not found to be discriminatory at all grades. This finding implies that children with dyscalculia did not experience any difficulty in transforming symbols into magnitudes (or inferring magnitude from symbols); therefore, we failed to confirm the access deficit hypothesis for this sample.

Symbolic and non-symbolic number comparisons yield mixing results, and are correlated with different arithmetic skills in further years. Moreover, each process has unique contribution to arithmetic learning (35). This approach confirms our findings in that different tasks for different grades would be more effective to determine dyscalculia.

Dyscalculic students’ failure in different cognitive functions at different grade levels could be explained by the fast brain development in younger ages (36). The sooner dyscalculia is determined, the better the opportunity to improve it (37). Hence, it is considered quite timely and important to revise the existing educational program, to screen learners for dyscalculic tendencies at earlier ages, and to provide them the necessary support. Finally, overall findings of the study are suggestive, but it is important to note that they should be evaluated carefully in view of the fact that the study has a limitation of large sample size differences between comparison groups like 32 students in lower 10% group vs. 284 students in upper group. Additionally, it should be noted that during the generation of TAB-DSB sub-tests, the font name Calibri was used across all tablet tasks. In order to remove potential and experimental confounding factors arising from orthographic/perceptual factors, it is suggested to use monospace fonts rather than Calibri for further studies.