Memory for curvature of objects: Haptic touch vs. vision

Abstract

The present study examined the role of vision and haptics in memory for stimulus objects that vary along the dimension of curvature. Experiment 1 measured haptic–haptic (T–T) and haptic–visual (T–V) discrimination of curvature in a short-term memory paradigm, using 30-second retention intervals containing five different interpolated tasks. Results showed poorest performance when the interpolated tasks required spatial processing or movement, thereby suggesting that haptic information about shape is encoded in a spatial-motor representation. Experiment 2 compared visual–visual (V–V) and visual–haptic (V–T) short-term memory, again using 30-second delay intervals.

The results of the ANOVA failed to show a significant effect of intervening activity. Intra-modal visual performance and cross-modal performance were similar. Comparing the four modality conditions (inter-modal V–T, T–V; intra-modal V–V, T–T, by combining the data of Experiments 1 and 2), in a global analysis, showed a reliable interaction between intervening activity and experiment (modality). Although there appears to be a general tendency for spatial and movement activities to exert the most deleterious effects overall, the patterns are not identical when the initial stimulus is encoded haptically (Experiment 1) and visually (Experiment 2).

The mechanisms underlying tactile perception of form remain relatively little understood, despite long interest in the topic. This interest extends at least several hundred years, for instance, to Molyneux's question, whether a blind person suddenly given sight would immediately recognize by vision objects previously identified by touch, and to Diderot's letter on the blind (see Morgan, 1977). Subsequently, Katz (1925/1989), Révész (1950) and Gibson (1962, 1966) invigorated the study of tactile perception by emphasizing the role of movement.

Haptic perception of form takes place largely by inspection of shapes by the palm and fingers of the hand as they move over the surface of an object. The property of shape or form refers to the spatial layout of the object, which may be specified in terms of contours. An isolated contour may itself be described with respect to its extent and orientation, and interactions among contours, or patterns, may in-turn be described in terms of component contours or as combinations of angles, straight or curved surfaces, and other distinctive features. The shapes of many objects may be adequately described in terms of angles and curvatures of various degrees and proportion. In haptic exploration of shapes, the fingers move over the various angles or bend over the curves. The main question posed in the present study is: How do haptic touch and vision compare with regard to people's ability to perceive and remember curvature in three-dimensional objects?

One way to study curvature perception in vision and touch is to compare perceptual judgments obtained within and between the two modalities. Studies have revealed differences between the two modalities that may be attributed to several factors. For example, Kappers, Koenderink, and Oudenaarden (1997) found that observers overestimated haptic curvature relative to visual curvature, concluding that mutually inconsistent representations of surface curvature in the haptic and visual modalities coexist within a single observer. But the influence of shape on curvature perception is still largely unexplained (Vogels, Kappers, & Koenderink, 1999) and the underlying mechanisms still await elucidation. The present study aims to compare curvature of three-dimensional objects within and between the modalities of vision and touch, using a dual task paradigm to test for some of the processes that may underlie the perception of curvature.

A large body of literature concerned with the haptic and visual memory systems has provided two broad accounts. The first is that object memory is multi-sensory and object representations are shared across modalities if the same object properties are encoded. The second is that objects are stored as modality-specific representations that require a recoding from one memory system to another. Such recoding is costly, in time and errors, to cross-modal performance relative to intra-modal performance.

Many recent findings support the idea that visual and haptic memory for objects is shared. For example, neuroimaging evidence suggests that the cortical area involved in visual object recognition is also involved in haptic object recognition (Grill-Spector, Kourtzi, & Kanwisher, 2001). Amedi, Malach, Hendler, Peled, and Zohary (2001) reported that activation in the lateral occipital complex (LOC) is not specific to the visual modality. The LOC is also activated during haptic object recognition, though not by auditory information about objects (Amedi, Jacobson, Hendler, Malach, & Zohary, 2002). These studies suggest that the LOC is involved in recovering the geometrical shapes of objects (James et al., 2002) and behavioural evidence suggests that object representations are shared across modalities (Easton, Greene, & Srinivas, 1997; Newell, 2004).

Other studies indicate, however, that tactile memory is not equivalent to visual memory and that information is modality specific. For example, in tests of tactile memory, when a delay intervenes between the presentation of the test stimulus and the test of memory, tactile memory is adversely affected by verbal tasks (counting backwards) interpolated during a 15-second delay (Gilson & Baddeley, 1969) and by arithmetic tasks after a 5-second delay (Sullivan & Turvey, 1972). In a study on three-dimensional object recognition in children, Millar (1974) found that haptic matching performance for nonsense objects was better with an inter-stimulus interval (ISI) of 1 second than with intervals of 5 and 30 seconds, suggesting that haptic memory starts to decay immediately after the exploration of an object.

Other studies report that tactile memory can be sustained for 15 seconds (Kiphart, Hughes, Simmons, & Cross, 1992) and is vulnerable to articulatory suppression (Mahrer & Miles, 2002). However, performance can also be affected by the task demands or stimulus complexity and the degree of familiarity with the objects being explored (Millar, 1981). For example, Norman, Norman, Clayton, Lianekhammy, and Zielke (2004) observed that accuracy of tactile–visual matching differed across different stimuli, though tactile performance improved with exploration time. Norman et al. concluded that observers can match objects known only through touch with other objects known only through vision. Recently, Woods, O'Modhrain, and Newell (2004) suggested that tactile–visual object recognition may rely on modality-specific representations. Although the effect of delay between cross-modal presentations was the same whether the initial object was coded visually or haptically, recall was better after a 0-second than a 30-second delay, indicating delay-induced decay in memory.

In view of the different interpretations, the present study was designed to test for inter- and intra-modal processes involved in memory for concave curved surfaces that are first presented (perceived) haptically or visually and then later compared with surfaces presented either haptically/visually (unimodal comparison) or visually/haptically (cross-modal comparison). Furthermore, by varying the characteristics of the Participants' activity during a 30-second retention interval (dual task design), we sought to illuminate the underlying mnemonic representation mediating the discrimination.

Dual task paradigms have been used to test for the demands of attention on the primary task (Brown, 1958; Peterson & Peterson, 1959). The paradigm tests the nature of coding in short-term memory by interpolating a secondary task during the delay interval between presentation and recall. Filling the delay intervals with stimuli in different modalities produces modality-specific interference. For example Logie (1986) found that monitoring random visual matrix patterns interferes with a visual imagery mnemonic task even after the main and the secondary tasks are equated for difficulty. Dual task paradigms that test visuospatial coding often require movement outputs. For example, tracing, pointing or other gestures are used as secondary tasks. Imagining the task during a delay interval instead of actually performing the task produces similar effects. Millar and Ittyerah (1991) for example, showed modality-specific motor memory in blind conditions not only in actual performance but also in imagined conditions, which excluded any influence of visual knowledge. Thus, it is possible to use movement imagery in the recall of guided movements, as imagined movements biased recall as much as actual movements did. Further, articulatory suppression had no effect on performance, implying little or no role for translation of extent of movement into a verbal format. However, the nature of this modality-specific motor memory is still not well understood. The question arises, for example, whether motor memory is represented spatially in terms of spatial extent or extent of movement. Studies of movement have shown that extent of movement is encoded differently from spatial location in short-term memory (Laabs, 1973; Laabs & Simmons, 1981). Short-term memory could rely, therefore, on kinaesthesis even when inputs are not coded spatially, although memory for extent of movement is not very accurate.

Different mechanisms might encode different features of objects perceived haptically. Katz (1925) suggested that relatively coarse surface structures are processed spatially, whereas very fine textures are processed temporally. Furthermore, some evidence (Lamotte & Srinivasan, 1993; Srinivasan & Lamotte, 1987) suggests that the geometrical features of an object's shape are represented in the spatial–temporal responses of slowly adapting and rapidly adapting cutaneous mechanoreceptors. If memory and comparison relate directly to the outputs of these receptors, then interpolating between the first and second presentations a task requiring processing of spatial information should interfere markedly with performance. Further, interpolating a task requiring participants to move their hands should show whether haptic curvature is represented in terms of the extent to which the hand or arm moved over the stimuli. Findings of Millar (1991), for example, imply that movements of the arm or hand may facilitate haptic representations. Millar's findings indicate that blind children can represent shapes of body parts in the form of two-dimensional or stick figures even though the children are unable to recognize the drawings. Sighted children, on the other hand, may recognize objects or drawings but have difficulty producing them (Maccoby & Bee, 1965). Blind children need to produce movements even if they cannot recognize them, because movements facilitate their haptic representations.

If haptic representations are stored in a spatial format, it is nevertheless possible that the final stored representation is verbal. To test this possibility, one of the intervening tasks required the participants to vocalize. Vocalization is known to suppress articulation by preventing verbal rehearsal, consistent with the view that verbal rehearsal is articulatory. Consequently, requiring participants to utter a sequence of irrelevant sounds should interfere with rehearsal by occupying the articulatory system while otherwise imposing relatively little ‘load’ on the system (Baddeley, 1986). Counting has been used successfully to interfere with rehearsal in children of 7 and 11 years (Hagan & Kail, 1973) and it has been argued that counting backwards prevents any form of rehearsal in adults (Peterson & Peterson, 1959).

Are differences between haptic and visual perception of curvature evident in memory? We expect that haptic, visual and cross-modal memory will differ in their sensitivity to the activities in which participants engage during the interference period. During a delay period of 30 seconds between the presentation of a test stimulus and its recall, we introduce any one of the four following activities, such as counting aloud, rehearsing the test stimulus visually or haptically, spacing paper clips in equal distances or moving books from one hand to the other, as well as an unfilled delay that served as a control. We expect that tasks requiring spatial processing may have deleterious effects on performance in both modalities, whereas tasks requiring movement should exert greater effects on haptic memory.

Experiment 1

Haptic and haptic–visual memory for curvature

Experiment 1 used a dual-task paradigm to shed light on the nature of the mnemonic representations of curvature assessed through touch. Although the nature of this memory is yet not clear, it seems likely that, in the absence of vision, haptically presented information about shape is encoded and represented in a spatial (Nefs, Kappers, & Koenderink, 2001) and/or kinesthetic format. Consequently, in Experiment 1, during the 30-second retention interval between the presentation of the first and second stimuli, we had participants engage in different activities expected to interfere with haptic processing.

Method

Participants

Thirty-two participants, 16 men and 16 women from Yale University, between the ages of 18 and 30 years (mean age 24 years), participated, each serving in one of the two experimental conditions. Participants had no tactile or visual defects and had normal or corrected-to-normal visual acuity. Equal numbers of men and women served in each condition.

Stimulus materials and experimental set-up

Two sets of four concave stimuli were constructed specifically for Experiments 1 and 2. These stimuli were constructed from blocks of maple wood measuring 5.08 × 10.16 × 3.81 cm. These dimensions were identical for all stimuli. Only the depth of concavity, and thus the radius of curvature, for each stimulus varied. Curvature is inversely related to the radius of curvature: the greater the radius of curvature, the less curved (flatter) the stimulus. The radii of curvature of the stimuli from the flattest to the most curved were 34.1, 25.7, 20.6 and 17.3 cm, corresponding to maximal depths of indentation of 3.81, 5.08, 6.35 and 7.62 mm, respectively (equal differences in indentation and near-equal differences in radii of curvature). For convenience of exposition, we label the successively increasing curvatures of these four stimuli by the numerals 1–4.

Figure 1 shows each of the four stimuli which differ from one another only in their heights from the mid-point of each stimulus to its end. For example, stimulus 1 has a difference of 3.81 mm between its mid-point and its height at the ends. Stimulus 2 has a difference of 5.08 mm between its mid-point and the height at its ends and, therefore, greater curvature than stimulus 1. The remaining stimuli vary from each other in a similar manner, such that stimulus 4 has the greatest curvature and stimulus 1 the least curvature. All stimuli were smooth to touch and did not differ in texture. These stimuli were created in order to test for discrimination of minimal differences, both within and between the modalities of touch and vision. Experiment 1 used stimuli 2–4, selected to ensure that the adjacent pairs would be neither too easy nor too difficult to discriminate (see Pont, Kappers, & Koenderink, 1999). In the analyses reported below, the stimulus pairs were defined in terms of step sizes, as follows: step size 0 = same pairs, 2&2, 3&3 and 4&4; step size 1 = different pairs 2&3 and 3&4; and step size 2 = different pair 2&4.

Figure 1.

Stimuli.

Each participant sat before a table that supported a box measuring 46 × 53 × 30.5 cm. Between the top surface of the table and the bottom of the box was a gap, located approximately waist-high, of about 12 cm into which the participant could reach (but could not see) in order to examine stimuli haptically. For visual inspection, stimuli could also be placed on top of the box, at eye level, about 30 cm from the participant, for viewing without constraint. The stimuli were presented in a fronto-parallel plane with the long side of the stimulus facing the participant, so that the participant could inspect the curved plane.

Procedure

On each trial, the participant was presented with two stimuli. The first was presented haptically (T) and the second, 30 seconds later, was presented for recognition either haptically (T) or visually (V), using a same–different procedure. Thus, there were two experimental conditions: a purely haptic condition (T–T) and a cross-modal condition (T–V).

To the first (haptic) stimulus, the participant was required to place the top portion of her/his preferred forefinger at one end of the curved surface and move it in a single continuous motion to the other end. A single continuous movement of the finger was considered to specifically focus on the curvature being felt, whereas free exploration of the stimulus with the entire hand would have distracted the participant with the irrelevant dimensions of the stimulus. There was no definite time limit for a single continuous movement of the finger on the curvature, although this was considered equivalent to a 1-second visual exposure of the same stimulus. The fingertip was positioned at the centre of the surface of the curvature and not at the edges, so that the finger could easily move from one end of the curved surface to the other end. There were no restrictions in the movement of the elbow as the finger felt each surface. This was followed, after a 30-second delay, by a haptic (T–T) or visual (T–V) comparison. In the T–T condition, the participant again moved the forefinger across the curved surface, whereas in the T–V condition, the visual stimulus was placed on the box at eye level for a 1-second exposure. Within each of these two conditions, five different activities could intervene during the delay interval:

Unfilled delay

The participant was instructed to sit back for 30 seconds and think of nothing. This served as a baseline for the other conditions since the participant was not instructed to do anything specific.

Rehearsal

During the 30-second interval, the participant repeatedly examined the stimulus by moving her or his forefinger over it as in the initial presentation, while looking at a picturesque glass placed on the box at eye level so that the participant did not see the test stimulus. It is known that practice whether actual or imagined improves performance (see introduction). Therefore, this task is expected to improve performance.

Articulatory suppression

Immediately after raising her or his finger from the stimulus, the participant began counting until asked to stop after 30 seconds. This task was introduced to test for the effects of verbal mediation or subvocal rehearsal of the felt curvature. If subvocal rehearsal is used, it is expected to disrupt recall.

Spatial

During the interval, the participant placed a set of paper clips on the table top before her/him at uniform distances, as instructed, such as 2 cm apart, 4, 1, 3 cm etc. To prevent the participant from looking at the paper clips during the intervening task, the participant was required to look at a picturesque glass located directly ahead at eye level. This is a spatial task, because there are distinct reference points between a pair of clips. Remembering reference points, such as ‘one inch apart’, is considered to disrupt the spatial processing of the curvature to be remembered (based on global configurations).

Movement

During the interval, the participant used both hands to move books from one pile and stack them in another, while viewing the picturesque glass. In this task as well, the participant was not allowed to look at her/his hands or the books while performing the task. This task differs from the spatial task because the participant is required to repeatedly move both hands to place a book in the same location without having to vary distances as is common in spatial tasks.

The orders of the intervening activities were randomized both within and between participants. At the end of each delay interval, the second stimulus was presented and the participant indicated whether it was the same as or different from the first. Each of the three possible target stimuli (2, 3 or 4) was paired in each experimental condition with itself and with each of the other two stimuli, making six different stimulus combinations in all, three possible same pairs, < 2,2 >, < 3,3 > and < 4,4 >, and three possible different pairs, < 2,3 >, < 2,4 > and < 3,4 >. Each stimulus pair was presented thrice at random, making 18 trials for an intervening activity. Each of these 18 stimulus combinations was presented with each of the five possible intervening activities, making 90 trials in all per condition for a single participant. Each participant performed the experiment for over an hour. Rest pauses were allowed after a session with one intervening activity.

Statistical analysis

When response proportions, p, result from a binomial process, the variance of p is not independent of the mean, but varies in proportion to p (1 – p); moreover, distributions of binomial proportions, especially at extreme values, are commonly highly skewed. Consequently, it is appropriate to transform the proportions so as to minimize the statistical consequences of heteroskedasticity and skewness. Statistical analyses (analysis of variance) were performed on response proportions after the values were transformed by the modified arcsin equation of Kendall and Stuart (see Pollard, 1977, for a full account of the rationale). The transformation takes the form

$${\text{W}}_{\text{i},\text{j}}=\text{arcsin}(\text{sqrt}[({\text{n}}_{\text{i},\text{j}}+1/6)/(\text{N}+1/3)])$$

where W_i,j is the transformed average response proportion made by participant j to stimulus i and n_i,j is the number of correct responses on the N trials. After calculating for each stimulus, the mean of these transformed responses, W_i,j/N, the resulting arcsin values could then be transformed to mean response proportions, M_i, by applying the inverse of the transformation W. That is, after transforming each participant's score to a value of W, the mean values of W could then be inversely transformed back into the corresponding and more familiar units of response proportion. The mean proportions M were also used to generate measures of d′, which serve as a convenient summary, combining performance on both same and different trials. To accomplish this, the values of d′ were computed using a differencing model of same–different judgment (see Macmillan & Creelman, 1991).

Results

The mean proportions of correct responses (arcsin transformed, then converted back to percentages) given to same and different pairs under the various conditions appear in Tables 1 and 2. In general, there is a strong tendency, most noticeable in the judgments of same pairs, for performance to be best with the unfilled delay, rehearsal, and articulatory conditions and poorer with the spatial and movement conditions. The judgments were examined in detail by subjecting them to two ANOVAs with repeated measures. The first ANOVA was performed on the responses given both to the same stimulus pairs (responses to step size 0) and to all of the different stimulus pairs (pooling the responses across step sizes 1 and 2). The second ANOVA omitted the responses to the same stimulus pairs (step size 0) and analysed responses to different stimulus pairs only (step size 1 and 2), treating step sizes 1 and 2 separately.

Table 1.

Experiment 1. Percentage correct for comparisons of same stimuli for the five delay conditions

Delay tasks	Tactual–Tactual	Tactual–Visual
Rehearsal
Mean	100	100
SD	0	0
Articulatory suppression
Mean	100	100
SD	0	0
Unfilled delay
Mean	100	100
SD	0	0
Spatial
Mean	76.5	88.3
SD	2.6	4.5
Movement
Mean	75.6	99.9
SD	1.6	3.3

Table 2.

Experiment 1. Percentage correct for comparisons of different stimuli for the five delay conditions at step sizes 1 and 2

	Tactual–Tactual		Tactual–Visual
Delay tasks	Mean	SD	Mean	SD
Step 1
Rehearsal	100	0	65.9	6.2
Articulatory suppression	100	0	76.0	2.8
Unfilled delay	100	0	73.2	5.6
Spatial	77.2	2.6	46.6	5.1
Movement	78.5	2.5	59	1.6
Step 2
Rehearsal	100	0	100	0
Articulatory suppression	92.5	3.2	100	0
Unfilled delay	100	0	100	0
Spatial	73.5	1.6	72.0	7.3
Movement	72.5	5.2	85.0	7.2

Analysis of same pairs (step size 0) and different pairs (step sizes 1 and 2)

The first ANOVA (arcsin transformed scores) used four factors: (1) modality of second stimulus (T–T, T–V), (2) gender (F, M), (3) kind of stimulus pair (same, different) and (4) intervening activity (unfilled delay, rehearsal, articulatory suppression, spatial, movement), with repeated measures on the last two factors. The results showed two significant main effects: kind of stimulus (F(1, 28) = 18.02, p < .001), reflecting greater accuracy with same pairs than with different pairs (which could reflect the participant's criterion for responding ‘same’ and ‘different’) and intervening activity (F(4, 112) = 19.3, p < .001), reflecting greater accuracy with unfilled, rehearsal and articulatory tasks, and poorer accuracy with spatial and movement tasks. There was no effect of modality (F < 1), implying that it did not matter whether the second stimulus was presented visually or haptically or of gender (F < 1), implying that performance of women and men did not differ. Further, the ANOVA revealed only one reliable interaction, between modality and stimulus (F(1, 28) = 4.58, p < .05); Newman–Keuls post hoc tests showed that the same judgments of T–V are more accurate than that of T–T (p < .05). Neither the interaction of modality × intervening activity (F(4, 112) = 0.415, p > .79) nor the interaction of stimuli × interfering activity (F(4, 112) = 0.965, p > .4) is significant.

Step sizes 1 and 2

The second ANOVA (arcsin transformed scores) also used four factors: modality (T–T, T–V), gender (M, F), step size (1, 2) and activity (unfilled, rehearsal, articulation, spatial, movement), with repeated measures on the last two factors. The results again showed the main effects of modality and gender to be insignificant (F < 1 for both). The effect of step size was significant (F(1, 28) = 41.66, p < .001), showing that accuracy was greater with step size 2 vs. 1, as was, again, the effect of intervening activity (F(4, 112) = 8.97, p < .001), indicating that the spatial and movement tasks interfered most with performance. The only significant interaction was that between gender and activity (F(4, 112) = 2.75, p < .05); Newman–Keuls post hoc tests showed that, relative to the other conditions, both men and women performed poorly with the spatial activity although the men were better than the women with the spatial activity and the women were better than the men for the movement activity (p < .05). The modality × step size interaction (F(1, 28) = 1.0096, p > .32), the modality × intervening activity interaction (F(4, 112) = .554, p > .7) and the step size × intervening activity interactions (F(4, 112) = 0.438, p > .78) are not significant.

The overall picture of performance in the memory task is the same when we combine proportions on corresponding same and different trials into values of d′ for each modality, stimulus step size and intervening activity. As Figure 2 shows, (1) d′ increases by about one as step size increases for the present curvature stimuli; (2) in each condition, d′ is roughly the same with haptic and visual presentations of the second stimulus and (3) in each condition, d′ is greatest with the unfilled, rehearsal, and articulation activities during the retention interval, poorer with the spatial and movement activities.

Figure 2.

Discriminability (d′) as a function of the activity intervening between the first stimulus (always tactile) and the second stimulus (visual or tactile) for small differences (1 step) and larger differences (2 steps) in curvature (Experiment 1).

Experiment 2

Visual and visual–haptic memory for curvature

The results of Experiment 1 showed performance in haptic–haptic and haptic–visual memory tasks was poorest when the retention interval between the first and second stimuli required spatial processing and movement, suggesting that information about the curvature of the stimuli may be processed spatially, or by the extent to which the finger moved over the stimuli. By implication, haptic memory may primarily involve spatial processes (Katz, 1925). The question of interest here is whether such spatial processing is modality specific. To help answer this question, Experiment 2 was designed to test memory for curvature using visual (V–V) and cross-modal (V–T) stimulus presentations.

Method

Participants

Two groups of 12 participants, 6 women and 6 men between the ages of 18 and 30 years (mean age 25 years) participated separately in each of the two experimental conditions: V–V and V–T. Thus, there were 24 participants in all. All of the participants were university students from Delhi with no known sensory defects.

Stimuli

The stimuli were numbers 1, 2, 3 and 4 with radii of curvature of 34.1, 25.7, 20.6 and 17.3 cm, respectively (see Figure 1). Three stimuli were the same as those in Experiment 1, to which the additional stimulus 1 was added to increase the range of stimuli to be discriminated.

Extending the definitions of step size used in Experiment 1, step size 0 = the same stimulus pairs 1&1, 2&2, 3&3 and 4&4; step size 1 = the stimulus pairs 1&2, 2&3 and 3&4; step size 2 = the stimulus pairs 1&3 and 2&4; and step size 3 = the stimulus pair 1&4.

Procedure

The experimental set up was the same as that in the previous experiment. The participants were presented two stimuli on each trial, the first visually (V) and the second for recognition either visually (V) or haptically (T), using the previously described same–different procedure. After looking at the first (visual) stimulus for a period of 1 second, a delay interval of 30 seconds followed, after which a test stimulus was presented for visual or haptic recognition. In the V–T condition, the participant moved her/his forefinger across the curved surface, whereas in the V–V condition, the visual stimulus was placed on the box at eye level. Separate groups of 12 participants served in each condition, which combined modality (V–V or V–T) with a retention interval of 30 seconds. In each of the two conditions, each of the previously described five activities could intervene between the first and second stimuli presented on a given trial. In the V–V and V–T conditions when the rehearsal was visual, the participant looked at the visual stimulus placed on the box at eye level during the 30-second delay.

At the end of each retention interval, the second stimulus was presented and the participant indicated whether it was the same as or different from the first stimulus. Each of the four possible target stimuli (1, 2, 3 or 4) was paired in each experimental condition with itself and with each of the other three stimuli, making ten different stimulus combinations in all, four possible same pairs, < 1,1 >, < 2,2 >, < 3,3 > and < 4,4 >, and six possible different pairs, < 1,2 >, < 1,3 >, < 1,4 >, < 2,3 >, < 2,4 > and < 3,4 >. Each of these 10 stimulus combinations was presented thrice at random within each of the five possible intervening delay activities, making 150 trials per condition for a single participant. The experiment lasted for about an hour and a half for each participant. Rest pauses were allowed after a session with one intervening activity.

Results

The judgments in all conditions were converted to proportions and arcsine transformed. Since the results of Experiment 1 indicated differences in the processing of same and different pairs of stimuli, we subjected same and different pairs to separate analyses of variance, using Statistica, 1997, version 5.5. This approach also has the virtue of reducing the number of interaction terms, many of which would not readily be interpretable. The mean percentages for all pairs of stimuli in both modality conditions are presented in Tables 3 and 4. In the results to follow, the mean values are presented as percentages.

Table 3.

Experiment 2. Percentage correct for comparisons of same stimuli for the five delay conditions

	Visual–Visual		Visual–Tactual
Delay tasks	Mean	SD	Mean	SD
Rehearsal	67	21	51	23
Articulatory suppression	59	27	63	20
Unfilled delay	70	22	63	22
Spatial	57	25	58	21
Movement	61	22	53	25

Table 4.

Experiment 2. Percentage correct for comparisons of different stimuli for the five delay conditions at step sizes 1, 2 and 3

	Visual–Visual		Visual–Tactual
Delay tasks	Mean	SD	Mean	SD
Step 1
Rehearsal	23	21	20	24
Articulatory suppression	22	21	26	26
Unfilled delay	25	23	18	20
Spatial	19	23	19	23
Movement	29	26	20	23
Step 2
Rehearsal	49	27	46	20
Articulatory suppression	49	26	53	19
Unfilled delay	65	23	33	24
Spatial	50	29	34	18
Movement	47	30	38	18
Step 3
Rehearsal	74	20	77	20
Articulatory suppression	83	18	81	15
Unfilled delay	77	20	77	20
Spatial	65	19	78	17
Movement	77	27	75	22

Same pairs

An analysis of variance (arcsin transformed scores) used variables of gender (2), modality (2), interfering activity (5) and stimuli pair (4), with repeated measures on the last two factors. The main effect of gender was not significant (F(1, 20) = 1.7, p > .21; Mean for women and men=62.33 and 57.95%, respectively). The main effect of modality was not significant (F(1, 40) = 2.45, p > .13; M for V–V and V–T = 62.77 and 57.51%, respectively) nor was the main effect of interfering activity (F(4, 80) = 2.08, p > .09; M reh = 58.86, M mov = 56.91, M spa = 57.31, M ufd = 66.56, M as = 61.06%, respectively). Note in particular that the means with the movement and spatial activities (as well as with rehearsal) are numerically smaller than the means after unfilled and articulatory activities, a pattern at least roughly similar to that observed in Experiment 1.

Performance differed significantly among the four same pairs (F(3, 60) = 4.21, p < .0001), being best with stimulus 4 followed by stimulus 1 (M 1&1 = 63.33, 2&2 = 47.77, 3&3 = 54.91, 4&4 = 74.54%, respectively). Further, the interaction of gender and stimuli was significant (F(3, 60) = 4.21, p < .009); Newman–Keuls post hoc tests showed that the women performed better than the men with stimuli 1&1 and 4&4 (p < .05).

Different pairs

Women and men did not differ significantly (F(1, 20) = 0.005, p > .94; M for women=39.89, men=39.68%, respectively). Performance did not differ between the V–V and V–T conditions (F(1, 20) = 1.66, p > .21; M for V–V = 41.66, V–T = 37.9%) nor did performance across the five intervening activities (F(4, 80) = 2.16, p > .08; M reh = 39.11, M mov = 39.08, M spa = 35.49, M ufd = 42.57, M as = 42.67%, respectively). Note again, the means with the movement and spatial activities (as well as with rehearsal) are numerically smaller than the means after unfilled and articulatory activities. That the same pattern appears in judgments of both the same and different pairs, suggests that, considered overall, the five intervening activities may exert a similar effect on performance.

Only the main effect of stimuli was significant (F(5, 100) = 103.44, p < .0001; M for 1&2 = 29.89, 1&3 = 66.43, 1&4 = 76.04, 2&3 = 14.68, 2&4 = 29.78, 3&4 = 21.89%, respectively), with performance best when the stimulus steps were large (1 and 3, and 1 and 4 more accurate than the others).

Measures of d′

As in Experiment 1, data obtained with same and different pairs in all of the conditions were combined into unitary measures of overall performance, d′, using a differencing model. Figure 3 shows the results, plotting d′ against the stimulus step size for each of the two modality conditions (V–V, V–T) for 30 seconds retention. Performance at step size 1 in both modality conditions is near chance level, although performance is better in the visual (V–V) condition than the cross-modal (V–T) condition at step size 2. At step size 3 when the difference between pairs of stimuli is relatively larger, performance in the cross-modal (V–T) condition is comparable to that in the unimodal (V–V) condition. Indeed, with the least discriminable pairs of stimuli (step size=1), d′ is actually negative in all V–T conditions. In the V–V conditions, d′ is negative at 30 seconds for all activities save doing nothing (unfilled). The negative values of d′ reflect lower hit rates (calling different pairs ‘different’) compared with false positive rates (calling same pairs ‘different’).

Figure 3.

Discriminability (d′) as a function of the activity intervening between the first stimulus (always visual) and the second stimulus (visual or tactile) for small differences (1 step) and larger differences (2 and 3 steps) in curvature (Experiment 2).

Values of d′ indicate that visual discrimination (V–V) at step size 1 (Experiment 2) was poorer than was tactual discrimination of step size 1 (Experiment 1). Note that the present experiment included an additional stimulus (1, see Figure 1) with the least curvature, which may have engendered the differences between discrimination in the two experiments. More pertinent to present concerns, however, the results of Experiment 2 in general imply that spatial processing and movement do not have an especially deleterious effect on visual memory for curvature.

To compare results of Experiments 1 and 2 with respect to the effects of modality and step size, a third analysis was performed, using the data of stimuli 2, 3 and 4 (which were common to the two experiments), with unequal numbers of participants (32 in Experiment 1 and 24 in Experiment 2). This analysis makes it possible to deploy the data of Experiments 1 and 2 in order to compare effects of the various interpolated tasks on performance in the visual and haptic modalities with a common set of stimuli. The data were analysed separately for the same and different responses.

Note that in this analysis there are only 45 trials of same responses and 45 trials of different responses for each participant (90 in all, as in Experiment 1) unlike that of Experiment 2, which had 150 trials per participant since it included the judgments of stimuli pairs with stimulus 1.

Same pairs

The arcsin transformed data of Experiments 1 and 2 were subjected to a three factors, unequal numbers analysis of variance of gender (2) ( × modality (4: V–V, V–T, T–T, T–V) × interference (5), with repeated measures on the last factor, using Statistica 1997, version 5.5. The results indicate that the main effect of gender is not significant (F(1, 48) = 0.57, ns.). Performance of women (M = 69%) and men (M = 65.8%) did not differ across the four modality conditions. There are differences among the modalities (F(3, 48) = 73.21, p < .0001), with V–V (M = 62%) and V–T (M = 56%) differing from T–T (M = 90.4%) and T–V (M = 97.6%). Furthermore, post hoc Duncan's multiple comparison of means indicate that although performance in V–V and V–T do not differ, performance in V–V and V–T do differ from performance in T–T and T–V, and performance in T–T differs from performance in T–V (p < .05). The effects of interference activities during the delay period are also significant (F(4, 192) = 11.31, p < .0001). Post hoc Duncan's test showed that overall performance after rehearsal (M = 74.1), articulatory suppression (M = 74.1) and unfilled delay (M = 80.1) activities all exceed overall performance after spatial (M = 53.3) and movement (M = 55.5) activities (p < .05). The modality × interference interaction is significant (F(12, 192) = 1.99, p < .03), indicating that performance following rehearsal, articulatory suppression and unfilled delay activities is better in the T–T and T–V conditions than in the V–V and V–T conditions. Post hoc Newman–Keuls tests show that performance is better for T–T and T–V than for V–V and V–T with all the five intervening activities (p < .05). Therefore, the intervening activity is more important during tactile than visual processing.

Different pairs

The responses to the different pairs of stimuli were subjected to a 2 (gender) × 4 (modality) × 5 (interference) × 2 (step sizes) unequal numbers analysis of variance, with repeated measures on the last two factors. The results indicate that the main effect of gender is not significant (F(1, 47) = .34, ns; M for women and men=34.86 and 32.86%, respectively). The modalities differ (F(3, 47) = 200.26, p < .0001). Post hoc Duncan's test indicates that V–V (M = 27%) and V–T (M = 20%) differ from T–T (M = 89.4%) and T–V (M = 77.8%, p < .05). Performance also differs across the interference activities (F(4, 188) = 8.67, p < .0002). Post hoc Duncan's test indicates that rehearsal (M = 38.37%), articulatory suppression (M = 38.12%) and unfilled delay (M = 39.63%) differ from spatial (M = 24.85%) and movement (M = 29.78%, p < .05). The step sizes differ from each other (F(1, 47) = 18.04, p < .0001), with better performance at step size 2 (M = 39.12%) than step size 1 (M = 29.14%, p < .05).

The modality × interference interaction is significant (F(12, 188) = 2.4, p < .007). Post hoc Newman–Keuls test shows that V–V and V–T differ from T–T and T–V with all five delay activities (p < .05). This indicates that the intervening activity is more effective in the tactile than the visual modality for the different judgments as well. The modality × step size interaction is significant (F(3, 47) = 10.16, p < .003). Post hoc Newman–Keuls tests show that performance is better at step size 2 than step size 1 (p < .05) for the V–V, T–T and T–V conditions. There are no differences between the step sizes for the V–T judgments. The gender × modality × delay interaction is significant (F(12, 188) = 2.16), p < .01). Post hoc tests show that for both women and men, performance is better in the T–T and T–V conditions than the V–V and V–T conditions over all the intervening activities (p < .05).

General Discussion

Several investigations in the past have shown that when three-dimensional objects are presented for comparison, discrimination is poorest with purely tactile presentation, distinctly better with visual–tactile (cross-modal) presentation and best with purely visual presentation (e.g. Abravanel, 1971; Easton et al., 1997; Lobb, 1965; Millar, 1981; Rudel & Teuber, 1964). Norman et al. (2004) found that unimodal haptic comparisons of natural three-dimensional shapes were as good as visual–haptic and haptic–visual comparisons. Norman et al., therefore, concluded that vision and touch have functionally overlapping, though not equivalent, representations of three-dimensional space.

It is interesting to note: Experiment 1 found that when three-dimensional objects were presented sequentially, with a 30-second delay between the first (touch) and the second (touch or vision) stimuli, tactile performance and cross-modal performance were fairly similar, indicating that unimodal judgments may not always be better than cross-modal judgments (see Figure 2). In the delayed discrimination of curvature, a primary variable governing performance is the nature of the activity in which the participant engages during the interval between the first and second stimuli. Engaging in activities that require spatial processing and/or movement impair performance substantially, relative to doing nothing, rehearsing or counting, all of which produce comparable levels of performance.

Although the patterns of performance evident in the results of Experiments 1 and 2 are similar, they are not identical: first, the effect of intervening activity statistically reliable in Experiment 1 but not in Experiment 2; second, a global analysis, performed on results of both experiments together, showed a reliable interaction between intervening activity and experiment (modality). Although there appears to be a general tendency for spatial and movement activities to exert the most deleterious effects overall, the patterns are not identical when the initial stimulus is encoded haptically (Experiment 1) and visually (Experiment 2).

The present findings indicate that performance was not only better in haptic than visual conditions but also that the intervening activities exerted greater effects with haptic than visual presentations. Prior evidence has shown that vision improves shape matching and seems to dominate over touch and haptic inputs that involve touch and movement (Held, 1963, 1965; Rock and Victor, 1964). It is also generally agreed that vision is most important in spatial tasks (Senden, 1932/1960; Attneave & Benson, 1969). Furthermore, two recent studies suggested that non-informative vision can improve perception by touch (Kennett, Taylor-Clarke and Haggard, 2001; Newport, Rabb, and Jackson, 2002). These findings are not consistent, however, with the evidence that people who are totally blind from birth can be equally or more proficient than the sighted on spatial tasks (Hollins, 1986; Millar, 1994). This evidence implies that proficiency varies with the spatial information that is available from other sources. For example, in a spatial location task of six landmarks, Millar and Al-Attar (2005) found that vision affects haptic processing only if vision adds task relevant information. Touch with diffuse light perception that excluded spatial cues and touch without vision did not differ in accuracy of performance. Millar and Al-Attar conclude that the differences between performance with spatially relevant and spatially irrelevant visual information provide new evidence against the hypothesis that vision affects haptic processing even if it does not add task-relevant information. Therefore, the relatively better performance of tactile compared with visual judgments in the present study may reflect differences in the relevance of the delay tasks used here to probe the tactile and visual systems.

Not surprisingly, performance in both experiments was consistently better with larger step sizes between the stimuli. Norman et al. (2004), for example, found that same/different judgments of their naturally shaped stimuli differed across modalities, with lower accuracy for greater similarities in object shape. Their findings agree with those of the present study. Larger differences between the curvatures of a pair of stimuli are more discriminable than smaller differences.

It is pertinent to consider how the effects of the different activities performed during the delay interval relate to other findings in the literature. The clip spacing task is clearly spatial, as it required participants to estimate equal space between pairs of clips. Presumably, then, this task invokes processes of imagery and therefore of spatial representations in imagery. Studies of visual imagery have reported memory for pictures and non-verbal materials in both adults (Paivio, 1971) and children (Reese, 1970; Rowher, 1970), and Kosslyn (1980) reviews considerable evidence that visual images operate in a spatial medium. Therefore, memory, like imagery, cannot be described solely in terms of inner speech or verbal content.

The book moving task, involving movement per se, likely disrupts the kinesthetic inputs obtained from moving the finger over the curvature stimulus. Movement coding has been demonstrated in short-term memory for non-verbal material. The tasks are spatial in character in that they demand memory for distances or locations, though the inputs and tests are hand and arm movements. Memory for linear movements has been disrupted with secondary tasks and Posner (1967) has argued that visual and movement (kinesthetic) codes differ. Laabs (1973) experimentally distinguished between kinesthetic (felt movement) and spatial coding in blind conditions. The main interpretation is that recall of movement distance is variable and less accurate, because it depends mainly on memory of the kinesthetic (movement) input and that recall of end locations is accurate because end locations can be coded according to body centred frames (Marteniuk, 1978; Russell, 1976). Movement coding can also be elicited in the absence of vision when information about reference anchors is less reliable (Laabs, 1973; Laabs & Simmons, 1981; Marteniuk, 1978; Millar, 1981, 1994).

Participants could retain information about curvature over intervals of 30 seconds between the first and second stimulus in each trial, even when required to count aloud during that period – an activity designed to suppress any articulations that might be associated with verbal labelling of the stimuli. Although there are suggestions that counting may not be as effective as repeating a single syllable (Baddeley, 1986), counting has been found to interfere with recall of verbal lists (Hagan & Kail, 1973) and memory for locations (Millar, 1994) that could be coded reliably by reference to body-centred frames. Instructing participants to count aloud was not substantially more or less disruptive than asking them to do nothing during the interval or allowing them to sample the initial stimulus repeatedly. Performance deteriorated only when the intervening activity involved irrelevant spatial and movement actions. Therefore, it is unlikely that the mediation was verbal because articulatory suppression had no effect on any group for any of the conditions, suggesting that participants did not translate the visual or tactile effects of the curvature into some verbal description that they repeated subvocally.

The value of d′ following articulatory suppression was higher (Experiment 1) than the value of d′ following the irrelevant spatial and movement tasks. Rehearsing the shape of the curvature during the delay period showed effects similar to those of unfilled delay or articulatory suppression, suggesting that memory for the felt stimulus did not deteriorate differentially from any of these activities. Therefore, tactile memory for curvature can be sustained by mental practice.

The participants in the present study were not blindfolded, although their hand and the stimulus were shielded from view during haptic performance. Therefore, they could have benefited from spatial information in their external surround, unlike blindfolded participants. Though both external and self referent cues were available to all participants, the overall performance indicates that the intervening activities involving movement and spatial processing interfered with haptic recognition of curvature, and did so both when the second stimulus was presented haptically and when it was presented visually, the first stimulus being presented haptically in both cases (Experiment 1), thus requiring movements of the forefinger and forearm. Evidence indicates that active movement enhances the representation and tactual recognition of shape (Millar, 1991). Thus, interpolating irrelevant movements in delay periods significantly distorts the recall of target movements (Adams & Dijkstra, 1966; Millar & Ittyerah, 1991).

Finally, comparison of results of Experiments 1 and 2 shows cross-modal performance to be better when the remembered stimulus was haptic and the comparison was visual (T–V; Figure 2) than when the remembered stimulus was visual and the comparison haptic (V–T; Figure 3). The reason for the difference is not obvious because haptic rehearsal had little effect on performance, relative to baseline (unfilled delay), in Experiment 1, and visual rehearsal had little effect on performance, relative to baseline (unfilled delay), in Experiment 2. In a recent study, Zuidhoek, Kappers, Van der Lubbe, and Postma (2003) found that a delay of 10 seconds between setting a test bar parallel to a reference bar using haptic information in a horizontal plane improved performance as compared to the no delay condition. Zuidhoek et al. interpreted the improvement as a shift from egocentric towards an allocentric frame of reference during the delay period. In the present study, we found that haptic and haptic–visual comparisons were better than visual and visual–haptic comparisons after delay, although the difficulty of discrimination between pairs of stimuli was mostly confined to smaller step sizes that may have made curvature discrimination more difficult for vision. Vandierendonck, Kemps, Fastame, and Szmalec (2004) report different effects of quite similar concurrent computer key tasks in the recall of Corsi blocks presented visually and attribute their findings to differences in the working memory components. All this indicates that concurrent tasks of a similar nature can affect the memory for visual or tactile information differently.

It is conceivable that the differences in mnemonic representations for haptic and visual curvature are related to differences in haptic and visual perceptual processing. Haptic processing depends on movements of the hand or finger over the stimulus and therefore is subject to spatial constraints on motion, whereas visual processing is often global (Navon, 1977) and the sensory detection and discrimination are limited by the physical properties of the stimuli themselves, such as photon counting at low intensities (De Vries, 1943). In linear movement tasks, the demands on memory involve the starting and ending locations of the movement (Laabs & Simmons, 1981; Millar, 1994). For haptically felt curvature, the demands on memory seem to be confined to the slope differences over the far ends of the stimulus (Gordon & Morison, 1982; Pont et al., 1997, 1998) for both static and dynamic touch of curved strips (Pont et al., 1999). Subsequently, Louw, Kappers, and Koenderink (2000) measured haptic discrimination between flat and curved surfaces with both concave and convex Gaussian profiles. When the spatial width of the Gaussian profile was greater than 1 mm, discrimination threshold (amplitude) increased as the 1.3 power of spatial width, for both concave and convex stimuli. Louw et al. concluded that the haptic system is sensitive to the local slope of the stimulus (see also Pont et al., 1998, 1999).

The present findings suggest, in-turn, that memory for haptic and perhaps for visual representations of curvature, perhaps representations of slope, may be particularly disrupted by tasks that involve spatial processing and movement – as assessed with the dual task paradigm used to test for the demands of attention on the primary task (Brown, 1958; Peterson & Peterson, 1959). Filling the delay interval with a spatial or a movement task in the haptic and visual modalities produces modality-specific interference (e.g. Logie, 1986). These results, though specific to the present tasks, may be generalized as being important characteristics in the perception of the curvature of objects. Besides, memory for information from touch and movement has been demonstrated by showing effects of coding texture for unfamiliar shapes and kinesthetic coding for unfamiliar movements (Millar, 1999). This indicates distinct effects for tactile memory. Coding can also involve the mental rehearsal of movements (Millar & Ittyerah, 1991), showing modality-specific aspects of the input information. However, the informational conditions in which such coding depends differ from those in which visual cues are present. Nevertheless, haptic representations are not recoded into visual coordinates, since impaired memory for recognition in one modality is generally dissociated from performance in the other modality (Farah, 1990; Reed, Caselli, & Farah, 1996).

According to Norman et al. (2004), touch and vision are similar in that they are both sensitive to an object's global or overall shape (Lakatos & Marks, 1999; Norman, Dawson, & Raines, 2000), though the human ability to perceive differences in local metric structure between objects such as depth and curvature is not accurate (Norman et al., 2004). Therefore, there are important similarities between vision and haptics. This may suggest that the visual and haptic systems may be interconnected (e.g. Amedi et al., 2001). If touching an object should trigger some visual representation of it, then seeing an object might have a similar effect on the haptic system. In the present study, however, the haptic recall of curvature was better with delay than was visual recall indicating different effects between modalities with delay. The results of the present study indicate that visual and haptically obtained information about curvature is likely retained in both spatial and movement formats. This does not necessarily indicate that the representations are shared between modalities or that they are recoded from one modality to another, but could indicate that certain shape characteristics are perceived similarly within and between modalities.