Roles of familiarity and novelty in visual preference judgments are segregated across object categories

Abstract

Understanding preference decision making is a challenging problem because the underlying process is often implicit and dependent on context, including past experience. There is evidence for both familiarity and novelty as critical factors for preference in adults and infants. To resolve this puzzling contradiction, we examined the cumulative effects of visual exposure in different object categories, including faces, natural scenes, and geometric figures, in a two-alternative preference task. The results show a clear segregation of preference across object categories, with familiarity preference dominant in faces and novelty preference dominant in natural scenes. No strong bias was observed in geometric figures. The effects were replicated even when images were converted to line drawings, inverted, or presented only briefly, and also when spatial frequency and contour distribution were controlled. The effects of exposure were reset by a blank of 1 wk or 3 wk. Thus, the category-specific segregation of familiarity and novelty preferences is based on quick visual categorization and cannot be caused by the difference in low-level visual features between object categories. Instead, it could be due either to different biological significances/attractiveness criteria across these categories, or to some other factors, such as differences in within-category variance and adaptive tuning of the perceptual system.

One of the essential components of adaptive behavior is ability to shape actions in accordance with internally stored preferences. Some preferences can be set initially by genetic and developmental programming, such as the orientation preference toward face-like stimuli exhibited by human infants (1). Preferences are also shaped by previous experience, and such influences are best understood in experimental contexts where actions have explicit consequences, such as operant conditioning. However, preferences also operate in contexts that lack direct forms of reinforcement or punishment. We find certain visual objects aesthetically pleasing, and our preferences change with further exposure to the same or other objects (2, 3). In such cases, it can be difficult to parse the rules by which preferences are modulated, but two basic factors have been variously proposed. First, preferences often are said to be driven by novelty; a novel stimulus is preferred over an old one by both infants (“dishabituation”) (4) and adults (5). Second, preferences often are said to be driven by familiarity; many studies have confirmed that repeated exposure to a stimulus/object monotonically increases the stimulus's/object's attractiveness (“mere exposure effect”) (6–8). Although these two factors are in direct opposition, few attempts have been made to reconcile their contributions, except for some studies in the domains of developmental changes (9, 10) and the effects of long delay periods (11). Very few studies have directly pitted novelty against familiarity as the two alternatives in a preference task, and most studies typically choose just one object category (such as faces or natural scenes), neglecting a potential differences across categories.

Results

We conducted a sequential preference-judgment experiment, in which subjects were presented with two stimuli side by side on a CRT display (Fig. 1A; Fig. S1 provides additional sample stimuli for all object classes). In each trial except for the first of each block, one of the stimuli was old, and the other was new to the subject. Each block of trials consisted of stimuli within a particular object category— faces, natural scenes, or geometric figures. Thus, for instance, the same face was repeatedly presented throughout a block of 26 trials, but always paired with a new face in a randomized lateral arrangement. The old stimulus for each block was of median attractiveness according to preliminary rating data. The subject's task was to indicate which stimulus appeared more preferable, as well as the strength of the relative preference, by clicking a computer mouse with a cursor on a 7-point scale (from −3 to 3).

Fig. 1.

Preference judgment experiment. (A) Stimuli and procedure. A trial consisted of the three steps: (i) initial fixation of 1 s, (ii) free visual inspection until decision, and (iii) reporting of relative preference on a 7-point scale using a mouse. The next trial was essentially the same, except that one of the faces was presented again at a random position (left or right) and paired with a new face. Thus, the same stimulus was presented in all of the 26 trials, but paired with a new stimulus in each trial. (B) Time course of subject preference ratings through a block, averaged for each object category (n = 11). The mean value of the 7-point rating is plotted against trial number. The top of the abscissa indicates a stronger preference for familiarity, whereas the bottom indicates a stronger preference for novelty. Orange represents faces; green, natural scenes; maroon, geometric figures. Error bars represent ± SEM.

The results are shown in Fig. 1B, which plots the average time course of choice ratings within a block, with each object category plotted separately. The positive values along the ordinate denote preference for the familiar stimulus, whereas the negative values denote preference for the novel stimulus. The time courses of subject preferences were different for each object category; with face stimuli, subjects showed a preference for the familiar stimulus that accumulated as the block progressed, whereas for natural scenes, subjects showed a preference for the novel scene that developed quickly and then saturated. Thus, these two categories showed opposing tendencies. The geometric figures fell somewhere in between, with a slight tendency toward novelty preference in the second half of the block (the 16th trial and after). Differences in the slope of linear regression for the first four trials (one-way within-subjects ANOVA, P < 0.001), the mean of the last 22 trials (P < 0.01), and the final trial (P < 0.001) were all significant across the three object categories, whereas those for the very first trial were not significant, as expected (P = 0.103). These results are not consistent with the findings of studies that have demonstrated mere exposure effects (i.e., familiarity preferences) in a wide variety of objects, including geometric figures (7, 12). The difference in results might have been caused by differences in tasks, such as the nature of the task (i.e., passive viewing or subjective attractiveness judgment) or the structure of the task [i.e., absolute rating or two-alternative forced choice (2AFC)]. Moreover, the current result is not consistent with conjectures from developmental studies that familiarity preference dominates earlier in experience, but then gradually shifts to novelty preference with excessive experience regardless of the type of stimuli (9, 10). This may be due to the different processing capacities of adults and infants (13).

What properties of these stimulus classes contribute to the disparate preference patterns? The observed segregation of preference across object categories might be caused simply by their difference in low-level visual features, such as luminance, color, texture, and spatial frequency. To test this possibility, we converted all of the face and natural scene pictures used in this study into line drawings (Fig. 2A) and repeated the experiment. As shown in Fig. 2B, the category-specific effect of memory on preference was clearly preserved with the line drawings. The separation of preference became even larger if the subcategory “lake” of natural scene, which turned out to be the hardest to recognize, was excluded from analysis. Compared with the line drawings of faces, the line drawings of natural scenes had a larger mean and variance in terms of number of pixels used to draw lines (Fig. S2A). However, even when only the line drawings of natural scenes that had a comparable number of pixels as the line drawings of faces were chosen, the tendency for novelty preference was still observed (Fig. S2B).

Fig. 2.

Line drawing experiment. (A) Original and line drawing version of face and natural scene image samples. (B) Results of preference judgment with line drawings (n = 10). Relative preference is plotted against serial trial numbers in each category, as in Fig. 1B. Orange represents faces; light green, natural scenes including all 8 subcategories; green, natural scenes without “lake” subcategory, which subjects found most difficult to recognize. Error bars represent ± SEM.

Although line drawings can effectively exclude the influence of some low-level features, such as luminance, color, and texture, they might not do so for spatial frequency. Thus, we analyzed the spatial frequency components of all of the original face and natural scene images. Spatial frequency profiles for the face stimuli were all quite similar, whereas those of the natural scenes were more varied (Fig. S3A, Left). We chose a subset of natural scenes with the same peak spatial frequency as faces, resulting in nearly matching profiles (Fig. S3A, Right). Even when their spatial frequency components were matched, the segregation of preference across faces and natural scenes was well maintained (Fig. S3B). These results suggest that differential distribution of low-level physical attributes of face and natural scene images is not the main cause of the observed effects.

The clear segregation of the three categories of objects can be interpreted in terms of differences in ecological significance. That is, people may evaluate different aspects of the stimuli when they make preference decisions regarding faces as opposed to natural scenes; for instance, people may implicitly judge happiness or trustworthiness specifically on faces. Another possibility is that the distinctively holistic and orientation-specific nature of face perception might be related to a unique bias toward familiarity. To examine these factors, we repeated the experiment for faces and natural scenes with all of the stimuli upside down. The results are shown in Fig. 3). For comparison, the original upright results and a larger set of upright results pooled from several experiments are plotted as well. The pattern of results closely follows the pattern seen in the upright results. Differences in slope of linear regression for the first four trials (one-way within-subjects ANOVA, P < 0.01), the mean of the last 22 trials (P < 0.01), and the final trial (P < 0.01) were all significant across the three object categories. As expected, the differences in very first trial were not significant (P = 0.328). There was no statistically significant difference between the upright and upside-down conditions.

Fig. 3.

Results with upside-down stimuli (n = 11). Relative preference is plotted against serial trial numbers. Red denotes upside-down faces; light green, upside-down natural scenes. For comparison, the results with upright stimuli (orange, faces; green, natural scenes) are replotted from Fig. 1B. The dark-red curve for faces and the dark-green curve for natural scenes represent the data pooled from several different experiments (n = 42), all with upright stimuli and procedures that were identical up to the 15th trial. Error bars represent ± SEM.

In this inversion experiment, there was no limitation on viewing time, so as to match the conditions of the main upright experiment. However, it is possible that the long exposure to the inverted stimulus might have hidden the potential effects of inversion. To test this possibility, we made two changes in our paradigm. First, the two pictures of an old--new pair were presented sequentially at the screen center as opposed to simultaneous/lateral presentation in the original paradigm. The presentation order of the old and new stimuli was randomized. Second, the presentation time of each picture was limited to 250 ms as opposed to the unlimited presentation in the original paradigm. The pattern of results from this new paradigm (serial and brief presentation) with both the upright and the upside-down configurations was virtually the same as that from the original paradigm (simultaneous and unlimited presentation) (Fig. S4).

Thus, familiarity preference dominates faces and novelty preference dominates natural scenes whether or not stimuli are presented simultaneously or serially, in a self-paced manner or briefly, or upright or upside-down. These results suggest that the segregation of novelty and familiarity preference between the object categories is robust, and that the familiarity dominance in faces is not due to the holistic and orientation-specific nature of faces.

The results to this point indicated a strong effect of stimulus exposure on preference. However, the time scale of decay in such effects, and whether or not the object categories differ in this regard, remained unclear. To address these issues, we repeated the same experiment but added either a 1-wk or 3-wk blank between the 15th and the 16th trials. The results with a 3-wk blank are shown in Fig. 4. The effect of a 1-wk blank was qualitatively similar, but somewhat weaker (Fig. S5). The preference returned to the neutral level in all three categories after the blank, followed by a repetition of the initial cumulative changes. The segregation between familiarity and novelty preferences was maintained in the same way as in the initial tendencies. The geometric figures behaved more like natural scenes. Although the main effect of test day was not significant (two-way within-subjects ANOVA, P = 0.275), category main effect and interaction were significant (P < 0.01 and P < 0.05, respectively). In some pilot explorations, we found no such effects for blank periods, which extended several trials to several minutes. The results seem to be consistent with an intermediate-term (several weeks) modulation. They are distinctly different from a previous report of a reversal from novelty to familiarity preference of the same objects with blank periods ranging from several minutes to 1 y (11). What is more noteworthy is that the effects of the blank are symmetrical for faces and natural scenes even though the sign is the opposite, suggesting that novelty and familiarity preference might be modulated with similar temporal dynamics.

Fig. 4.

Results of preference judgment with a 3-wk blank (n = 11). Relative preference is plotted against serial trial numbers. Orange represents faces; green, natural scenes; maroon, geometric figures. The absence of lines connecting the 15th and the 16th trials indicates that there was a 3-wk blank between the two trials. Error bars represent ± 1 SEM. The blank seemed to be sufficient to cancel out the memory effects in all of the categories, and the same cumulative effects were duplicated over trials after the blank.

Discussion

Our findings demonstrate a decisive segregation of experience-dependent preference across visual stimuli from different object categories—familiarity preference for a set of face stimuli and novelty preference for a set of stimuli depicting natural scenes. These results were replicated across a wide range of stimulus manipulations, demonstrating the robustness of this divergence in preference patterns. Specifically, the same results were found even after stimuli were turned into line drawings, matched for spatial frequency, inverted, and switched to a brief serial presentation format. These manipulations also allowed us to narrow down the stimulus properties that might contribute to the divergence in preference.

The results with line drawings argue against the role of lower-level visual factors, such as luminance (gray level), color, and spatial distributions. The inversion results argue against a local/holistic difference between perceptions of natural scenes and faces as the critical factor for preference segregation. In particular, the robustness of preference segregation even with the 250-ms duration of stimulus might be related to findings of quick visual categorization (14). Likewise, the results of the blank experiment, along with our preliminary failure to show weakening effects of either familiarity or novelty preference with several trials or minutes of blank, indicate that the memory-based mechanisms involved have a relatively well-sustained or slow decaying function. Overall, our results are most consistent with a quick perceptual categorization mechanism that requires only a very short stimulus exposure but nonetheless feeds into an intermediate-to-long recognition memory.

The opposing memory effects between faces and natural scenes that survived line drawing manipulation, stimulus inversion, and limited exposure time may provide another line of evidence for domain specificity in visual perception, which could be due to differing biological significance across these categories.

The possibility of another explanation for the different preference patterns across object categories exists. Faces vary little in lower-level, local features. Relative to faces, natural scenes vary much more widely in shapes, colors, scales, and other features and thus are typically novel. Moreover, faces are massively experienced and learned for fine discrimination starting in the early developmental period, with great social significance. The geometric figures that we chose might be intermediate in terms of within-category variances. If the human visual system is flexibly tuned for efficient coding (15, 16) and attractiveness is defined as maximum information to acquire, such characteristics may modulate the attractiveness toward novelty, but relative to different functional distances in the parameter space. This interpretation may explain the familiarity/novelty segregation across object categories with a single mechanism. Although highly speculative and incomplete, it is consistent with our finding that not only do the upside-down stimuli behave similarly to the upright, but also the time scale of dynamic changes with a blank are similar between faces (with familiarity dominance) and natural scenes (with novelty dominance).

Either way, we found a very robust segregation of familiarity and novelty dominances in faces and natural scenes, respectively. Neuroimaging studies have suggested that different brain areas are responsible for perception of faces and natural scenes (17). Our results raise the question of whether familiarity and novelty preferences are processed in separate neural circuits.

Methods

Subjects.

A total of 22 normal adult subjects (7 females and 15 males) participated in the preliminary attractiveness rating: 12 for faces, 5 for natural scenes, and 5 for geometric figures. A total of 82 subjects (35 females and 47 males) participated in all seven experiments: 11 in the main preference experiment, 10 in the line drawing experiment, 11 in the upside-down experiment, 17 in the serial version with upright images, 12 in the serial version with upside-down images, 10 in the 1-wk-blank experiment, and 11 in the 3-wk-blank experiment. Each subject participated in only one experiment.

Preliminary Attractiveness Rating.

Faces.

Faces were generated by FaceGen software (Singular Inversions) for each of 16 subcategories: 4 races (African American, Indian, Asian, and European) × 2 sexes and 2 ages (young and old). Two examples of young female Asian faces are shown in Fig. 1A. The attractiveness of these faces was rated on a 7-point scale and averaged across the 12 subjects (4 females and 8 males). Because our preliminary observers reported that these artificial faces are not so attractive in general, we picked the 27 most attractive faces in each subcategory for the experiments, and used the 14th-place (median) face as an “old” stimulus. This choice was designed to adjust the baseline probability of novelty and familiarity preferences at the chance level of 50%. This was also done for the natural scenes and geometric figures.

Natural scenes.

Photos were collected from websites in the following eight subcategories: mountain, river and lake, sky, ocean and beach, animal, food, desert, and flower (Fig. S1A). The photos were selected mainly to provide more variety in each subcategory. The attractiveness of each image was rated on a 7-point scale, and averaged across the five subjects (two females and three males). We then picked the 27 most attractive scenes for the experiments, and used the 14th- place (median) scene as the old stimulus.

Geometric figures.

More than 50 Fourier descriptors were generated by a Matlab program (MathWorks) in four categories, symmetrical/asymmetrical × simple/complex (Fig. S1B). Their attractiveness was rated on a 7-point scale and averaged across the five subjects (one female and four males). We then picked the 27 most attractive figures for the experiments, and used the 14th-place (median) figure as the old stimulus.

Main Preference Experiment with Upright Stimuli.

According to the preliminary attractiveness rating described above, the median face (that ranked 14th in attractiveness out of 27) was paired with the remaining 26 faces with randomized order and locations (left or right) to be used in 26 trials in a block (for each subcategory of faces). The same procedure was applied to generate 26 pairs in each subcategory of natural scenes and geometric figures. Thus, the same median stimulus was presented in 26 trials, but with a new stimulus each time. In all of the three object categories (and thus in three blocks of the experiment), the subject performed the preference decision task for eight sub-blocks. For faces, each subject's own race was always included, and one other race was randomly selected from the remaining three races. The two sexes and the two ages were always included. Thus, eight sets of 26 face pairs (old vs. new) were constructed and were assigned to eight sub-blocks in a random order. For natural scenes, eight sets were constructed for right subcategories and assigned to eight sub-blocks in a random order. For geometric figures, there were only four types (two levels of complexity × two levels of symmetry), so two sets of 26 stimulus pairs were constructed in each type; thus, all told, eight sets were constructed and assigned to eight sub-blocks in a random order. Faces, natural scenes, and geometric figures were tested separately in three blocks, with the order counterbalanced across participants.

In each trial, a pair of stimuli was presented side by side, as illustrated in Fig. 1A, and the subject was asked to judge relative preference on a 7-point scale, with −3 or +3 as a strong preference for the left (or right) stimulus and 0 as neutral. Because one old stimulus and one new stimulus were always presented in each trial, the subject's judgment was then encoded as familiarity or novelty score. (In the first trial, the old stimulus was only nominal.) We then averaged the familiarity-novelty scores on the 7-point scale across subcategories and across subjects for each object category at each sequential trial number. The results are shown in Fig. 1B.

We applied one-way within-subjects ANOVA to the four selected data sets (i.e., slope of the first 4 trials, mean of the last 22 trials, and the very first and last trials) to examine for statistical differences across the object categories. We also converted the 7-point-scale data into 2AFC values, and reapplied the same statistical tests to evaluate the robustness of the statistical results. The results were qualitatively the same as those from the original analysis based on the 7-point-scale data.

Experiment with Line Drawings.

All of the face and natural scene images used in the main experiment were converted into line drawings using Matlab (MathWorks). The parameters for this conversion were kept constant within a category and across categories. Other than the stimuli used, the procedure was identical to the main experiment. Because of the large differences across subcategories of natural scene in terms of ease of recognizing the line drawing version of natural scenes, we collected subjects’ rating on the ease of recognition for each of eight subcategories. The “lake” subcategory had the lowest score. Responses were obtained with button press in a 2AFC rather than on a 7-point scale.

Experiment with Upside-Down Stimuli.

The stimuli and the procedure were identical to those in the main experiment except that we ran the experiment only for faces and natural scenes (not for geometric figures) and we presented the sets of stimuli (identical to those in the main experiment) upside-down. We applied one-way within-subjects ANOVA to the data to investigate for statistical differences across the object categories. The statistical procedures were the same as those used in the original experiment with upright stimuli. We converted the 7-point-scale data into 2AFC values and then reapplied the statistical test to evaluate the robustness of our conclusions. The results were qualitatively the same as those of the original analysis based on the 7-point-scale data. We also applied two-way ANOVA [condition (upright vs. upside-down) × object category] to check for any significant differences between the two conditions. Our findings are presented in Results.

Serial and Brief Presentation with Upright and Upside-Down Stimuli.

The stimuli (upright and upside-down) were identical to those used in previous experiments, but the procedure for stimulus presentation was changed in this experiment. Each stimulus of a pair (old and new) was first presented serially one by one, then presented once for 250 ms before the decision. The temporal order of old and new stimuli was randomized across trials.

Experiment with a Blank.

The stimuli and the procedure were identical to those in the main experiment, except that we stopped after the 15th trial in each stimulus category. After a 1-wk or 3-wk delay, the subject was brought back, and the 16th–26th trials in each category were rerun. The results with the 3-wk delay are shown in Fig. 4, and the results with the 1-wk delay are shown in Fig. S5.

We took the data for the last trial of day 1 and the first trial of day 2 and applied two-way (test day × object category) within-subjects ANOVA to test for a significant gap effect or an interaction. We also converted the 7-point- scale data into 2AFC values and reapplied the same statistical tests. The results were qualitatively the same as those from the original analysis based on the 7-point-scale data.