The Effects of End-of-Day Picture Review and a Sensor-based Picture Capture Procedure on Autobiographical Memory using SenseCam

Abstract

Emerging “life-logging” technologies have tremendous potential to augment human autobiographical memory by recording and processing vast amounts of information from an individual’s experiences. In this experiment undergraduate participants wore a SenseCam, a small, sensor-equipped digital camera, as they went about their normal daily activities for five consecutive days. Pictures were captured either at fixed intervals or as triggered by SenseCam’s sensors. On two of five nights, participants watched an end-of-day review of a random subset of pictures captured that day. Participants were tested with a variety of memory measures at intervals of 1, 3, and 8 weeks. The most fruitful of six measures were recognition rating (on a 1–7 scale) and picture-cued recall length. On these tests, end-of-day review enhanced performance relative to no review, while pictures triggered by SenseCam’s sensors showed little difference in performance compared to those taken at fixed time intervals. We discuss the promise of SenseCam as a tool for research and for improving autobiographical memory.

The study of human cognition has much to gain from advances in technology. Increasingly compact and powerful devices—such as SenseCam, a wearable, sensor-equipped digital camera—enable the automated chronicling, or “life-logging,” of many aspects of a person’s daily experience (cf. Bell & Gemmell, 2007, 2009). One research domain in which such technology has great potential is the study of autobiographical memory: “memory for information related to the self” (Brewer, 1986, p. 26). Such research explores the ways in which humans maintain and remember personal experiences, and how those memories serve to “support a coherent and effective self” (Conway, 2005, p. 596).

Serious methodological challenges are posed by the study of autobiographical memory, as peoples’ everyday experiences are not experimentally controlled, can span decades, and are largely unknowable to others (cf. Brewer, 1986; 1996; Conway, 1990). Galton (1879a, 1879b) created a technique for the study of autobiographical memory (later modified by Crovitz & Schiffman, 1974), that used experimenter-chosen probe words to elicit participants’ associated memories. This method suffers from potentially unrepresentative sampling of memory and inconsistency in retrieval and response strategies across participants. Another approach to the study of autobiographical memory used the contents of long-running daily diaries to cue and assess memory (cf. Linton, 1975; Wagenaar, 1986); but this method suffers from biased selection of experiences by the participants themselves. Advances in technology enabled the advent of experience sampling methods (cf. Conner, Barrett, Tugade, & Tennen, 2007), such as Brewer’s (1988) experiments in which participants carried small electronic devices that prompted them at random intervals to record certain aspects of their current experience on paper response cards. Portions of these cards were later used to test memory. This method suffers the disadvantage of frequently and actively involving participants in the capture of information, potentially altering the very memories it sets out to measure. SenseCam can unobtrusively capture a pictorial record from the lives of individuals, thus opening a new route to the investigation of autobiographical memory.

SenseCam also has potential for augmenting human cognition: as a cognitive prosthetic for people with memory impairments (Berry et al., 2007) or a memory aid for those with unimpaired memory (Sellen, Fogg, Hodges, & Wood, 2007). In this paper, we consider how the capabilities of the device and the nature of its use can serve to investigate and enhance autobiographical memory. Specifically, we evaluate whether end-of-day review of SenseCam pictures enhances memory for the day’s events, and also whether the sensors built into SenseCam capture more memorable pictures than those captured using a simple interval timer.

One way in which SenseCam can augment memory is by providing pictures with which one can easily review a day’s experiences. It has long been known that repeated presentation of stimuli improves memory (Ebbinghaus, 1885/1913). More specifically, prior work has shown that reviewing still pictures from a previously viewed movie of everyday events enhances performance on subsequent free recall and recognition (Koutstaal, Schacter, Johnson, Angell, & Gross, 1998; Schacter, Koutstaal, Johnson, Gross, & Angell, 1997). Koutstaal, Schacter, Johnson, and Galluccio (1999) found similar effects on free recall (Experiment 1) and recognition combined with cued recall (Experiment 2) when participants performed a variety of activities in the laboratory and then reviewed photographs of actors performing those same activities. The current experiment sought to extend these findings to the investigation of unconstrained personal activities, review of first-person pictures, and longer retention intervals. We predicted that an end-of-day review of SenseCam pictures would improve memory.

Life-logging technologies like SenseCam can produce a wealth of durable external personal memory; however, this raises new problems of information management, organization, and searching (Jones & Ross, 2007). One way to ameliorate these problems is to strategically filter which data are captured. SenseCam can do so by using sensors (white light, passive infrared, and accelerometer) to capture pictures potentially relevant for future memory needs. If there is a systematic relationship between objectively measurable physical characteristics of experience (e.g., incidence of white light) and psychological characteristics of experience (e.g., memory for an event), and if SenseCam’s sensors and triggering algorithm are well calibrated to this relationship, then we would expect sensor-triggered pictures to elicit memory better than pictures simply captured at regular intervals throughout the day.

The goal of the present experiment was to gather data that would allow us to (a) evaluate a variety of potential memory measures, (b), compare memory for days that had been reviewed with memory for days that had not been reviewed, (c) compare memory for a set of pictures triggered by SenseCam’s sensors with a set taken at regular time intervals, and (d) to examine the course of forgetting under each of these conditions.

Method

Participants

There were 12 undergraduates who participated for payment of $50 each. There were 4 males and 8 females, ages 18 to 21.

Recruitment and selection

Participants were recruited via posters and web notices. Of the 24 undergraduates interviewed and who returned completed questionnaires, 12 participants were selected on the basis of their punctuality, ability to follow directions precisely, self-rated responsibility in caring for electronic devices, and having living situations and daily activities that would least interfere with wearing a SenseCam for a considerable portion of each day.

Privacy

Several aspects of the experimental procedure were designed to safeguard the rights and privacy of participants and others. Potential participants were told to complete the initial questionnaire only if they were comfortable with the full procedure as described in the interview. Each participant’s data were identified only by a unique code number assigned to each individual. Pictures from an individual participant’s acquisition period were never seen by anyone except the experimenters and that participant.

Participants could deactivate the SenseCam (using the on/off button or the privacy button) at any time that they chose to avoid the possibility of pictures being captured. Pressing the privacy button deactivates picture capture for 7 min, with a warning beep sounding 15 s before reactivation. Participants were further instructed to deactivate the SenseCam in several specific situations: restroom, changing/locker room, doctor’s office, ATM or bank, and any time someone else requested that it be deactivated. Furthermore, when entering their workplace or a private residence, participants were instructed to deactivate the camera until they received permission to reactivate it. Finally, participants could note time periods for deletion in the small notepad they carried (e.g., if they suddenly realized they had left the SenseCam on while using the restroom). All pictures falling in such time periods were deleted at the end of each day without having been seen by anyone.

Apparatus

SenseCam is a small, wearable, sensor-equipped digital camera developed by the Microsoft Research lab in Cambridge, UK. The SenseCams used in this experiment were hardware version 2.3b and firmware version 2.6.7. A built-in adjustable lanyard allows SenseCam to be worn around the neck at upper chest height. Pictures are captured using a fisheye lens with a wide angle of view (119° diagonal) and stored with timestamps on a 1Gb SD flash memory card. The pictures are saved as JPEG files with display size 640×480 pixels, resolution 72 dots per inch, color depth 32-bit (16.7 million colors), and file size approximately 30 kilobytes. The maximum rate of picture capture is approximately once every 5 s. Capture may be triggered by three different methods, each of which can be independently enabled or disabled by experimenters: (a) manually (by pressing the manual trigger button); (b) by timer; and (c) by sensors. The device gives no audible or visible indication of picture capture due to timer or sensor triggering, although it beeps upon manual triggering.

SenseCam Sensors

SenseCam has sensors for measuring: (a) temperature (in units of .5° C); (b) white light intensity (on a scale of 0 to 10,000); (c) change in infrared radiation (e.g., from a moving warm body) as either presently occurring (1) or not (0) within a range of approximately 2.5 m for a human-sized object; and (d) acceleration in three dimensions (in units of 1/819th g). Data from these sensors are recorded, along with date and time, approximately every 1.5 s. Pictures and sensor data are transferred to a computer via a USB interface, which is also used for recharging the battery.

Sensor-Triggering Algorithm

The default algorithm and threshold values for sensor-triggering of picture capture were established by Microsoft Research through informal testing. The algorithm compares current sensor values to the set of values most recently measured (typically about 1.5 s prior). If a picture is not already currently being captured and saved, a new capture will be triggered if either (a) there is a change in white light level greater than 4,500 units as measured by the detector, or (b) the change in accelerometer value on all three axes is no greater than 500 units as measured by the detector (indicating that the device is probably stationary) and the passive infrared value is 1 (indicating change in infrared, typically due to nearby human movement). More detailed specifications can be found in Hodges et al. (2006).

Design

The experiment used a 3 × 2 × 2 within-subjects factorial design with 3 retention intervals (1, 3, 8 weeks); 2 types of trigger condition (sensor-triggered, timer-triggered); and 2 types of review condition (no review, review). The order in which participants received each trigger and review condition was counterbalanced using a balanced Latin Square.

Procedure

Acquisition Period

Participants wore a SenseCam as they went about their normal daily activities for five consecutive weekdays (i.e., Monday through Friday). The first day was treated as practice, and pictures from that day were not reviewed or tested. SenseCam operated automatically throughout the day, capturing pictures according to the trigger condition for that day (described below).

The starting and ending time of acquisition for each day were determined by each individual participant, and thus varied by participant and by day. For the four experimental days, the mean starting time was 09:55 (SD = 1 hr 11 min), the mean ending time was 19:17 (SD = 1 hr), and the mean duration of acquisition per day was 9 hours and 22 min (SD = 1 hr 37 min, range = 5 hr 13 min to 12 hr 43 min).

Participants were informed before the acquisition period that their memory for that period would later be tested, and were told the general nature of the questions they would be asked (e.g., “You may be asked to write down details about the events that occurred during the days on which you wore the SenseCam, either cued by images from those days or with no such cues”).

At the end of each day participants returned to the lab. The experimenter transferred all of the day’s pictures from the SenseCam onto a computer, deleted any pictures falling in a time period marked for deletion by the participants in their notepads, and updated the SenseCam’s trigger setting for the following day. Participants then received a review of that day’s pictures (described in detail below), if the day had been assigned to be a review day. This entire procedure was repeated for five days.

Trigger condition

Manual triggering of picture capture was disabled for the experiment. On any given day, the SenseCam captured pictures either as triggered by its sensors (see Apparatus), or as triggered by its fixed-interval timer. The measured mean timer interval for the timer-triggered condition was 10.6 s (SD = 1.3). Note that because timer-triggered pictures were not triggered by any external criteria, and because pictures were randomly selected for review and for testing, this condition provides a de facto random sampling of experience and thus serves as a logical control condition against which to test sensor-triggering.

End-of-day review

The end-of-day review was a movie composed of a sequence of randomly selected pictures captured by the SenseCam on that day for that participant. The review was shown on a computer monitor with pictures presented one-at-a-time for 1 s each in chronological order. Participants were told that they would be watching a movie composed of a sequence of pictures taken throughout that day; they were given no special instructions on how to process the review. Each participant received two end-of-day reviews, one on a sensor-triggered day and one on a timer-triggered day.

The number of pictures used to create the review movie was half of the number of pictures captured on the practice day for that participant (which was always the first day and used sensor-mode without review). We could not use the same number of pictures for all participants because it was not known ahead of time how many pictures would be captured for each participant on each day. Our main concern was to ensure that, for each individual participant, the same number of images was used for both end-of-day reviews. A participant’s first day of SenseCam use provided an estimate of the number of daily pictures we could expect to be captured for that individual. To account for the possibility of fewer pictures being captured on subsequent days, we conservatively set the number of pictures to be used for review as half of the number captured on each participant’s first day (M = 1,022, SD = 235). The mean duration of the reviews was 17 min 2 s (SD = 3 min 55 s).

Memory measures

Memory for events that occurred during the acquisition period was tested with six different test procedures:

Recollection judgment

Participants were shown a picture and made a yes/no judgment in response to the following instructions: “Recollective memory is when you are able to recall a specific episode or moment from your life. It is almost like reliving or re-experiencing the past event. Please indicate whether you have a recollective memory for this picture.”

Recognition rating

Participants were shown a picture accompanied by the following instructions: “Please rate the strength of your memory for the scene(s) and/or event(s) depicted by this picture.” Participants responded using a 1–7 scale, with 1 labeled as “no memory” and 7 labeled as “extremely strong memory.” In contrast to the standard old-new recognition method used in laboratory experiments, no lures were presented. Since lures could be made arbitrarily easy or difficult to correctly reject (e.g., a picture taken on the Moon vs. a picture taken from 10 cm to the participant’s right), we saw no value in including them.

Picture-cued recall

Participants were shown a picture accompanied by the following instructions: “Please describe in detail the visual scene(s)/event(s) as they would appear to the SenseCam in the one minute following this picture. If you can’t remember anything for this picture, you may type a question mark to continue to the next picture.” There was a maximum of 1 min to respond to each item, with no minimum time requirement.

Time-cued recall

Participants were shown a date and time from their acquisition period (e.g., “Tuesday 11:09 AM”) accompanied by the following instructions: “Please describe in detail the visual scene(s)/event(s) as they appeared to the SenseCam at the above day and time. If you can’t remember anything for this day and time, you may type a question mark to continue to the next trial.” There was a maximum of 1 min to respond to each item, with no minimum time requirement.

Recall of time

Participants were shown a picture and asked to give the day and time it was taken. They made a forced choice on day from among the days they wore SenseCam (e.g., Monday, Tuesday, Wednesday, Thursday, Friday), and a forced choice on time of day (12:00 AM to 11:55 PM in intervals of 5 min). A response was required for each field, and participants were instructed to give their best guess if they were not sure.

Temporal order judgment

Participants were shown two pictures arranged vertically on the screen and they made a forced-choice judgment about which was captured first. Paired pictures were selected from the same acquisition day (and thus from the same review condition), with acquisition order counterbalanced.

Memory Test Procedures

Tests were administered individually on computers with instructions presented entirely via onscreen text. Pictures were selected randomly (without replacement) from each participant’s total set of captured pictures for inclusion on the tests for that participant. Pictures were never tested more than once. For a given test session each participant completed the following number of items: Recollection Judgment (192); Recognition Rating (192); Picture-cued Recall (48); Time-cued Recall (8); Recall of Time (96); Temporal Order Judgment (192). Fewer trials were given for the more time-consuming memory measures in order to reduce the total duration of the testing sessions.

Counterbalancing ensured equal representation in each phase of each test for each participant for all manipulated variables, as well as whether or not the picture had been included in the review if it came from a review day. Participants were only shown and tested on pictures from their own acquisition periods. They had been informed about this aspect of the experiment, so they knew that all of the pictures were ones from their lives.

Each test consisted of 4 phases, with participants allowed to take 5 min breaks between phases. Each of the first 3 phases included 5 blocks, with each block consisting of only one type of item (all but the Time-cued Recall items). Block order was determined randomly. The fourth phase consisted of the Time-cued Recall items.

Memory Test Intervals

Participants returned to the lab for testing sessions at retention intervals of approximately 1, 3, and 8 weeks. The mean number of days that the testing sessions occurred after the final acquisition day are as follows: Test 1: 5.67 days (SD = 2.01, range = 4–10); Test 2: 29.50 days (SD = 1.12, range = 27–31); Test 3: 57.50 days (SD = 2.43, range = 52–62).

Payment Schedule

Participants were each paid $50 in cash in four installments: $15 at the end of the acquisition period, $10 upon arriving for the first and the second test sessions, and $15 upon arriving for the third (final) test session.

Results and Discussion

For all tests reported below, an alpha level of .05 was used to indicate statistical significance. Effect sizes for ANOVAs are reported as estimated partial ω² calculated using the formulae provided by Maxwell and Delaney (2004).

Pictures Captured

For the four experimental days, the mean number of pictures captured per day was 2390 (SD = 566, range = 1,358–3,479) in the timer-triggered condition and 2031 (SD = 565, range = 721–3,200) in the sensor-triggered condition. The mean number of pictures captured per hour was reliably greater for the timer-triggered condition (M = 272, SD = 60) versus the sensor-triggered condition (M = 209, SD = 60), t(11) = 3.37.

Of pictures captured in the sensor-triggered condition, 95% were triggered by passive infrared (which can only be triggered when the camera is stationary), 4% were triggered by white light level, and 1% were triggered by both. Thus, for the types of events in the weekday lives of American undergraduates, the sensor algorithm was most often triggered by the passive infrared sensor. This may reflect the sedentary nature of common activities in undergraduates’ daytime weekday lives (e.g., sitting in classrooms, working at computers, watching television) along with their interactions with sources of infrared (e.g., motion of other people or of the participant’s hands in front of the device).

Memory Measures--Scoring Criteria

The six memory measures were scored as follows: (a) Recollection Judgments: 0 for “no” and 1 for “yes,” (b) Recognition Rating: 1 to 7, where 7 was “extremely strong memory,” (c) Picture-cued Recall and Time-cued Recall: amount recalled as response length in number of characters (length = 0 for “?” responses), (d) Recall of Time: 0 to 2, with one point each for correct day and correct time to within one hour, and (e) Temporal Order Judgment: 0 for incorrect and 1 for correct.

Overall data

In the Results and Discussion section we focus on those aspects of the data that let us answer the questions we set out to study. However, for completeness we give the entire data set for all six memory measures as a function of retention interval, review condition, and trigger condition in the Appendix.

Appendix.

Mean and SD of Participant Means of Six Memory Measures by Retention Interval, Review, and Trigger Condition

	1 Week		3 Weeks		8 Weeks
Memory Measure	No Review	Review	No Review	Review	No Review	Review
Recollection Judgment
Sensor	0.41 (0.18)	0.54 (0.19)	0.35 (0.19)	0.38 (0.19)	0.36 (0.16)	0.37 (0.19)
Timer	0.44 (0.19)	0.52 (0.19)	0.31 (0.21)	0.39 (0.19)	0.28 (0.19)	0.36 (0.15)
Recognition Rating
Sensor	3.89 (0.94)	4.14 (0.60)	3.28 (0.94)	3.37 (0.89)	2.97 (1.06)	3.07 (1.17)
Timer	3.61 (0.86)	4.17 (0.78)	3.11 (1.02)	3.34 (0.95)	2.65 (1.03)	3.22 (0.92)
Picture-Cued Recall
Sensor	64.18 (26.52)	68.33 (29.21)	46.25 (26.33)	47.54 (24.27)	38.57 (22.03)	40.44 (25.88)
Timer	51.56 (24.71)	62.53 (24.12)	38.94 (23.23)	45.42 (25.42)	33.53 (22.41)	39.05 (26.05)
Time-Cued Recall
Sensor	78.17 (50.35)	76.38 (54.06)	46.00 (23.16)	55.08 (31.30)	41.29 (19.63)	54.83 (32.89)
Timer	60.17 (50.02)	84.04 (47.40)	46.96 (29.25)	69.88 (31.79)	55.67 (27.08)	48.00 (26.32)
Temporal Order Judgment
Sensor	0.62 (0.09)	0.63 (0.09)	0.63 (0.08)	0.65 (0.08)	0.57 (0.11)	0.64 (0.09)
Timer	0.63 (0.07)	0.65 (0.09)	0.64 (0.07)	0.64 (0.08)	0.58 (0.11)	0.61 (0.09)
Recall of Time
Sensor	1.23 (0.31)	1.27 (0.24)	1.15 (0.34)	1.39 (0.21)	1.11 (0.42)	1.26 (0.21)
Timer	1.03 (0.26)	1.20 (0.29)	1.10 (0.30)	1.24 (0.32)	1.11 (0.39)	1.08 (0.33)

Evaluating the Memory Measures

We assumed that over 8 weeks there would be substantial forgetting, enabling a rough comparison of the sensitivity of our various memory measures. We reasoned that the greater the decline in performance across tests for a particular measure, the greater the degree to which actual retrieval of episodic details was required for that measure, and thus the more sensitive it is as a measure of autobiographical memory. The values for each memory measure were first standardized within participants, to account for individual differences in absolute levels of performance and to allow direct comparison of memory measures. The mean of the participant means of these standardized values, across retention interval, is shown in Table 1. The right-most column shows the difference from the first test to the third test.

Table 1.

Memory Measures: Means of Participant Means of Scores Standardized Within Participants

	Retention Interval
Memory measure	1 week	3 weeks	8 weeks	1 week – 8 weeks
Picture-Cued Recall	.37	−.10	−.28	.65
Recognition Rating	.29	−.07	−.22	.50
Time-Cued Recall	.30	−.10	−.20	.49
Recollection Judgment	.18	−.07	−.11	.29
Temporal Order Judgment	.02	.03	−.05	.06
Recall of Time	.00	.05	−.05	.05

The Picture-cued Recall measure showed the largest amount of forgetting. One would expect a standard probed recall measure for the content of experienced events to be a good response measure. It appears to work well using just the amount written down without scoring for accuracy. The Recognition Rating (1–7 scale) for the pictures was the next most sensitive measure. This is consistent with previous studies of autobiographical memory that found this type of measure to be effective (e.g., Brewer, 1988). The data for the Time-cued Recall task suggest that time was not as good a probe for recall as were the pictures of the actual events. This is somewhat analogous to the findings reported in Brewer (1988) that time was a relatively poor probe cue in autobiographical memory. Furthermore, we think that when given a time and a day of the week, participants could have used knowledge of their weekly schedules during that semester to infer where they would have been and what the visual scene would have been, without necessarily having any specific episodic memory for that time. The simple binary Recollection Judgment shows some modest memory sensitivity. Note that in some ways this measure can be thought of as a binary version of the more sensitive 1–7 recognition scale. The Temporal Order Judgment and Recall of Time measure show little sensitivity to forgetting. Friedman (1993) has made powerful arguments that much time information in autobiographical memory is reconstructed from knowledge of script-based information. We think it is likely that these two measures reflect this type of reconstruction and thus do not show episodic forgetting over the time intervals studied. For example, it would have been trivial for participants to correctly order a daytime and a nighttime picture from the same day, without any recourse to episodic memory. Similarly, the approximate time of day could be inferred from information in the picture along with participants’ knowledge of their schedules.

On the basis of the above analysis, and for the sake of brevity, we have chosen to focus on Recognition Rating and Picture-Cued Recall as the memory measures for the rest of the analyses. They are the top two measures in terms of forgetting effect size, and the use of two different measures (one recognition and one recall) allows us to see how consistent our findings are across different indexes of memory.

Overall ANOVA for Recognition Ratings

A three-way ANOVA was conducted to evaluate the effect of retention interval (3 levels), review condition (2 levels), and trigger condition (2 levels) on Recognition Rating. Mauchly’s test indicated that the assumption of sphericity was met for all main effects and interactions. There was a statistically significant main effect of retention interval, F(2, 22) = 11.19, MSE = 1.07, ω² = .170, and of review condition, F(1, 11) = 5.02, MSE = 0.65, ω² =.028, but not of trigger condition, F(1,11) = 0.54, MSE = 0.69, p = .476, ω² = −.004. No interactions were statistically significant. Follow-up analyses are presented in separate sections below.

Overall ANOVA for Picture-Cued Recall

A three-way ANOVA was conducted to evaluate the effect of retention interval (3 levels), review condition (2 levels), and trigger condition (2 levels) on Picture-cued Recall. Mauchly’s test indicated that the assumption of sphericity was violated for the interaction of retention interval with trigger condition, χ²(2) = 8.08, p = .018, and nearly violated for the main effect of retention interval, χ²(2) = 5.41, p = .067. Analyses were conducted using Greenhouse-Geisser adjusted degrees of freedom for these two effects, but because the results did not differ importantly from analyses assuming sphericity, we present the latter here for the sake of consistency. There was a statistically significant main effect of retention interval, F(2, 22) = 7.09, MSE = 1,017.54, ω² = .129, and of review condition, F(1,11) = 5.24, MSE = 175.18, ω² = .012, but not of trigger condition, F(1,11) = 3.62, MSE = 324.45, p = .083, ω² = .013. No interactions were statistically significant.

Forgetting function

Figure 1 shows the decline in memory performance across the three time intervals for both Recognition Rating and Picture-cued Recall. Mean performance on both measures decreased reliably across the three retention intervals, as confirmed by separate least-square fittings of a power function (P = at^b, where P is performance and t is time) for each participant for Recognition Rating (mean standardized b = −0.22, t(11) = −4.12, one-tailed), and for Picture-cued Recall (mean standardized b = −0.19, t(11) = −2.44, one-tailed). One-tailed tests were used based on the strong theoretical prediction that memory declines over time.

Figure 1.

Mean recognition rating (top) and mean picture-cued recall length (bottom) as a function of mean retention interval for the three testing sessions.

End of Day Review

Figure 2 shows that an end–of-day review improved performance for both Recognition Rating and Picture-cued Recall. Recognition Ratings were reliably higher for reviewed days (M = 3.55, SD = 0.69) versus non-reviewed days (M = 3.25, SD = 0.80). Picture-cued Recall lengths were reliably higher when there had been an end-of-day review (M = 50.55, SD = 20.85) versus no such review (M = 45.51, SD = 18.49).

Figure 2.

Mean recognition rating (top) and mean picture-cued recall length (bottom) as a function of review condition. Error bars represent standard error of the difference between conditions.

Trigger Condition

Figure 3 shows that the default SenseCam sensor settings provide little improvement in selecting items yielding higher memory performance compared to randomly selected items for both Recognition Rating and Picture-cued Recall. Recognition Ratings were not reliably different for pictures captured as triggered by the sensors (M = 3.45, SD = 0.75) versus the timer (M = 3.35, SD = 0.75). Picture-cued Recall lengths were not reliably different for pictures captured as triggered by the sensors (M = 50.89, SD = 20.05) versus the timer (M = 45.17, SD = 19.94). Recognition Ratings were reliably greater for pictures triggered by the passive infrared sensor (M = 3.47, SD = 0.76) versus the white light level sensor (M = 2.92, SD = 1.13), t(11) = 2.83. There were not enough data to conduct the same comparison for Picture-cued Recall.

Figure 3.

Mean recognition rating (top) and mean picture-cued recall length (bottom) as a function of trigger condition. Error bars represent standard error of the difference between conditions.

General Discussion

The results showed substantial forgetting across retention intervals, and also revealed that certain memory measures were better indicators of episodic memory than others (i.e., they were more sensitive to forgetting), as shown in Table 1. Performance was greater for reviewed days than for non-reviewed days, but did not reliably differ for sensor-triggered versus timer-triggered pictures.

Our data suggest that an end-of-day review of an individual’s activities (via pictures captured by SenseCam) increases performance on several memory measures for that day. However, we need a deeper understanding of the factors involved in the review process. Such factors include (a) types of pictures chosen for review, (b) presentation order (e.g., non-chronological ordering might actually enhance memory), (c) presentation speed, (d) self-paced versus experimenter-paced review, and (e) spacing (e.g., the number and scheduling of review presentations). Furthermore, picture review needs to be compared to other forms of review. For example, Koutstaal et al. (1998) found that reviewing a verbal description of events enhanced memory to the same extent as reviewing photographs of the events. Picture review should also be compared to unaided mental review.

As a research tool for autobiographical memory, SenseCam has two important attributes--it can unobtrusively capture a pictorial record of an individual’s experiences, and it has sensors which can be set to automatically select the pictures to be taken (as well as providing sensor data). In the present study the current sensor-based trigger algorithm showed little success at selecting the more memorable events. That is, the sensor-triggered picture capture strategy did not have an impact on participants’ memory as we measured it when compared to a strategy of capturing pictures at regular intervals and randomly sampling for review and testing. The question of whether a sensor-driven approach to input filtering is appropriate depends on the user’s goals and on the extent to which the measured physical environment is systematically related to those goals. Ongoing analyses of the rich data collected here have begun to uncover systematic relationships between sensor values and improved memory performance, and have also begun to demonstrate ways to devise more effective thresholds and selection algorithms (Finley, Brewer, & Benjamin, 2009).

The variability in memory performance that can be accounted for by physical characteristics of the environment almost certainly varies by the structure of peoples’ lives, the nature of their activities, their cognitive functioning, and their particular memory goals. Any of these may differ between individuals, and may particularly differ for memory rehabilitation patients versus the healthy young adults in the present study. Thus, adaptive calibration of sensor thresholds and triggering algorithms may be necessary on the individual user level in order to implement effective sensor-based strategies of picture capture.

SenseCam constitutes a new tool for autobiographical memory research that has different capabilities and different disadvantages from those previously available. For example, although SenseCam does provide a veridical visual record which should be useful for evaluating recall accuracy, there are several issues that will have to be resolved: (a) Script based responses. It is often difficult to differentiate between information in participants’ responses that was genuinely recalled versus information that was provided in the picture itself or inferred from the picture and participants’ general knowledge of the structure of their own lives (e.g., “I am walking to Spanish or German class. I know this because the tree angle is very dramatic and it wouldnt be at noon, which is when I’d normally leave this area.”). (b) Narrative summarization. It seems likely that participants often did not provide as much detail as they actually had in memory (e.g., “Dinner with friends”). (c) Non picture information. Many recall responses contain specific details that cannot easily be verified by the picture alone, such as: thoughts or emotions, the topic of a lecture or conversation, the identities of other people, the name of a building (e.g., “I was in design class being bored out of my mind at this point…”, “telling my boyfriend on aim [AOL Instant Messenger] that all the rum is gone on friday…”, “I am showing off bad paint brushes that no one washed off so they were useless. I am behind the front desk at [X] Hall…”).

Making maximum use of the picture information will require addressing these difficulties. For example the occurrence of script based responses could be studied by seeing how much the experimenters (acting as control participants) might themselves be able to describe the visual scene or infer what would be pictured next. The issue of narrative summarization could be addressed by more elaborate instructions or by adjustments in the scoring procedures. The occurrence of non picture information could be studied using additional experience sampling methods (e.g., Brewer, 1988), or using other information sources (e.g., course lecture notes). Additional research possibilities will continue to arise as additional sensor information becomes available (e.g., audio, location, physiological data).

In this paper we have focused on the use of an end-of-day review of SenseCam pictures to fortify autobiographical memory, and on their effectiveness as memory cues. However, as with any form of external memory, SenseCam pictures could be used for a variety of other purposes (cf. Bush, 1945), such as a tool for reminiscing about the past, for evaluating allocation of activities during a day, to complement human memory by revealing what may not have been noticed or encoded upon initial experience, for backup reference in case human memory fails, or even to supplant human memory for certain purposes. Proliferation of such technology should continue to transform not just research, but the way we live.