Do film soundtracks contain nonlinear analogues to influence emotion?

Abstract

A variety of vertebrates produce nonlinear vocalizations when they are under duress. By their very nature, vocalizations containing nonlinearities may sound harsh and are somewhat unpredictable; observations that are consistent with them being particularly evocative to those hearing them. We tested the hypothesis that humans capitalize on this seemingly widespread vertebrate response by creating nonlinear analogues in film soundtracks to evoke particular emotions. We used lists of highly regarded films to generate a set of highly ranked action/adventure, dramatic, horror and war films. We then scored the presence of a variety of nonlinear analogues in these film soundtracks. Dramatic films suppressed noise of all types, contained more abrupt frequency transitions and musical sidebands, and fewer noisy screams than expected. Horror films suppressed abrupt frequency transitions and musical sidebands, but had more non-musical sidebands, and noisy screams than expected. Adventure films had more male screams than expected. Together, our results suggest that film-makers manipulate sounds to create nonlinear analogues in order to manipulate our emotional responses.

1. Introduction

A system can be described as nonlinear when the output from it is not proportional to the input into it. Many acoustic systems are linear within certain parameters, and nonlinear beyond them. For instance, if you turn up your stereo volume too much, at some point, you will experience a loss in fidelity. A variety of animals, including humans, produce what in the bioacoustic literature are referred to as vocalizations with nonlinear attributes (Wilden et al. 1998; Fitch et al. 2002). Such nonlinearities include: noise and deterministic chaos, sidebands and subharmonics, and abrupt amplitude and frequency transitions. Nonlinearities are commonly produced when animals are under duress, such as the fear screams produced when animals are attacked by predators (Gouzoules et al. 1984; Held et al. 2006; Blumstein et al. 2008). Indeed, the predator-specific alarm calls of meerkats (Suricata suricatta) emit get noisier as the urgency of a situation increases (Manser et al. 2002). Other vocalizations, such as baby cries, vary from those with and those without a variety of nonlinear acoustic attributes that seem to be produced as a function of arousal (e.g. Facchini et al. 2005).

While nonlinear vocal attributes may be an unavoidable by-product of asymmetries in an individual's vocal apparatus (e.g. Herzel & Wendler 1991), there are other hypothesized adaptive functions (Fitch et al. 2002). One such adaptive hypothesis is that they are designed to capture the attention of perceivers (Fitch & Hauser 1995; Fitch et al. 2002). Because deterministic chaos and noise sound ‘harsh’, and because abrupt frequency transitions may be unpredictable, their presence may make sounds containing them particularly evocative and difficult to habituate to. Thus, baby cries and marmot vocalizations with nonlinear attributes are more evocative than those without them (e.g. Green et al. 1987; Blumstein & Récapet 2009).

If nonlinearities are used by humans, and other vertebrates, to capture a receiver's attention, we might expect them to be also used by film score composers and audio engineers to manipulate the emotions of those watching a film. Previous work has focused on the relationship between emotion and the temporal and frequency characteristics of music and film soundtracks (e.g. Bolivar et al. 1994; Huron et al. 2006), and we know that the dramatic sad music that makes us cry in a film soundtrack sounds very different from the music in an action/adventure film with a throbbing low-frequency beat that keeps us on the edge of our seats. But is it simulated nonlinear sounds that make these scenes especially evocative?

We formally tested the hypothesis that humans capitalize on this seemingly widespread vertebrate response when they create nonlinear analogues in film soundtracks to evoke particular emotions. We focused on adventure, dramatic, horror and war films because: (i) they represented the genres with the most Internet-based polls; and (ii) if nonlinearities are manipulated in soundtracks, these film's arousal profiles should be distinguishable. Specifically, if nonlinearities function to increase activity, they should be used more in adventure, war and horror films, and less present in dramatic films. Alternatively, if nonlinearities are used more generally to manipulate emotions, they may be used in all types of films, but the type of nonlinear attribute used might vary.

2. Material and methods

We used Internet film sites (boston.com, rottentomatoes.com, ew.com, reel.com, afi.com, virginmedia.com, imdb.com, filmcrave.com, about.com, the-top-tens.com, wanderlist.com, movies.ign.com, channel4.com, moviefone.com) to obtain broadly based, public polling lists of ‘best films’ by genre (adventure, dramatic (often referred to as ‘sad’ on the web poll sites), horror, war) that we then further consolidated. Because we relied on the popular vote, some films might better fit in different categories (e.g. Lawrence of Arabia might be more accurately adventure, Aliens might be more accurately under horror). From 102 films (24 adventure, 35 dramatic, 24 horror, 19 war; electronic supplementary material), we selected an iconographic scene that epitomized the film's genre, picked a hard film cut (i.e. edit) so as to mark the location, and then extracted exactly 30 s from the cut. The soundtrack was extracted using Handbrake 0.9.3 (www.handbrake.fr) at a sample rate of 48 kHz and a bit rate of 160 Kbps. We then made spectrograms (2000-point fast Fourier transform; brightness and contrast set to 60) of each sound clip using Raven Pro (v. 1.3; Cornell Lab of Ornithology, Ithaca, NY, USA), and PRAAT (v. 5.1.11; www.fon.hum.uva.nl/praat/), in which we focused on the top 40 dB of dynamic range. Because of the rich tapestry of sound contained in film soundtracks, automatic feature extraction was impossible (electronic supplementary material, figure S1). Thus, we worked together to define quantifiable criteria and then applied them to soundtracks until they were scored consistently. Ultimately, we created a visual reference that was used when scoring soundtracks (electronic supplementary material) and zoomed in and out of sounds along both temporal and frequency axes to identify acoustic traits.

We scored the presence or absence of the following nonlinear analogues, so-called, because sound engineers manipulate a variety of systems (e.g. vocalizations, diegetic sound, Foley and music) to created a soundtrack. Noise was scored as present when there were no defined spectral bands, but rather sound was present in many frequencies. We examined diegetic noise, musical noise and noisy sound effects. Abrupt amplitude fluctuations were scored as present when the amplitude intensity changed by more than 10 per cent of the clip's total mean intensity in less then 500 ms. Abrupt frequency fluctuations were scored as present when visible tonal frequency bands were seen to abruptly shift leading to a change in the fundamental frequency. Musical sidebands were scored by closely following musical frequency contours, while non-musical sidebands were scored when we saw bands of non-harmonic sound surrounded by a frequency band. Focusing on screams, we scored male and female screams, when present, as noisy or not. Tonal screams had distinctive frequency bands while noisy screams were not tonal.

We used χ²-tests to see if the proportion of films within a particular category had these nonlinear analogues present more or less likely than expected by chance. We present the p-value for the entire 2 × 4 contingency analysis and note those categories for which the cell χ²-value was significantly (p < 0.05) different than expected.

3. Results

Dramatic films had a lower frequency of noise of all sorts, and fewer amplitude fluctuations (figure 1). Dramatic films had more abrupt frequency shifts both up and down, and more musical sidebands (perhaps reflecting that music is in the foreground in dramatic films), but fewer non-musical sidebands. Dramatic films had fewer screams than would be expected. Horror films had fewer abrupt frequency shifts and musical sidebands. They had more non-musical sidebands than would be expected. Horror films had more noisy female screams. Adventure films had more noisy male screams than expected, and war films had more amplitude fluctuations than expected.

Figure 1.

The presence (black) and absence (white) of specific acoustic attributes measured from film soundtracks as a function of film type (adventure, horror, sad, war). (a) Noise diegetic, p = 0.019; (b) noise musical, p = 0.003; (c) noise sound effects, p = 0.003; (d) abrupt amplitude fluctuation, p < 0.001; (e) abrupt frequency change shift down, p < 0.001; (f) abrupt frequency change shift up, p = 0.002; (g) sidebands musical, p = 0.002; (h) sidebands non-musical, p = 0.002; (i) noisy screams female, p = 0.002; (j) noisy screams male, p = 0.003. p-values are from a χ²-test, asterisks highlight those individual categories that are larger or smaller than expected by chance.

4. Discussion

Film soundtracks may contain sounds that, if produced naturally, would be classified as nonlinear vocal attributes. The use of these simulated nonlinearities is not random, but rather appears to be specifically used to enhance the emotional impact of scenes.

Film score composers have traditionally used knowledge of the natural, nonlinear possibilities of western orchestral, musical instruments, to modify harmonic spectrum and perceived roughness (e.g. the overblowing of the brass and wind instruments, the metallic rasp of the stopping of the French horns (Cuivré) or directing the string players' bow strength and location). An orchestral percussion section contains many commonly used instruments of an inharmonic noise-like nature (e.g. various gongs and cymbals), while contemporary popular music percussion practice is capable of unnaturally consistent, high amplitude levels and frequency of inharmonic, sudden onsets. The feedback loop between an electric guitar and its amplification is an oft-used example of a semi-controllable nonlinear effect (generating sudden pitch, amplitude and harmonic changes), along with the overdriven amplifier's electronic based, nonlinear signal distortion (roughness). Both the percussion and guitar-based sounds are apparent in the fear/threat vocalization-based music samples used in Snowdon & Teie (2010) and suggest that these attributes are generally arousing.

There are musical composition techniques that mimic what would naturally be called a nonlinear acoustic attribute. These include frequency-based effects, such as the intentional sidebands (both upper and lower) created by the use of harmonic dissonance, trills, vibrato and sudden pitch change, and amplitude-based effects, such as tremolo string bowing, flutter-tonguing wind instruments, or sudden amplitude change (e.g. the use of the dynamic change modifier, subito). Broadband noise generating techniques are found in the work of the twentieth century composer Krysztof Penderecki (Schwinger 1989), as well as in later sub-genera of rock music (Hegarty 2007). The use of Penderecki's music in the films The Exorcist (1973, Director Friedkin) and The Shining (1980, Director Kubrick) would inspire the use of noise techniques as a style marker of horror genre films.

Soundtracks contain more than simply music and sound engineers can create sounds that would be impossible for an individual to produce. King Kong (1933, Director Cooper) saw the first use of recorded, animal vocalizations, pitch, timbre and temporally changed through manipulation of the playback medium, as a naturally sourced, nonlinear base material for the creation of affective sound (Boone 1933). This is still the practice for many prehistoric, alien or otherwise monstrous cinematic characters (Jackson 2010). That a natural source is often used may be because the complexity of affective nonlinear sounds may be difficult to synthesize. A notable exception to the practice is Hitchcock's 1963 film, The Birds, in which a horrifying avian language was performed solely on an early electronic instrument, the Trautonium (Wierzbicki 2008), displaying a rich set of nonlinear characteristics.

In summary, we found non-random use of nonlinear analogues in film soundtracks. From this, we infer that specific types of nonlinear analogues are used to elicit fearful responses (noise in horror films), while others are used to elicit more dramatic emotional responses (abrupt frequency shifts). Nonlinearities thus seem to be broadly evocative in vertebrates and their analogues can be used to influence human emotions.