“Sound design is the main contributing factor to the mood and atmosphere of any film,” Charlie Battin wrote in a 2015 piece for the BFI Film Academy. “The visuals are what the viewer tends to mostly focus on and the sound subconsciously alters how the visuals are perceived.” While the connection between visuals and audio in film production (including television and advertising) is well-known, it is not always well-understood. Why do some songs just seem to work with certain visuals? Here, we dig into the psychology of pairing image and audio in order to better understand the core reasons certain audio tracks click with their paired visuals to produce a pitch-perfect experience.
Visuals and the Brain
Why are we so attracted to visuals and so good at recalling them? According to Dr. Haig Kouyoumdjian at Psychology Today, it’s partly because the human brain is primarily an image processor.
“The part of the brain used to process words is quite small in comparison to the part that processes visual images,” Dr. Kouyoumdjian writes. Because visuals are more concrete to the brain than words, they’re easier to remember, as well.
Our love of visuals is obvious in the digital world. Gonzalo Sanchez at The Brandery wrote a piece for Piktochart noting that “since early 2007, the Internet has seen a 9900 percent increase in the use of visualized information.”
All this highly visual content is deeply appealing to a species that uses about 50 percent of its brain to analyze and store 70 percent of what it learns — all through visual processing, Sanchez says.
Sanchez also explored how visuals elicit emotions. The visceral, emotional reactions that strong visuals can evoke are even quicker for our brains to process than emotion-neutral visuals — 13 milliseconds on average. Responding to sound is only slightly slower, at 146 milliseconds on average, according to Matt D’Angelo at Business News Daily.
By contrast, the world’s fastest speed reader, Howard Berg, has been clocked at comprehending 25,000 words per minute, or 24 milliseconds per word. (Most of us read about 100 times slower than that.) Visuals make us feel things strongly, and they do it more quickly than language processing. But what about sound?
Sound and the Brain
Humans have had a love affair with music for tens of thousands of years — perhaps even longer. Yet figuring out why we love music hasn’t been as easy as determining why we love visuals, according to Ben Mauk at LiveScience.
While visual processing is done in the brain’s visual cortex, which has a known location at the lower back of the skull, “researchers have yet to find a ‘music center’ in the brain,” Mauk says. Instead, “the tasks involved in processing and enjoying music are distributed across several brain areas.”
One reason, however, appears to be the emotional reactions we have to music. A 2001 study at McGill College, cited by Mauk, found that subjects listening to music showed neural activity in the same areas of the brain that respond to other euphoric stimuli like food, sex or drugs. In other words, our brains find music uniquely rewarding.
Sound, Sight and Sense: How Sound-Visual Combinations Affect Viewer Perspective
When sound combines with visuals in a short film or ad clip, it’s easy to predict that the brain’s response will incorporate both the sound and visual processing centers. But this isn’t all that happens.
“99 percent of our daily life depends on multisensory – also called multimodal – processing,” says Caltech psychology professor Shinsuke Shimojo, whose research team explored ways in which stimulating one sense, such as hearing, could cause the brain to produce feedback that appealed to a related sense, like vision.
The Complexity of Multimodal Processing
The researchers on Shimojo’s team found that sound and vision don’t have a 1:1 correlation; instead, they interact with each other in highly complex ways. One of them is the ability for one input to provoke the other, as Heather Murphy explored in a 2017 New York Times article on silent GIFs that nevertheless evoke a sense of sound in many viewers.
Research into multimodal processing has long focused on relatively simple combinations, like the sight of a bird and the sound of its chirp. However, researchers are beginning to explore more complex combinations, like movies and their soundtracks, says Northwestern University research associate Julia Mossbridge, because the complexity of sound and visual design in film more closely matches how we process multisensory inputs in real life.
Whether or not they have articulated it, artists have long known that sight and sound are intertwined. “Design and music, as anyone with even a passing interest in either knows, are inextricably linked,” Emily Gosling wrote in a 2016 article in Eye on Design, noting how many great musicians have come from art schools and how many great graphic designers got their starts in the music world.
How What We Hear Influences What We Think We See
In fact, good sound design in a film often doesn’t even register for us consciously. The best sound effects are so subtle that we don’t notice them.
Swedish researchers Johnny Wingstedt, Sture Brandstrcm and Jan Berg agree. In a paper titled “Narrative Music, Visuals and Meaning in Film,” they noted that “what we (think we) see is to a large degree determined by what we hear.” The researchers analyzed a scene from the 1975 film Jaws, in which the eponymous shark takes some bait attached to a car tire float, pulling so hard that it breaks the pier, knocking a man into the water, and he barely escapes to shore. The music, the researchers noted, “is based on the (now) famous Jaws leitmotif” of the alternating minor-second interval, which is linked to the shark by continuous repetition of the motif every time the shark appears — in other words, by combining the visual with experiences of the same sound.
By studying the ways in which this well-known two-note interval appears when the shark does, accompanied by other musical elements like dissonant chords and musical resolutions as characters reach safety, the researchers demonstrated how the now-famous notes come not only to signal “here is the shark” but to cause a rising sense of discomfort and danger in the audience.
The Converse is True, Too
The sight-sound relationship goes both ways: What we think we hear is also influenced by what we see. In a senior thesis project at Haverford College, Brittany Ebendorf examined how visuals could affect the perception of music.
Ebendorf used videos that had been previously rated either “positive” or “negative,” pairing them with “unfamiliar and affectively ambiguous tones” to see whether the videos would influence whether the subjects rated the music as either “positive” or “negative.”
It did. Ebendorf found a clear correlation between the way each video had been rated and the way the music accompanying it was rated when the two were experienced together. Not only does sound affect what we see, but visuals affect what we hear, too.
A Future for Sound and Visual
As creatives continue to push the boundaries of both visual and sound-based genres, new ways of linking sound with visuals emerge.
A blog post at Ableton explores the connections between music and visuals that have become possible through the use of digital technologies. “While a few musicians are also visual artists and create their own visual element, the vast majority of A/V performances are collaborations,” the Ableton team writes. “Though working with an artist from another discipline can be challenging, it has the potential to augment and even transform a performance in many ways.”
Popular digital forms include direct visualizations of sound waveforms, immersive environments with “mesmerizing visual patterns,” narrative worldbuilding, live collaborations and even setting sound to actual or simulated natural events, such as the northern lights.
In 2016, Alyssa Buffenstein at ArtNet explored the work of 12 artists whose work, while focused on sound, frequently contains a performance element that is both visual and tactile — further pushing the boundaries of multisensory art. Meanwhile, sound may become the next step forward for branding, according to Will Burns at Forbes. The Audio Branding Academy describes the use of sound to build brands, or audio branding, as “building solidly a brand sound that represents the identity and values of a brand in a distinctive manner.” By doing so, brands can further tap into the brain’s various abilities to remember, associate and build positive memories across sensory channels.
Given the human brain’s fascination with both sound and visuals and the creative depths to which the two can be combined, the future for sound-image combinations is undoubtedly a bright one.
Images by: Federico Beccari, Robert Collins, Lopez Rob
