Hear with Your Eyes: The McGurk Effect

In Hack #59 of their book Mind Hacks, Tom Stafford and Matt Webb use a classic illusion known as the McGurk Effect to show how our senses combine to completely change our ultimate experience of perception. Knowing what’s going to happen doesn’t even keep you from experiencing the illusion. It just weakens the effect a bit.

Let’s try it out. Watch my lips as I make a few sounds:

[[McGurk Effect clip]]

Now, listen to just the audio portion from that clip:

[audio clip]

Besides perhaps being a little bit creepy, when you watched the video, I should have appeared to be saying “da da,” but when you listen to the audio without the video, it’s clear I’m saying “ba ba.”

This illusion can’t happen in real life. Like McGurk, I made it by splicing the audio of me saying “ba ba” over a video of me making a different sound: “ga ga.” When you’re not watching the video, you hear what I’m actually saying. But when you see my lips moving, the two bits of information clash. The position of a person’s mouth is key in telling what sound someone is making, especially for distinguishing between speech sounds, called phonemes, like “ba,” “ga,” “pa,” and “da,” which are all made by popping air out.

Beyond a neat mind trick, the McGurk Effect has some practical uses as well. In Hack #57 of his book Digital Video Hacks, Josh Paul shows how to create the effect yourself, as I’ve done in this video, and use it in your own movies to fool your audience.

Here’s an example of an applying the effect. Watch my lips as I mouth a couple words, while dubbing over a completely different phrase:

[[olive juice effect]]

What did I say? Could you tell that I was saying “olive juice” in the video, before editing it with a different audio track? Does it even matter?

This type of editing occurs frequently when feature films are shown shown on TV. Some words obviously are frowned upon by the FCC, and when movie houses edit them out, they generally try to fool you into believing the audio is original to the movie, or at least keep you from noticing the difference too much.

A common example is when someone shouts “Forget you!” onscreen. Though you know a different phrase was used in the original, the illusion is convincing enough to keep you from being distracted by the dubbing. But when the illusion is used with words that don’t quite fit the context of the movie, the effect tends to break down. For example, when I hear John Goodman shout, “This is what happens when you meet a stranger in the Alps,” I know the attempt has failed miserably. – Link to video download.

Resources:

Mind Hacks, by Tom Stafford and Matt Webb
• Digital Video Hacks, by Josh Paul
• Arnt Masso’s McGurk Effect Video
• Hearing with Your Eyes collection of McGurk Effect movies