Hearing with your eyes: The McGurk Effect

17. Hearing with your eyes:
The McGurk Effect (with videos)

     How much do the visual cues (what we see) that we get from the face of a speaker influence what we hear? Quite a bit, in some cases. Click on this demonstration to find out. Watch and listen as the actress pronounces the syllables. What sounds is she saying? After watching and listening a few times, listen to the same video with your eyes closed. Is there any difference in what you believe she is really saying? If the direct link doesn't work, copy it and paste it into Windows Media Player (File - Open URL).

      Video I (YouTube):

http://www.youtube.com/watch?v=qc8MfDb9h-k&feature=related

      Video II: Patricia K. Kuhl, Ph.D., University of Washington

http://depts.washington.edu/ilabsts/McGurk.wvx

Source page: http://ilabs.washington.edu/kuhl/research.html

     Now you know you can't always believe your ears, especially when your eyes are allowed to contribute their input on the matter. This is due to visual dominance when conflicting information comes in from different senses.

      The McGurk effect was first noticed by Harry McGurk, a senior developmental psychologist at the University of Surrey in England, and his research assistant John MacDonald, and reported in a 1976 paper entitled "Hearing Lips and Seeing Voices" in the journal Nature. (source)

      Here are a few more demonstrations of the McGurk Effect for you to choose from and wonder over:

      Video III:

http://www.youtube.com/watch?v=rIWrnJH2jAY&NR=1

      Video IV (with text):

http://www.faculty.ucr.edu/~rosenblu/VSMcGurk.html

      Video V: YouTube

http://www.youtube.com/watch?v=T4fUi0eG1X4

      Video VI (with a bearded male face):

http://www.media.uio.no/personer/arntm/McGurk_english.html

      Video VII (with a female Japanese face):

http://www.brl.ntt.co.jp/IllusionForum/basics/auditory/mcgurk-e.html

     Here's a short page with more information on the McGurk effect:

http://scicom.ucsc.edu/SciNotes/9601/Speech/McGurk.html

     I personally have experienced a variation of the McGurk effect when transcribing samples of "Taiwan English". Informants had recorded series of similar-sounding (to a Taiwan learner of English) words, like bet, bed, bat, bad. I found that if I read the spelling of the words while I listened to the recordings of them, I had one judgment of the vowels I heard; I was usually more likely to judge the reading as "correct". But if I closed my eyes while listening, I often "heard" different vowels. In many cases, all four words turned out to sound exactly alike when listened to "blindfolded" – e.g. all of them might sound like bat! This again recalls a related phenomenon involving the processing of conflicting sensory input, the Stroop effect.

     A related question: Is TV and movie dubbing 'bad' for us, or does it hurt our 'McGurkian' sensitivities to how visual and auditory cues should agree? Read about the experiences and views of someone who went from a dubbing country (Germany) to a non-dubbing one (Britain), and who proposes a ban on dubbing! (Skim this only if you have time.)

http://people.cryst.bbk.ac.uk/~ubcge18/mcgurkdc.html

    Although Germans generally understand and speak English very well, most of their TV shows and movies are dubbed from the original language into German. You certainly have experience watching an English language movie or TV show dubbed into Mandarin, and feeling the mismatch between lip movements and the sounds you're hearing, not to mention bad translations. (Bad lip synchronization of actors trying to mouth the lines of a recording in their own language can be irritating in a similar way.) Some of you may choose to switch to subtitles and listen to the English, if your TV allows you to do so. Here's a short scene from the US sitcom "Married with Children" dubbed into German. How much do you feel the mismatch between sound and lip movements in this clip?

Married with Children clip with German dubbing (source)

    Finally, for your amusement, we conclude with a short video (don't miss this!!!) that's making the rounds on the Internet. Another corollary of the McGurk effect is that we can make anybody appear to be saying – or singing – anything we want them to, if we match the words with carefully selected bits from video footage. How convincing do you find this?

     We have been learning how visual input can take precedence over auditory, how we often 'hear with our eyes'. But you know from your experience of hearing stories told or read to you that you can also 'see with your ears' – auditory language can 'paint pictures' in your mind. But can sound input give us visual information without the use of language? Cutting-edge research scientist and inventor Peter Meijer of Philips Electronics, in Eindhoven in the Netherlands, is betting it can. Peter is developing a system intended to help the blind experience the visual world through soundscapes. Can this work, and if so, how, and how well? Find out more on the next page!

Next: Seeing with your ears: Peter Meijer's vOICe I

on to next page back index I index II home