Reading Upside-down Lips

Visual speech perception involves perceiving language from faces. This fact is interesting because the language and face perception functions are usually considered to be very separate. How much of visual speech perception involves the language function and how much involves the face function?

Many scientists have thought that visual speech has little to do with face perception. After all, we look at different parts of the face for visual speech perception (mostly the mouth) and face recognition (eyes, nose, mouth, configuration of these features, and face shape). There is also evidence that we use different parts of our brains for visual speech perception and face recognition. However, some new findings suggest that the face function might be involved with visual speech perception in an interesting way. . .

The Margaret Thatcher Effect:The demonstration above is called the "Margaret Thatcher Effect" (named after the former Prime Minister of England whose face is depicted). Look at the face on the left. The eyes and mouth of Margaret Thatcher have been inverted relative to the rest of her face. It's interesting that these distortions are much easier to notice when the face is upright than when it is upside-down.

What the effect means: The Margaret Thatcher Effect demonstrates that seeing an upright face makes it easier to notice distortions in the face image. The effect adds to other findings that show we are especially sensitive to the information contained in upright faces. In fact, it is much easier to recognize an upright face than an inverted face. Furthermore, many scientists believe that this heightened sensitivity to the upright image is specific to face perception. But what about visual speech perception? Please read on. . .

The 'McThatcher Effect': The demonstration below tests visual speech perception by combining the McGurk Effect and Margaret Thatcher Effect (hence, the 'McThatcher Effect'). In the video below, you will see five audiovisual syllables produced by these face (and lip) images:

The video shows these images in order from left to right. Watch the mouth closely, but concentrate on what you're hearing. The movie will repeat itself until it is stopped. You can watch it as many times as you need to be sure of the syllables you hear. After you feel certain of what you perceive, stop the movie and continue reading the text below.
PLEASE START THE MOVIE NOW.

If you're like most people, what you hear depends on which face (or lip) image you see. Here is what most people report:

'Va' 'Va' 'Va' 'Ba' 'Va'

If you would like, you can now play the movie over and see if you perceive VA from all of the images except for the fourth.

How the stimuli were made: All five of these stimuli were made by dubbing an auditory BA syllable onto a visual syllable VA According to the McGurk effect, most people report perceiving VA with this syllable combination. However, these images were also manipulate in other ways. As you can see, some of the images show an inverted face. If you look closely, you may also notice that in some images, the mouth is inverted relative to the face. Finally, the last image is of an inverted mouth without a face. These manipulations were accomplished by using a computer to digitally isolate and invert the mouth and face. To see a picture of the equipment used to create these stimuli, CLICK HERE.

What the effect means: For most people, all but the fourth image produces a McGurk effect. What is special about the fourth image? This image is composed of an upright face with an inverted mouth. It is much like the Margaret Thatcher effect image. As discussed above, the Margaret Thatcher effect demonstrates that seeing an upright face increases our sensitivity to image distortions. Similarly, the visual speech 'McThatcher effect' shows that an upright face increases our sensitivity to articulatory distortions. Specifically, the upright face enhances our sensitivity to the upside-down mouth so that it can no longer induce the McGurk effect. Notice that the upside-down mouth can still produce the McGurk effect when it is shown in the context of an inverted face (Image #2), or when it is shown without any face context at all (Image #5). The McGurk effect disappears only when the inverted mouth is shown in the upright face.

So, upright face information seems to be important for both face perception and visual speech perception. It heightens sensitivity for both functions. This could mean that the two functions are more closely related than was originally thought.