Audiovisual Speech Perception

The Use of Visual Cues In the Perception of Non-Native Consonant Contrasts

by Valerie Hazan

published as: Hazan, V., Sennema, A., Faulkner, A., Ortega-Llebaria, M., Iba, M., Chung, H. (2006). The use of visual cues in the perception of non-native consonant contrasts. Journal of the Acoustical Society of America 119(3), 1740-175

This study assessed the extent to which second-language learners are sensitive to phonetic information contained in... more This study assessed the extent to which second-language learners are sensitive to phonetic information contained in visual cues when identifying a non-native phonemic contrast. In experiment 1, Spanish and Japanese learners of English were tested on their perception of a labial∕labiodental consonant contrast in audio (A), visual (V), and audio-visual (AV) modalities. Spanish students showed better performance overall, and much greater sensitivity to visual cues than Japanese students. Both learner groups achieved higher scores in the AV than in the A test condition, thus showing evidence of audio-visual benefit. Experiment 2 examined the perception of the less visually-salient ∕l∕-∕r∕ contrast in Japanese and Korean learners of English. Korean learners obtained much higher scores in auditory and audio-visual conditions than in the visual condition, while Japanese learners generally performed poorly in both modalities. Neither group showed evidence of audio-visual benefit. These results show the impact of the language background of the learner and visual salience of the contrast on the use of visual cues for a non-native contrast. Significant correlations between scores in the auditory and visual conditions suggest that increasing auditory proficiency in identifying a non-native contrast is linked with an increasing proficiency in using visual cues to the contrast.

Atypical audiovisual speech integration in infants at risk for autism

by Przemyslaw Tomalski

Guidau J., Tomalski P., Kushnerenko E., Ribeiro H., Davies K., Charman T., Elsabbagh M., Johnson M.H., and the BASIS Team, PLOS One, in press

The language difficulties often seen in individuals with autism might stem from an inability to integrate audiovisual... more The language difficulties often seen in individuals with autism might stem from an inability to integrate audiovisual information, a skill important for language development. We investigated whether 9-month-old siblings of older children with autism, who are at an increased risk of developing autism, are able to integrate audiovisual speech cues. We used an eye-tracker to record where infants looked when shown a screen displaying two faces of the same model, where one face is articulating /ba/ and the other /ga/, with one face congruent with the syllable sound being presented simultaneously, the other face incongruent. This method was successful in showing that infants at low risk can integrate audiovisual speech: they looked for the same amount of time at the mouths in both the fusible visual /ga/ - audio /ba/ and the congruent visual /ba/ - audio /ba/ displays, indicating that the auditory and visual streams fuse into a McGurk-type of syllabic percept in the incongruent condition. It also showed that low risk infants could perceive a mismatch between auditory and visual cues: they looked longer at the mouth in the mismatched, non-fusible visual /ba/ - audio /ga/ display compared with the congruent visual /ga/ - audio /ga/ display, demonstrating that they perceive an uncommon, and therefore interesting, speech-like percept when looking at the incongruent mouth (2-way repeated ANOVA: congruent/incongruent displays: F(1,19) = 4.450, p = 0.048, displays x fusion/mismatch condition interaction: F(1,19) = 9.382, p = 0.006). In contrast, the looking behaviour of infants at high-risk did not differ according to the type of display, suggesting difficulties in matching auditory and visual information (2-way repeated ANOVA: displays: F(1,32) = 1.501, p = 0.229, displays x condition interaction: F(1,32) = 0.607, p = 0.442). In some cases this reduced ability might lead to the poor communication skills characteristic of autism.

Neural Correlates of Interindividual Differences in Children's Audiovisual Speech Perception

by Eswen Fava

Children use information from both the auditory and visual modalities to aid in understanding speech. A dramatic... more Children use information from both the auditory and visual modalities to aid in understanding speech. A dramatic illustration of this multisensory integration is the McGurk effect, an illusion in which an auditory syllable is perceived differently when it is paired with an incongruent mouth movement. However, there are significant interindividual differences in McGurk perception: some children never perceive the illusion, while others always do. Because converging evidence suggests that the posterior superior temporal sulcus (STS) is a critical site for multisensory integration, we hypothesized that activity within the STS would predict susceptibility to the McGurk effect. To test this idea, we used blood-oxygen level dependent functional magnetic resonance imaging (BOLD fMRI) in seventeen children aged 6 to 12 years to measure brain responses to three audiovisual stimulus categories: McGurk incongruent, non-McGurk incongruent and congruent syllables. Two separate analysis approaches, one using independent functional localizers and another using whole-brain voxel-based regression, showed differences in the left STS between perceivers and non-perceivers. The STS of McGurk perceivers responded significantly more than non-perceivers to McGurk syllables, but not to other stimuli, and perceivers' hemodynamic responses in the STS were significantly prolonged. In addition to the STS, weaker differences between perceivers and non-perceivers were observed in the FFA and extrastriate visual cortex. These results suggest that the STS is an important source of interindividual variability in children's audiovisual speech perception.

Processing of audiovisual stimuli in aphasic and non-brain-damaged listeners

by Dörte Hessler

together with Roel Jonkers and Roelien Bastiaanse, currently in press in Aphasiology

Background
During speech perception not only auditory but also visual information (seen speech) is processed.... more Background
During speech perception not only auditory but also visual information (seen speech) is processed. This was shown by, for example, McGurk and MacDonald (1976). In their study participants watched videos showing a speaker articulating the syllable /ka/, while the sound of /pa/ was dubbed onto the video. The most commonly reported perception was neither of the two syllables, but a fusion of both (/ta/). This effect of audiovisual integration has been called the “McGurk effect”. Klitsch (2008) showed that the McGurk effect was also found for participants with aphasia and its strength was comparable to an age-matched control group. However her task was an offline measure and therefore provided limited information about the processing itself.

Aims
In this study reaction time data will be analysed in addition to answer types. By this we aim to find out whether there are qualitative differences between healthy and aphasic processing, shown by different patterns in the reaction times.

Methods & Procedures
Three aphasic participants and a group of fourteen non-brain–damaged control participants took part in a nonword identification task in which they were presented with a speaker pronouncing a syllable. Afterwards they had to choose the matching syllable fromthree written options. This task was carried out in four conditions: “auditory only”, “visual only” (articulatory movements), “audiovisual”, and “McGurk” (incongruent auditory and audiovisual information). Along with the answer types, the reaction times were also recorded for all participants.

Outcomes & Results
The aphasic participants made more errors and were slower than the non-brain-damaged control participants across conditions. Within the McGurk condition we found different patterns for the non-brain-damaged and the aphasic participants: reaction times were increased for the control group whenever a McGurk-type answer was chosen (compared with answers representing the auditory or visual part of the stimulus), while there were no influences of the chosen answer type on the reaction times for the aphasic participants.

Conclusions

Processing of audiovisual information was not only slower in three aphasic participants but also differed qualitatively from the non-brain-damaged control group. For the control participants reaction times differed depending on what answer type

The influence of phonetic dimensions on aphasic speech perception

by Dörte Hessler

together with Roel Jonkers and Roelien Bastiaanse, published 2010 in 'Clinical Linguistics and Phonetics 24, pp 980-996'

Individuals with aphasia have more problems detecting small differences between speech sounds than larger ones. This... more Individuals with aphasia have more problems detecting small differences between speech sounds than larger ones. This paper reports how phonemic processing is impaired and how this is influenced by speechreading. A non-word discrimination task was carried out with ‘audiovisual’, ‘auditory only’ and ‘visual only’ stimulus display. Subjects had to decide whether two presented stimuli were the same. Six aphasic subjects with speech sound processing difficulties and 14 non-brain-damaged control subjects participated in this study. It was found that the aphasic subjects have difficulties in discriminating pairs of non-words, which are more profound for small differences. Differences in ‘voicing’ were least often detected and therefore seem most difficult to perceive. This implies different processing of the phonetic dimensions in speech sound perception. Performance improved when speechreading was possible. As this improvement is not based on differences in place of articulation only, theories of audiovisual processing need to be revised.

Development of auditory-visual speech perception in English-speaking children: The role of language-specific factors

by Doğu Erdener

It has been found that auditory-visual speech
perception differs over languages, particularly
between... more It has been found that auditory-visual speech
perception differs over languages, particularly
between English and Japanese speakers. This
difference emerges due to increased use of visual
information by English speakers between 6 and 8
years [14]. This study investigates the linguistic
factors that may cause changes in auditory-visual
speech perception development. Children aged
between 5 and 8 years were given tests of reading,
articulation, language-specific speech perception
and auditory-visual speech perception. The results
show that only language-specific speech perception
predicts auditory-visual speech perception. Possible
reasons for this relationship are discussed.

The Development of Auditory-Visual Speech Perception across Languages and Age

by Doğu Erdener

how it develops. We know that young infants perceive speech auditory-visually by the fact that they perceive the... more how it develops. We know that young infants perceive speech auditory-visually by the fact that they perceive the auditory-visual illusion known as the McGurk effect; that visual information use increases over age in English-language children; and that Japanese-language adults use less visual information than do their English-language counterparts. Here we complete the developmental scene and probe the processes involved. In Experiment, with 6-, 8-, and 11-year-old and adult Japanese- and English-language participants tested on a McGurk task, while 6-year-olds from both language groups were equivalently influenced by visual speech information, there was a significant jump in auditory-visual speech perception between 6 and 8 years in English- but not Japanese-language participants. To in-vestigate this further, in Experiment 2 we gave English-speaking 5-, 6-, 7- and 8-year-olds and adults a McGurk ef-fect task as well as a language-specific speech perception (LSPP) test with native- and non-native speech sounds, and reading and articulation tests. For children, but not adults, visual-only speech perception (lipreading) ability and LSSP predicted McGurk performance – children with good auditory-visual speech perception tended to be those who focussed more on native than non-native speech sounds. In Experiment 3, with 3- and 4-year-olds tested for McGurk effect, LSSP, receptive vocabulary, and cognitive skill, regression analyses showed that auditory-only speech percep-tion and cognitive skill, but not LSSP, predicted auditory-visual speech performance. Together the results show that there is an increase in auditory-visual speech perception between 6 and 8 years in English- but not Japanese-language children, and in English-language children this is related to language specific speech perception processes specifically around that age (5, 6, 7, 8 years) and not before (3, 4 years) or after (adults). It is suggested that LSSP is most vari-able and most predictive of visual influence in speech perception in the presence of significant linguistic challenges, such as those at the onset of reading instruction.

The effect of auditory, visual and orthographic information on second language acquisition

by Doğu Erdener

Visual information from the lips and face is an integral part of speech perception. In addition, orthography can play... more Visual information from the lips and face is an integral part of speech perception. In addition, orthography can play a role in disambiguating the speech signal in foreign/second language (L2) perception and production. The current study investigates the effect of auditory and visual speech information and orthographic depth, the degree to which a language is transparent (high phoneme-grapheme correspondence), or opaque (low phoneme-grapheme correspondence) on L2 acquisition. Speakers of Turkish and Australian English (transparent and opaque orthographies, respectively) were tested for their production of legal non-words in Spanish and Irish (transparent and opaque orthographies, respectively) . Transparent orthographic input (Spanish) enhanced pronunciation in L2, and orthographic reproduction. Native speaker ratings of the participants’ productions also revealed that orthographic input improves accent. Overall results confirm previous findings that visual information enhances speech perception and production, and extend previous results to show the facilitative effects of orthographic input in L2 acquisition under certain conditions

Perception of the auditory-visual illusion in speech perception by children with phonological disorders

by Doğu Erdener

An example of the auditory‐visual illusion in speech perception, first described by McGurk and MacDonald, is the... more An example of the auditory‐visual illusion in speech perception, first described by McGurk and MacDonald, is the perception of [ta] when listeners hear [pa] in synchrony with the lip movements for [ka]. One account of the illusion is that lip‐read and heard speech are combined in an articulatory code since people who mispronounce words respond differently from controls on lip‐reading tasks. A same‐different judgment task assessing perception of the illusion showed no difference in performance between controls and children with speech difficulties. Another experiment compared children with delayed and disordered speech on perception of the illusion. While neither group perceived many illusions, a significant interaction indicated that children with disordered phonology were strongly biased to the auditory component while the delayed group's response was more evenly split between the auditory and visual components of the illusion. These findings suggest that phonological processing, rather than articulation, supports lip‐reading ability.




Read More: http://informahealthcare.com/doi/abs/10.1080/02699200701660100

The Role of Audiovisual Speech and Orthographic Information in Nonnative Speech Production

by Doğu Erdener

Visual information from the face is an integral part of
speech perception. Additionally, orthography can play... more Visual information from the face is an integral part of
speech perception. Additionally, orthography can play a
role in disambiguating the speech signal in nonnative
speech. This study investigates the effect of audiovisual
speech information and orthography on nonnative speech.
Particularly, orthographic depth is of interest. Turkish
(transparent) and Australian English (opaque) speakers were tested for their production of nonwords in Spanish
(transparent) and Irish (opaque). We found that transparent
orthography enhanced pronunciation and orthographic
responses. Results confirm previous findings that
visual information enhances speech production and
extend them to show the facilitative effects of orthography
under certain conditions. Implications are discussed in
relation to audiovisual speech perception and orthographic
processing and practical considerations such as
second language instruction.

Development of Auditory-Visual Speech Perception in Young Children

by Doğu Erdener

Lending a Helping Hand to Hearing: Another Motor Theory of Speech Perception

by Howard Nusbaum