Audiovisual Information Processing by Monolinguals and Bilinguals: Effects of Intralingual and Interlingual Subtitles Dominique Bairstow University of Montpellier III (France) Jean-Marc Lavaur University of Montpellier III (France) Abstract This contribution explores the effects of language fluency and different types of subtitles (intralingual and interlingual) on film comprehension. In a first experiment, we examine the role of interlingual subtitles in two experimental conditions (with and without French subtitles). In this experiment, comprehension is assessed by a test just after viewing the sequence (images, dialogues and understanding of the situation). The results show facilitating effects of subtitling for the monolinguals and inhibitory effects for bilinguals. In a second experiment, the participants are divided into three groups depending on their knowledge of the oral language of the film (English). Subsequently, the participants are shown three versions of the sequence (non- subtitled, with English (intralingual) or French (interlingual) subtitles). The results indicate an interaction of both experimental factors. This implies an overall facilitating effect of subtitles for the beginner group (particularly when the subtitles are presented in the partLFLSDQWV¶ ILUVW language), which is in contrast with a distracting effect for the advanced group (more so when both known languages are on-screen). A third study seeks to evaluate the opportunities that these different types of subtitles may provide in foreign language acquisition or consolidation. Keywords audiovisual information, information processing, language fluency, subtitles 1. Introduction It is now established that audiovisual translation (AVT) studies have spread worldwide ± since 1990 especially ± with a proliferation of international conferences and the publication of numerous papers on the subject (Díaz Cintas 2003). Audiovisual studies have also become interdisciplinary, extending beyond the boundaries of translation and incorporating psychology, sociology, film studies as well as linguistics (Remael & Neves 2007). Research in cognitive psychology has widely investigated technical aspects of subtitled film reception since the very presence of subtitles can 273 Dominique Bairstow & Jean-Marc Lavaur PRGLI\ WKH YLHZHU¶V DXGLR DQG YLVXDO LQWHJUDWLRQ SURFHVVHV as a result of competition between the different sources of information (Roskos-Ewoldsen, Yang & Lee 2007). Indeed, a film contains various types of information that can be either audio, video or situational (combining information from the ILOP ZLWK WKH YLHZHU¶V SUHYLRXV NQRZOHGJH. All these types of information can be either linguistic or non-linguistic. The competition between these can RFFXU SDUWLFXODUO\ LI WKH LQIRUPDWLRQ LV FRQVLGHUHG ³GLVSDUDWH EXW VLPXOWDQHRXV´0|WW|QHQ & Sams 2008) as is the case with audio information in one language and written information in another (interlingual subtitles). Initial studies focused mainly on font, colour, size and number of lines (and words per line) in a subtitle, using eye-tracking devices for attention issues or post-viewing questionnaires to investigate comprehension (Wang & Kan G¶<GHZDOOH & DeBruycker 2007). Furthermore, most research studying the cognitive processing of movies used the most common type of subtitling (interlingual), although some did take into account other types of subtitling such as intralingual, subtitles for hearing-impaired viewers and ± more recently ± reversed subtitling. Although these studies have yielded some very interesting results, they do not systematically compare the effects of each type of subtitling on film comprehension. Moreover, the psychological approach of subtitling includes various dimensions, not only technical considerations, but also comprehension and memory processes when more than one language is involved. Therefore, on the same level as the type of subtitles presented on- screen, studies must focus on the characteristics of the audience members, not only their language skills but also their familiarity with different translation methods, as pointed out by Koolstra, Peeters and Spinhof (2002). This contribution aims to deal with the influence of languages in the media and their reception by the audience. Many factors can influence audiovisual processing, some of which are linked to the characteristics of the population and others to the various versions of one movie. The issue of this contribution is to understand how these factors can interact during film processing. With this end in view, the results of two experimental studies (see Bairstow 2011 and Lavaur & Bairstow 2011, for a complete description) that aimed to measure the relative effects of different types of subtitling on film FRPSUHKHQVLRQ GHSHQGLQJ RQ WKH YLHZHU¶V NQRZOHGJH RI WKH RQ-screen languages, will be presented below. In addition, in a third study, based on the findings of the earlier two, we attempt to evaluate the opportunities that these different types of subtitling may provide with respect to foreign language acquisition or consolidation. All three studies used post-viewing questionnaires to evaluate comprehension (Wang & Kan 2003; Lavaur & Nava 2008). 274  Audiovisual information processing by monolinguals and bilinguals 2. The role of interlingual subtitles in three aspects of film comprehension This first study followed the results of two experiments previously run in our laboratory. To begin with, Lavaur and Nava (2008) showed that when a film LVSUHVHQWHGLQWKHYLHZHUV¶QDWLYHODQJXDJHDQGWKHVXEWLWOHVDUe unnecessary to the understanding of the sequence, the subtitles have a negative effect on the perception of visual information, compared with situations in which viewers watch silent and dubbed versions of the same film. Secondly, interlingual subtitles were also shown to be distracting in a study by Grignon, Lavaur and Blanc (2007), but only as far as image processing was concerned. With regard to linguistic information, as expected, subtitles proved to be a great help for viewers with a low fluency level in the spoken language of the film. These results led us to imagine a new situation that would enable us to measure the effects of subtitles on three different aspects of film comprehension (visual, linguistic and situational), depending on the degree of subtitle QHFHVVLW\ IRU WKH YLHZHUV¶ SURSHU XQGHUVWDQGLQJ WKXV GHSHQGLQJ RQ language-fluency levels). As shown in the studies referred to above, the negative effect attributed to subtitles for fluent viewers, as well as the positive effect for non-fluent viewers, was expected to become evident in the overall comprehension scores. In contrast, with regard to the different types of information, subtitles were expected to have an all-round distracting effect for bilinguals, whereas they were thought to be of great help for the PRQROLQJXDOV¶ OLQJXLVWLF LQWHJUDWLRQ $V IRU WKH VLWXDWLRQ-based questions, it was thought that effects could vary since the understanding of a situation depends on the integration of both information from the film (perceptual data) and the viewers¶ previous knowledge. The experiment is described below. 2.1. Participants A total of 32 participants were recruited from the first- and second-year students of the University of Montpellier (France). They were divided into four groups, reflecting their fluency (high or low) in the spoken language of the movie (English) and the version (subtitled in French or not) of the film they were going to be shown. Thus, two high fluency groups (in English) of eight participants each saw either the original (English) or the subtitled version of the film and two low fluency groups (also in English) of eight participants each also saw either of the two versions. 275 Dominique Bairstow & Jean-Marc Lavaur 2.2. Materials The selection of material consisted of two stages: (1) the choice of the film sequence and (2) the selection of different versions of the film. In research on film comprehension, it is obvious that the choice of the sequence is crucial since it forms the basis of the whole experimental setting. Therefore, we shall briefly list the selection method used in the first two studies. In order to limit the risk of the film having been seen by our participants, we limited the choice to films produced between 1950 and 1980. A second criterion was based on the film content. Care was taken to ensure that an equal amount of action and dialogue occurred (for the comprehension questionnaire) and that the number of characters appearing in the clips was enough to create interest but not so many as to create difficulty in following the story. Finally, the dialogues needed to be rich enough for the viewer to make inferences from the situation (context understandable independently from the rest of the film), but also to be adaptable in another language (as subtitles). The extract which was finally selected was an 8-minute, 36-seconds sequence from AlIUHG+LWFKFRFN¶VNorth by Northwest (1959), for which we also had the bilingual scenario including all the dialogues in both English and French. Having extracted the sequence onto a separate DVD, we proceeded to adapt the subtitles to make them as compatible as possible with the spoken dialogues (while following the recommendations set forth by Ivarsson and Carroll (1998) regarding oral and written language modes). Two versions of the film were used: (1) English with French subtitles (interlingual version) and (2) non-subtitled English (original version). Furthermore, a comprehension questionnaire was drawn up. The understanding of a film is based on three essential types of information: (1) visual, (2) audio and (3) situational data (see above). A first questionnaire was created, which consisted of eighty multiple-choice questions about information representing any of these three factors in the sequence. The questionnaire was administered to a group of twelve native French participants, who had watched the French dubbed version of the excerpt (for optimal comprehension). The items for which the success rate was too far from the mean (too easy or too complicated) were eliminated, giving us a questionnaire with 45 questions (15 for each type of information) distributed evenly throughout the film (see appendix A). Each question had one correct answer, three wrong answers and one ,GRQ¶WNQRZ option. 276  Audiovisual information processing by monolinguals and bilinguals 2.3. Design In this experiment, participants were divided into four groups based on the language that they were most fluent in (English or French) and on the version of the film viewed (English or English with French subtitles). All participants had to answer the comprehension questionnaire, which consisted of all three types of question (visual, dialogue, situational). Therefore, the independent variables were (1) Language Fluency, with the two levels bilingual and monolingual, (2) Version of the Film, with the two levels English and English with French subtitles, and (3) Type of Question, with the three levels visual, dialogue, situational. The dependent variable was the number of correct answers to the items in the comprehension questionnaire. 2.4. Procedure All of the participants saw one version of the film and replied to the questionnaire individually on a computer by using a set of headphones to reduce interference from the environment. They were asked to watch the film attentively. In addition, they were warned that they would have to answer a series of questions afterwards. 2.5. Results The data were analysed using a 2 (Version of the Film) x 2 (Language Fluency) ANOVA design with a .05 level of significance Į7KH effect size used ZDV&RKHQ¶Vd. The main results of this experiment show a significant interaction between Version of the Film and Language Fluency, F(3, 28) = 27.84, p < .0001. As expected, after seeing the non-subtitled version of the film, the bilinguals obtained a significantly higher global comprehension score than the monolinguals. On the other hand, the monolinguals achieved a higher level of comprehension than the bilinguals after watching the subtitled version, showing the negative effect of subtitles for bilinguals (Figure 1). 277 Dominique Bairstow & Jean-Marc Lavaur 35 (1.92) (3.92) Mean comprehension scores 30 25 (2.19) (3.59) Monolinguals 20 Bilinguals 15 10 5 English version English version, French subtitles Version of the film Figure 1. Mean comprehension scores (standard deviations in parentheses) showing an interaction between the Version of the Film (English or English with French subtitles) and Language Fluency (monolingual or bilingual) Detailed analysis of the comprehension scores also showed a significant effect of both version and language fluency on the answers to each type of question (see Figure 2). With the original version, the bilinguals obtained higher scores than the monolinguals for both visual data, F(1, 14) = 7.45, d = 1.37, and dialogue data, F(1, 14) = 44.21, d = 3.33, whereas the opposite occurred with the subtitled version for both visual information, F(1, 14) = 12.53, d = 1.77, and dialogue information, F(1, 14) = 6.66, d = 1.29. As for the questions about the situation, the result of the analysis was only barely significant, following the same general pattern as the other two other types of information, but with much smaller differences between scores (best score for bilinguals watching the non-subtitled version; best score for monolinguals when subtitles are on-screen). For a more detailed statistical analysis, see Bairstow 2011. 278  Audiovisual information processing by monolinguals and bilinguals 14 12 (1.77) (1.98) (1.38) (1.41) 10 (2.50) (2.32) (1.59) (1.69) (1.68) Mean score 8 Monolinguals (2.05) (1.55) Bilinguals 6 (2.33) 4 2 0 OV SV OV SV OV SV Visual Dialogues Situation Type of question Figure 2. Mean scores (standard deviations in parentheses) for each Fluency Level (monolingual or bilingual) depending on the Type of question (visual, dialogue or situational) after watching the non-subtitled version (OV) or the subtitled version (SV) 2.6. Discussion The aim of this first study was to examine the effects of subtitles on film FRPSUHKHQVLRQGHSHQGLQJRQWKHYLHZHUV¶IOXHQF\OHYHOLQERWKWKHVSRNHQ and written languages on-screen. The overall results confirm the effects generally associated with subtitles, a distracting effect when they are unnecessary, against a facilitating effect when the viewers do not master the ILOP¶V VSRNHQ ODQJXDJH +RZHYHU WKH SDUWLDO UHVXOWV UHYHDO D PRUH unexpected finding: subtitles seem to help monolinguals to understand the linguistic information. They also seem to facilitate the processing of visual information for monolinguals. This last result is rather confusing since subtitles are generally associated with a loss of visual information perception (this is indeed the case for the near-bilingual sample of this study). A possible explanation for this visual information processing facilitation could lie in the composition of our groups. The very nature of the µforeign language¶ used in this study (English) means that there is very little chance of finding ± in a European country ± participants who have no knowledge of English whatsoever. Therefore, our French monolinguals must have had at least a low level of fluency in English. This means that they might have tried to use this limited knowledge to understand dialogues in the non-subtitled 279 Dominique Bairstow & Jean-Marc Lavaur version (thus using all their concentration to listen, rather than to look at the images). When subtitles are added, one could imagine that the French monolinguals were not concentrating so much on listening, but more on reading and on looking at the pictures. This hypothesis could explain the better processing of visual information by monolinguals when subtitles are on-screen, but it still needed to be put to the test. That is why we set up a second experiment, which addressed the language levels in a more detailed manner. Situation-based questions were not included in the analysis since they seemed, in retrospect, far too subjective for a satisfying statistical analysis. 3. The role of intralingual and interlingual subtitles in film comprehension depending on language levels The second study presented in this contribution focused on the composition of the audience. Indeed, studies usually compare fluent with non-fluent populations, with little regard for viewers in between WKHVH WZR µH[WUHPHV¶. That is why we set out to investigate the effects of subtitles on the comprehension of films by viewers with three different levels of fluency. This decision was taken to shed light on the progressive evolution of comprehension as a function of language fluency. What is more, the effects were measured depending on the type of subtitles ± interlingual or intralingual ± so as to examine their relative effects on the perception and integration of visual and linguistic elements. Both types of subtitling can be expected to be distracting for viewers who are fluent in the two languages of the film, whereas only interlingual subtitles can be expected to help monolingual viewers. For those viewers with a fluency level considered medium, the different types of subtitling should have variable effects since they would be presented in either their dominant or their non-dominant language. Furthermore, these effects should vary depending on the type of information on which the questions are based (visual or linguistic data). A summary of the second experiment and general results are presented below. For a full description and detailed results, the reader is referred to Languages on the screen: is film comprehension related to the viewers' fluency level and to the language in the subtitles? (Lavaur & Bairstow, in Press). The methodology is also explained below. 3.1. Participants For this second study, 90 participants comprising an international cross- section were selected from a French secondary school. Using a lexical test (translation task), elaborated according to the Laxen, Aparicio and Lavaur 280  Audiovisual information processing by monolinguals and bilinguals (2008) lexical database, we were able to separate the participants (from both the French and the international sections) into three groups depending on their language fluency level (in English): (1) beginners, (2) intermediate and (3) advanced. In addition, the participants were asked to fill in a personal data questionnaire regarding their familiarity with English and their film-viewing habits. Finally, the participants were invited to take part in an auto-evaluation task to determine their English and French fluency levels. 3.2. Materials Again, the film fragment and film versions had to be determined. The film used for this experiment was the same as the one used in the previous experiment since it still met our requirements about content, length and flexibility in the dialogues (North by Northwest, Hitchcock 1959). Three different versions of the film clip were prepared: (1) the original version (in English), (2) the original version with English subtitles (intralingual) and (3) the original version with French subtitles (interlingual). Each participant saw only one of the three versions of the film depending on the group that the participant was assigned to (see Section 3.3, Design). In order to create a questionnaire that would evaluate comprehension as accurately as possible, a group of twelve native French participants was asked to watch a silent version of the sequence, followed by a version dubbed into French. After each viewing, the participants in this group were asked to recall as much information as possible in five minutes and to write down the information on a sheet of paper. This setting led to a majority of visual-based information for the first recall (since the film was silent) and dialogue-based information for the second recall (since the participants in this group were asked to recall different information this time). After excluding erroneous and inferential information, we analysed the recalled information in terms of frequency of utterances. The most frequently mentioned data were considered to represent the most prevalent information in the film and that information was used in the making of the experimental questionnaire. In order to balance the proportion of visual and dialogue-based questions ± but also to ensure that visual and dialogue-based questions would be distributed equally throughout the sequence ± some questions used in the previous study were integrated into this new questionnaire. The final version of the questionnaire for the second study contained 42 items (see appendix A). Half of items concerned visual information, the other half dialogue data. 281 Dominique Bairstow & Jean-Marc Lavaur 3.3. Design In the second experiment, two independent variables were combined: (1) Language Fluency, with the three levels beginner, intermediate or advanced, and (2) Version of the Film, with the three levels English (original), English with English subtitles (intralingual) or English with French subtitles (interlingual). Language Fluency and Version of the Film are both between- subject variables while the Type of Question (with the two levels visual or dialogues) is a within-subject variable. As previously, the dependent variable was the number of correct answers to the comprehension questionnaire. 3.4. Procedure All of the participants took part in the experiment individually and in three distinct phases. They began by filling in the personal data questionnaire (with the auto-evaluation task) and the translation test. Next, they were asked to watch the film sequence, having been warned beforehand as to which version they would see. Finally, the participants had to fill in the comprehension questionnaire, presented with the Sphynx Lexica software on the same computer as the one used for watching the film. 3.5. Results The first analysis set out to estimate the linguistic proficiency of each of the three groups, using both the translation task and the auto-evaluation scale (included in the personal-data questionnaire). An overall positive correlation was found between both tasks (Bravais Pearson r (88) = .90 p < .001). This meant that participants who achieved the best scores also rated themselves as having high fluency levels in the auto-evaluation scale. The same correspondence between scores applied for the intermediate and beginner groups. The analysis of the general comprehension scores revealed an overall significant interaction between Version of the Film and Language Fluency, F(4, 81) = 21.22, Ș² = .512 7KH EHJLQQHUV¶ FRPSUHKHQVLRQ VLJQLILFDQWO\ increased when interlingual subtitles were added. However, for the advanced group, the presence of any kind of subtitles on-screen had a detrimental effect on information processing (see Figure 4). 282  Audiovisual information processing by monolinguals and bilinguals 35 (2.81) Mean comprehension scores 30 (1.63) (2.01) 25 (2.44) Beginners (2.21) (1.32) (2.01) Intermediate 20 (2.12) Advanced (2.76) 15 10 Original version Intralingual version Interlingual version Version of the film Figure 3. Mean global comprehension scores (and standard deviations) (experiment 2) showing an interaction between Version of the Film (original, intralingual, interlingual) and Fluency Levels (beginner, intermediate, advanced) The detailed analysis (see Lavaur & Bairstow, in press, for complete statistical data), taking into account the language fluency levels, revealed a significant effect of the presence of subtitles for two groups (beginner and advanced) out of three (no significant difference for the intermediate group). 20 (1.49) 18 (.97) (2.13) (.94) 16 (2.72) (1.23) Mean scores 14 (1.33) 12 (.97) (1.42) Beginners 10 (1.16) Advanced 8 (1.19) 6 (1.69) 4 2 V D V D V D Original Intralingual Interlingual Versions Figure 4. Mean scores (and standard deviations) for two Fluency Levels (beginners, advanced) depending on the Version of the Film (original, intralingual, interlingual) for visual questions (V) and dialogue questions (D) 283 Dominique Bairstow & Jean-Marc Lavaur If we take into account the type of information, the analysis showed the same result patterns as those for global comprehension. As already shown in previous studies, ERWK WKH EHJLQQHU DQG DGYDQFHG JURXSV¶ SURFHVVLQJ decreased ± when processing visual information ± with the addition of VXEWLWOHV :KHQ SURFHVVLQJ GLDORJXH LQIRUPDWLRQ WKH EHJLQQHUV¶ understanding increased slightly with intralingual subtitles and almost HTXDOOHGWKHDGYDQFHGJURXS¶VVFRUH ZLWKLQWHUOLQJXDOVXEWLWOHV. By contrast, DGYDQFHG YLHZHUV¶ FRmprehension decreased steadily with the addition of both types of subtitles. 3.6. Discussion The second study confirmed the general effects associated with subtitles, that is, distraction of overall attention with respect to visual information processing and a source of help for linguistic information comprehension for viewers who have not fully mastered the language spoken in the film. These effects are visible with both types of subtitling (intra- and interlingual) and for both linguistic groups (beginners and advanced viewers), although no significant effect was found for the intermediate group. The fact that the members of this group auto-evaluated their fluency level in a high manner YHU\ FORVH WR WKH DGYDQFHG JURXS¶V HYDOXDWLRQ PLJKW EH D FOXH DV to this absence of effect. This shall be considered in more detail in the general discussion. Moreover, intralingual subtitles seemed to provide a certain amount of help for the beginners, which led us to believe that these viewers read the subtitles ± even though they were in a foreign language ± and tried to use their limited knowledge of this language to understand the film. Bearing this in mind, we decided to set up a new experimental study to check whether viewers would still pay attention to foreign subtitles if the film were in their mother tongue (reversed subtitling) and, if so, whether this could be used for language learning purposes (compared with a situation of conventional/ standard subtitling). 4. The effects of standard and reversed subtitling on film comprehension and lexical retrieval The third and last experiment aimed to link comprehension and language acquisition research. Indeed, it has to be taken into consideration that more and more language learning courses ± in schools or with home learning software ± use audiovisual programmes for language acquisition in general and vocabulary acquisition in particular (Yuksel & Tanriverdi 2009). The reason for this is that a film provides a large semantic and cultural context that can help learners understand the meanings of certain words or 284  Audiovisual information processing by monolinguals and bilinguals expressions. Various cognitive studies have demonstrated that watching subtitled films can induce a certain amount of incidental learning and that viewers can actively use subtitles for vocabulary acquisition if they have been taught appropriate and intentional viewing strategies (Danan 2004). Once again, little comparison has been made of the various types of subtitles available, which led us to look into two different types of subtitling (standard and reversed) to see which type of subtitling induced both the best comprehension and the best memorisation of vocabulary. By comparing the comprehension scores and the number of items correctly memorised for each version, we hoped to determine which type of subtitling induces the best form of incidental vocabulary acquisition and optimum comprehension. Only preliminary results are presented in this contribution. The complete study will be presented at ISB8 (Oslo, 2011). The methodology for the experiment is explained below. 4.1. Participants For the third (and last) experiment, we will show the preliminary results for 24 participants, who were all native French-speaking university students with levels of English-language proficiency which ranged from poor to medium. Their language levels were established with a translation task and a personal data questionnaire (see Section 4.2, Materials). 4.2. Materials In this study, the film chosen was different from the film used in the first and second study. Since we were interested in evaluating only dialogue comprehension, a film with next to no action (yet not totally static so as not to run the risk of boring the participants) but with relatively complex dialogues was needed. Using the same selection method as before ± but adding these last two requirements to the list of constraints ± we ended up choosing a 6-minute, 43-second excerpt from Alfred HLWFKFRFN¶V  ILOP Strangers on a Train. The third experiment aimed to study the relative effects of different types of subtitles on film dialogue comprehension and memorisation. In order to do this, we used four different versions of the film sequence: two non-subtitled versions (the original English and a French dubbed version) and two subtitled versions (standard subtitles with dialogues in English and subtitles in French; and reversed subtitles with French dialogues and subtitles in English). Finally, two questionnaires, a recall task and a translation task were devised. The translation task consisted of six sentences to be translated from English into French and six from French into English. The sentences were extracted 285 Dominique Bairstow & Jean-Marc Lavaur from an English learning CD-ROM (one half of the sentences from the medium level and the other half from the advanced level) and were used to measure WKH SDUWLFLSDQWV¶ IOXHQF\ OHYHO in English. A personal data questionnaire was used to ensure that all of the participants could be considered as equal with respect to features such as age, education and even their film-watching habits. Subsequently, a comprehension questionnaire was created to assess dialogue comprehension only with the items shared out evenly throughout the length of the sequence. This questionnaire consisted of 40 multiple-choice questions (one correct answer, three wrong answers, and one possibility of answering , GRQ¶W NQRZ). Finally, a vocabulary memorisation test was created XVLQJWZHOYHH[WUDFWVIURPWKHILOP¶VGLDORJXHV (in English), in which a missing word had to be recalled by the participants. These twelve extracts were chosen on the basis of several qualities: the sentences were contained in a string of dialogue, not isolated; they contained a noun or an adjective (which was the word which had been removed); this word could be guessed only with great difficulty by using the context provided by the rest of the sentence. Of the twelve missing words, four were considered fairly easy (if the viewer was normally attentive to the film - easy) and eight were considered indicative of memorisation (four words were rather old-fashioned, rarely used nowadays - outdated; the other four words were misleading because many other words could fit in the sentences - ambiguous). 4.3. Design All of the participants had similar levels of fluency in English. They saw only one of the four versions of the film and answered only dialogue-based (linguistic) questions. Therefore, the experimental design had only one between-subjects variable, which was Version of the Film, with the four levels English, French, English with French subtitles or French with English subtitles. Two dependent variables measured the number of correct answers to the comprehension questionnaire for the first, and the number of correctly recalled words from the dialogues for the second. 4.4. Procedure All of the participants took part in the experiment individually and in two phases. First, the participants were asked to fill in the personal data questionnaire with the auto-evaluation task. Subsequently, the participants were asked to watch the film clip, having been warned as to which version they would be shown, and using a set of headphones so as not to be distracted. Following this first viewing, the participants had to answer the 286  Audiovisual information processing by monolinguals and bilinguals comprehension questionnaire (presented on the same laptop as the film). For the second part of this last study, the participants were first asked to perform the translation task, which was printed on a sheet of paper to optimise results. Finally, the participants were warned that they were going to see the same version of the extract as before, after which they would have to answer a final questionnaire. This was the vocabulary recall questionnaire and was administered to all of the participants, even those who had seen the French dubbed version. 4.5. Results The preliminary results of half the sample (24 participants, 6 per version) will be presented below. 35 Mean comprehension score 30 25 20 15 10 5 Original version Dubbed version Original version Dubbed version (English) (French) (English) French (French) English subtitles subtitles Version of the film Figure 5. Comprehension of the dialogues depending on the version of the film (original, dubbed, standard subtitles or reversed subtitles) As far as the comprehension scores are concerned, the preliminary results seem to indicate that the original version is the least well understood while the dubbed version gets the highest score, a result that we expected cRQVLGHULQJ WKH YLHZHUV¶ ORZ IOXHQF\ OHYHO LQ (QJOLVK :KDW LV PRUH WKH value of subtitles is demonstrated as the addition of French subtitles brings the comprehension score almost level with that of the dubbed version. Finally, the addition of English subtitles to the dubbed version does not seem to have a negative effect on the understanding of the film. For complete results and statistical data for the full sample, see Baisrtow and Lavaur (2011). 287 Dominique Bairstow & Jean-Marc Lavaur Mean number of recalled words 7 6 5 4 3 2 1 0 Original version Dubbed version Original version Dubbed version (English) (French) (English) French (French) English subtitles subtitles Version of the film Figure 6. Mean number of words (out of a possible maximum of twelve) correctly recalled depending on the version of the film (original, dubbed, standard or reversed subtitling) The preliminary results for the recall task (completion of twelve dialogue extracts in English missing one specific word) show that viewers watching the French version with English subtitles (reversed condition) obtain the highest number of recalled items, as opposed to those watching the dubbed version, who get the lowest score (since the film was presented in French only). As for the original version, global recall appears to be approximately the same as for the English version with French subtitles. 288  Audiovisual information processing by monolinguals and bilinguals Mean number of items recalled 3,5 3 2,5 Easy 2 Ambiguous 1,5 Outdated 1 0,5 0 Original version Dubbed version Original version Dubbed version (English) (French) (English) French (French) English subtitles subtitles Version of the film Figure 7. Mean number of words (out of a possible maximum of four) correctly recalled depending on the version of the film (original, dubbed, standard subtitling, reversed subtitling) and the type of item (easy, ambiguous, outdated) These results are clearer when looking at the type of words recalled after seeing each version of the film. Indeed, the dubbed version provided the viewer with only the possibility of guessing some of the easy words, as was mostly the case with the original version too. The viewing of the original version with French subtitles led to much the same recall pattern as the original version, whereas the dubbed version with English subtitles gave the best recall, especially for the outdated words. 4.6. Discussion These preliminary results seem to indicate that reversed subtitling is the best scenario as far as incidental memorisation (and maybe learning) of vocabulary is concerned. This could mean that reading rather than hearing a language is easier for both the comprehension and memorisation of a foreign language (as scores were high for both comprehension and memorisation) when a translation is available (auditorily in this case). 289 Dominique Bairstow & Jean-Marc Lavaur 5. General discussion This contribution set out to study the effects that different types of subtitles may have on film comprehension and the role they may play in foreign- language vocabulary memorisation. In order to do this, the results of three experimental studies were reported. The first experiment aimed to investigate the role of standard subtitles in the comprehension of two treatment groups, either fluent in only one language or fluent in both. Overall results confirmed the two effects generally associated with subtitles (distracting when unnecessary, helping comprehension if needed). These effects were extended to individual results depending on the type of information tested (although no significant effect was found for situation-based questions) for the bilingual viewers, but not for the monolinguals, for whom subtitles appeared to help with both linguistic and visual comprehension. This last result was hypothetically linked to the composition of our treatment groups as they might have had better knowledge of English than was assessed (see Section 2.6, Discussion for more in-depth explanations of this hypothesis). This idea was put to the test in a second experiment, ZKLFK IRFXVHG RQ WKH YLHZHUV¶ fluency levels, but also took into consideration the type of subtitling. The second set of results showed an overall interaction between the version of the film and language fluency. Indeed, the higher the fluency level, the more superfluous the subtitles were, and therefore the lower the comprehension scores for subtitled clips was. Furthermore, the distracting effect proved to be stronger when two known languages were on-screen simultaneously (audio dialogues and written subtitles). This effect was also apparent for visual information processing by both the intermediate and the beginner groups, which means that the effect found in the first experiment was not repeated. Moreover, no particular effect was found for the intermediate group, meaning that either the participants in the intermediate group were not sensitive to the presence, absence or type of subtitles, or that the effects of the subtitles were not measurable with the means that we used to test them. Either way, further research is needed to come to a satisfying conclusion. Another interesting result showed that beginners were helped by not only interlingual subtitles. They were partly helped even when the subtitles were intralingual, meaning both linguistic information sources were in a foreign language. This result led us to conclude that viewers pay attention to written text even if it is in a foreign language. This finding also led us to verify whether this would still be the case when the audio language was their own. In the third and last experiment, two subtitling types (standard and reversed) were compared to two non-subtitled versions (original version and French dubbed) of one and the same film. The aim of this study was to find 290  Audiovisual information processing by monolinguals and bilinguals out whether one scenario was better than another to achieve both a good understanding of the film and good memorisation of the vocabulary that was used. The viewers were all native French speakers and had a poor to medium fluency level in English. Effects were estimated following two main lines: (1) the level of comprehension achieved after viewing each version and (2) the number of words recalled from a selection of vocabulary from the film. The preliminary results of this research seem to indicate that reversed subtitling constitutes a situation in which both high comprehension levels and good memorisation can be achieved. Although standard subtitles gave better results than both non-subtitled versions, reversed subtitling clearly seemed to offer more opportunities for a non-fluent viewer. The three studies yielded some very interesting results for cognitive psychology but also for audiovisual translation (AVT) studies. Bearing in mind that a great number of Europeans today are at least bilingual (Media Consulting Group 2007) ± with many being actually multilingual ± studies taking into account fluency levels and the role the levels can play in film UHFHSWLRQ DUH YHU\ PXFK QHHGHG WR PHHW WKH DXGLHQFH¶V QHHGV 0RUHRYHU depending on the type of audiovisual product, the information can be of primary importance to any given film-maker since translation methods might need to vary too, depending on whether the linguistic information is there only to accompany the images or the other way round. This is why it is extremely important to study the ways in which various types of AVT (types of subtitling in this contribution) affect the processing of all the different types of information contained in a film. In the same way, audiovisual productions do not have a purely informational or recreational use anymore. They are also used in learning and are invading more and more classrooms and homes. Indeed, the merits of subtitles have already been highlighted in various studies (Yuksel & Tanriverdi 2009; Koolstra & Beentjes 1999) but have yet to be evaluated in greater detail. For example, future research must look into the pros and cons of each type of subtitling with regard to language learning to find the most efficient approach for viewers, measuring µcosts¶ both in terms of attention (using eye-tracking) and cognitive load (with event-related potentials). Another interesting line of research would be to investigate the various viewing strategies offered to viewers (see Danan 2004) so as to check whether some are better than others and whether it depends on the type of programme and subtitling being viewed. All in all, this field of research is vast and ever-changing. The tools to investigate it are numerous, which promises a long future of exchange on the subject. 291 Dominique Bairstow & Jean-Marc Lavaur References Bairstow 'RPLQLTXH  µ$XGLRYLVXDO Processing while Watching Subtitled Films: A Cognitive ASSURDFK¶ LQ 6HUEDQ, Adriana, Anna Matamala & Jean-Marc Lavaur (eds.) Audiovisual Translation in Close-Up: Practical and Theoretical Approaches. Bern: Peter Lang. 205-219. Bairstow, Dominique & Jean-Marc Lavaur. µ6WDQGDUGYVReversed Subtitling: Effects on Movie Comprehension and Lexical RHWULHYDO¶. Oral communication presented at International Symposium on Bilingualism (ISB8), Oslo, Norway, 15-18 June 2011. Danan 0DUWLQHµ&DSWLRQLQJDQG Subtitling: Undervalued Language Learning SWUDWHJLHV¶ in Meta 49(1): 67-77. Díaz Cintas -RUJH  µ$XGLRYLVXDO Translation in the Third MLOOHQQLXP¶ LQ $QGHUPDQ, Gunilla & Margaret Rogers (eds.) Translation Today: Trends and Perspectives. Clevedon: Multilingual Matters. 192-204. G¶<GHZDOOH, Géry & Wim DeBruycker  µ(\H Movements of Children and Adults while Reading Television SXEWLWOHV¶LQEuropean Psychologist 12(3): 196-205. Grignon, Pamela, Jean-Marc Lavaur & Nathalie Blanc.  µ7KH Effect of Subtitles on Film UQGHUVWDQGLQJ¶3DSHUSUHVHQWHGDWSeventeenth Annual Meeting of the Society for Text and Discourse, Glasgow, July 2007. Ivarsson, Jan & Mary Carroll. 1998. Subtitling. Simrishamn: Transedit. Koolstra, Cees & Johannes Beentjes.  µ&KLOGUHQ¶s Vocabulary Acquisition in a Foreign Language through Watching Subtitled Television Programs at HRPH¶ LQ Educational Technology Research and Development 47(1): 51-60. Koolstra, Cees, Allerd L. Peeters & Herman Spinhof. µ7KHPros and Cons of Dubbing and SXEWLWOLQJ¶LQEuropean Journal of Communication 17(3): 325-354. Lavaur, Jean-Marc & Dominique Bairstow (in pUHVV µ/DQJXDJHV RQ WKH Screen: Is Film Comprehension Related to the ViewerV¶ Fluency Level and to the Language in the Subtitles?¶LQInternational Journal of Psychology. DOI: 10.1080/00207594.2011.565343. Lavaur, Jean-Marc & Sophie Navaµ,QWHUIpUHQFHVLiées au Sous-Titrage Intralangue sur le Traitement des IPDJHVG¶XQHSéquence FLOPpH¶>,QWHUIHUHQFHVGXHWRIntralingual Subtitling on the Processing of Images from a Filmed Sequence]. In Hoc, Jean M. & Yves Corson (eds.) Proceedings of the French Society of Psychology Congress. France. 59-64. Laxen, Janika, Xavier Aparicio & Jean-Marc Lavaur. 2008. µ%DVH Lexicale Trilingue ESF: Mesure du Partage Orthographique et du Recouvrement Sémantique Interlangue [ESF Trilingual Lexical BDVH@¶. Paper presented at the French Society of Psychology; Bordeaux, France, September 2008. Media Consulting Group. 2007, September. Study on Dubbing and Subtitling Needs and Practices in the European Audiovisual Industry: Final Report. London: Peaceful Fish. Remael, Aline & Josélia Neves (eds.). 2007. Introduction. A Tool for Social Integration?: Audiovisual Translation from Different Angles. Special Issue of Linguistica Antverpiensia NS 6: 7-22. Möttönen, Riikka & Mikko Samsµ$XGLRYLVXDO,QWHUDFWLRQ¶LQ+DYHORFN, David, Sonoko Kuwano & Michael Vorländer (eds.) Handbook of Signal Processing in Acoustics. New York: Springer. 731- 745. Roskos-Ewoldsen, Beverly, David Roskos-Ewoldsen, Moon hee Yang & Mina Lee. 2007. µ&RPSUHKHQVLRQRIWKH0HGLD¶ Paper presented at The Annual Meeting of the International Communication Association. San Francisco, May 2007. Wang, An-Hsiang & Yun-Feng Kan. 2003. µ(IIHFWV RI Display Type, Speed, and Text/Background Colour-Combination of Dynamic Display on Users¶ Comprehension for Dual-Task in Reading Static and Dynamic Display IQIRUPDWLRQ¶ LQ The International Journal of Advanced Manufacturing Technology 23(1): 133-138. 292  Audiovisual information processing by monolinguals and bilinguals Yuksel, Dogan & Belgin Tanriverdi  µ(IIHFWV RI Watching Captioned Movie Clips on Vocabulary Development of EFL LHDUQHUV¶ LQ The Turkish Online Journal of Education Technology ± TOJET 8(2): 48-54. Filmography North by Northwest, Alfred Hitchcock (1959) USA. Strangers on a Train, Alfred Hitchcock (1951). USA. Appendix A. Examples from the comprehension questionnaires. Experiment 1: Visual question The first scene takes place: At the bottom of a staircase; In front of a building; In front of an elevator; 1HDUWKHVXEZD\HQWUDQFH,GRQ¶WNQRZ Dialogue question After getting in a taxi, Thornhill asks the driver to stop at: The theatre; The Ritz hotel; His apartment; The Plaza hotel, I GRQ¶WNQRZ Situational question Thornhill starts joking in order to: Amuse his abductors; Divert their attention; Hide his anxiety; Make them IHHOULGLFXORXV,GRQ¶WNQRZ Experiment 2: Visual question While crossing the building hall, the man speaks to: 7KHSRUWHU$FROOHDJXH$JLUOIULHQG+LVPRWKHU,GRQ¶WNQRZ Dialogue question ,QWKLVZD\ZHOHDUQWKHPDQ¶VQDPH .DSODQ7D\ORU7RZQVHQG7KRUQKLOO,GRQ¶WNQRZ 293