Multimodal Interaction

Cognitive distribution and human communication

by Stephen J. Cowley

This appeared as:
Cowley, S.J. (2012) Cognitive distribution and human communication / Актуальные проблемы лингвистики и межкультурной коммуникации: Мат-лы Всеросс. науч.-практ. конф-ции c междунар. участием, посвящ. 65-летию ф-та лингвистики и перевода / под ред.С.А. Песиной, И.Р. Пулехи, Ю.Л. Вторушиной, Л.В. Павловой. – Магнитогорск: МаГУ, 2012. – Вып. V. – с. 6-16.

The paper uses examples described elsewhere (Steffensen & Cowley, 2101; Cowley, 2010; 2012) to highlight the... more The paper uses examples described elsewhere (Steffensen & Cowley, 2101; Cowley, 2010; 2012) to highlight the pico-scale of human gestural and vocal communication. This shows how coaction lends unparalleled subtlety to human life as, together, we generate thoughts and actions that are inseparable from bodily coordination. This exemplifies how cognition is distributed as people draw on the (virtual) products of past behaviour to shape future prospects as, unwittingly, they change the circumstances in which they act. Cognitive distribution arises as people use the non-local constraints that dominate folk views of human communication. Given the verbal aspect of language, as Bakhtin saw, speaking (and writing) can renew the voices of others. The thoughts that result, however, are inseparable from interactivity or how bodies perceive, appraise, and re-evoke modes of action that sustain our human forms of life. Finally, having sketched a distributed view of how we communicate in familiar settings, I offer remarks on how scrutiny of human languaging can be pursued in the study of intercultural encounters.

Embodied conversational characters: representation formats for multimodal communicative behaviours

by Christopher Edward Peters

Krenn, B., Pelachaud, C., Pirker, H., and Peters, C. "Embodied conversational characters: representation formats for multimodal communicative behaviours". In P. Petta, C. Pelachaud and R. Cowie (Eds.), Emotion-Oriented Systems: The Humaine Handbook, pp. 389-415, Cognitive Technologies Series, Springer, January 2011
Bibtex available here: http://www.coventry.ac.uk/ec/~cpeters/bibtex/bibtex.html#HandbookRepresentation2011

This contribution deals with the requirements on representation languages employed in planning and displaying... more This contribution deals with the requirements on representation languages employed in planning and displaying communicative multimodal behaviour of Embodied Conversational Agents (ECAs). We focus on the role of behaviour representation frameworks as part of the processing chain from intent planning to the planning and generation of multimodal communicative behaviours. On the one hand, the field is fragmented, with almost
everybody working on ECAs developing their own tailor-made representations, which is amongst others reflected in the extensive references list. On the other hand, there are general aspects that need to be modelled in order to generate multimodal behaviour. Throughout the chapter we take different perspectives on existing representation languages and outline the fundament of a common framework.

Multimodal Parallel Russian Corpus (MultiPARC): Main Tasks and General Structure

by Elena Grishina

To be published at LREC'2012, workshops

The paper introduces a new project, the Multimodal Parallel Russian Corpus, which is planned to be created in the... more The paper introduces a new project, the Multimodal Parallel Russian Corpus, which is planned to be created in the framework of the Russian National Corpus and to include different realizations of the same text: the screen versions and theatrical performances of the same drama, recitations of the same poetical text, and so on. The paper outlines some ways to use the MultiPARC data in linguistic studies.

Living in the social meshwork: the case of health interaction

by Stephen J. Cowley

This appeared as:
Steffensen, S., Cowley, S.J. and Thibault, P.J. (2010). Living in the social meshwork: the case of health interaction. In Cowley, S.J., Steffensen, S. & J.C. Major, Signifying Bodies: Biosemiosis, Interaction and Health, pp. 201-237. Braga: Portuguese Catholic University Press.

The paper presents a new way of investigating real-time decisions in health interaction. Using a simulation... more
The paper presents a new way of investigating real-time decisions in health interaction. Using a simulation setting, it brings home how much depends on team-work that arises as people concert their movements with speech. In a world of norms, coaction both brings off desired results and, drives problem finding. Using close analysis of a single case, we also identify strengths and weaknesses of using  a simulation setting for training purposes. Its advantage, naturally enough, is that it offers an opportunity to practice situated interpretation drawing on scenarios that, in the real-world, may be challenging. This is surprisingly dependent on external resources, closely timed movements and the sharing and prompting of perspectives. At the same time, close investigation at shows that parties play their roles without losing awareness of their ‘real’ relationships. While of no relevance to the health care investigated, their performance in an emergency simulation is inseparable from enacting a social situation.

Experimental Platform for Mobile Information Systems

by Beat Signer

Rudi Belotti, Corsin Decurtins, Moira C. Norrie, Beat Signer and Ljiljana Vukelja, Proceedings of MobiCom 2005, 11th Annual International Conference on Mobile Computing and Networking. Cologne, Germany, August 2005

Interaction design is a major issue for mobile information systems in terms of not only the choice of input-output... more Interaction design is a major issue for mobile information systems in terms of not only the choice of input-output channels and presentation of information, but also the application of context-awareness. To support experimentation with these factors, we have developed a platform that supports the rapid prototyping of multi-channel, multi-modal, context-aware applications. The paper presents the main components of the platform and describes how it was used to develop a tourist information system for an international arts festival where interaction was based on a combination of speech input-output and interactive paper.

A Graphical UIDL Editor for Multimodal Interaction Design Based on SMUIML

by Beat Signer

Bruno Dumas, Beat Signer and Denis Lalanne, Proceedings of UIDL 2011, Workshop on Software Support for User Interface Description Language, Lisbon, Portugal, September 2011

We present the results of an investigation on software support for the SMUIML multimodal user interaction description... more We present the results of an investigation on software support for the SMUIML multimodal user interaction description language. In particular, we introduce a graphical UIDL editor for the creation of SMUIML scripts. The presented graphical editor is fully based on SMUIML for the representation of the underlying data as well as for the dialogue modelling. Due to the event-centered nature of SMUIML, the representation of the multimodal dialogue modelling in the graphical SMUIML dialogue editor has been realised via a state machine. The editor further offers a real-time graphical debugging tool. Compared to existing multimodal dialogue editors, the SMUIML graphical editor offers a dual graphical and textual editing as well as a number of operators for the temporal combination of modalities.

Mudra: A Unified Multimodal Interaction Framework

by Beat Signer

Lode Hoste, Bruno Dumas and Beat Signer, Proceedings of ICMI 2011, 13th International Conference on Multimodal Interaction, Alicante, Spain, November 2011

In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP interaction styles.... more In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP interaction styles. Existing multimodal fusion engines and frameworks range from low-level data stream-oriented approaches to high-level semantic in\-fer\-ence-based solutions. However, there is a lack of multimodal interaction engines offering native fusion support across different levels of abstractions to fully exploit the power of multimodal interactions. We present Mudra, a unified multimodal interaction framework supporting the integrated processing of low-level data streams as well as high-level semantic inferences. Our solution is based on a central fact base in combination with a declarative rule-based language to derive new facts at different abstraction levels. Our innovative architecture for multimodal interaction encourages the use of software engineering principles such as modularisation and composition to support a growing set of input modalities as well as to enable the integration of existing or novel multimodal fusion engines.

SpeeG: A Multimodal Speech- and Gesture-based Text Input Solution

by Beat Signer

Lode Hoste, Bruno Dumas and Beat Signer, AVI 2012, International Working Conference on Advanced Visual Interfaces, Capri Island, Italy, May 2012

We present SpeeG, a multimodal speech- and body gesture-based text input system targeting media centres, set-top boxes... more We present SpeeG, a multimodal speech- and body gesture-based text input system targeting media centres, set-top boxes and game consoles. Our controller-free zoomable user interface combines speech input with a gesture-based real-time correction of the recognised voice input. While the open source CMU Sphinx voice recogniser transforms speech input into written text, Microsoft's Kinect sensor is used for the hand gesture tracking. A modified version of the zoomable Dasher interface combines the input from Sphinx and the Kinect sensor. In contrast to existing speech error correction solutions with a clear distinction between a detection and correction phase, our innovative SpeeG text input system enables continuous real-time error correction. An evaluation of the SpeeG prototype has revealed that low error rates for a text input speed of about six words per minute can be achieved after a minimal learning phase. Moreover, in a user study SpeeG has been perceived as the fastest of all evaluated user interfaces and therefore represents a promising candidate for future controller-free text input.

Revealing the hidden rationality of user browsing behaviour

by Elizabeth FitzGerald (née Brown)

Full citation:
Brown, E., Brailsford, T., Fisher, T. and C. van der Eijk (2007) Revealing the hidden rationality of user browsing behaviour. Proceedings of the Eighteenth ACM Conference on Hypertext and Hypermedia (Hypertext 2007), Manchester, UK, 10-12 Sept 2007, pp85-94.

In this paper, we analyse web log data from user trials of the WHURLE-LS adaptive educational hypermedia (AEH) system... more In this paper, we analyse web log data from user trials of the WHURLE-LS adaptive educational hypermedia (AEH) system from a behavioural perspective. This system allows users to switch from one presentational mode to another (visual, verbal or neutral), and this paper investigates users’ choice of mode as they interacted with the system. We present the main findings of the browsing behaviours within the framework of rational choice theory, and discuss why and when switching might have occurred.

Real users, real results: examining the limitations of learning styles within AEH

by Elizabeth FitzGerald (née Brown)

Full citation:
Brown, E., Fisher, T. and T. Brailsford (2007) Real users, real results: examining the limitations of learning styles within AEH. Proceedings of the Eighteenth ACM Conference on Hypertext and Hypermedia (Hypertext 2007), Manchester, UK, 10-12 Sept 2007, pp57-66. [Awarded Theodore Holm Nelson Prize for Best Newcomer Paper]

This paper examines the current state of AEH (adaptive educational hypermedia) research into explicit learning style... more This paper examines the current state of AEH (adaptive educational hypermedia) research into explicit learning style modelling for user personalisation. It addresses the problem of non-naïve test subjects, who are often in user trials, thus contributing to experimental bias. Instead, the authors suggest using "real people", i.e. users with a range of backgrounds and abilities, in order to gain a truer insight into evidence-based research.

We report on a study carried out with "real" users: around 80 children at a UK primary school. The study investigated sequential and global learning styles as a personalisation mechanism in an AEH system. The user trial involved matching and mismatching users and learning environments to see if learning improved. The AEH system used by the children was DEUS, a new e-learning platform that is conceptually similar to WHURLE, an AEH that also used learning styles as its user model.

No statistically significant differences were found between experimental groups, learning style preferences or learning environments. We discuss the significance of this, and then critically analyse the use of learning styles in relation to this study and also in the wider context.

Adapting for visual and verbal learning styles in AEH

by Elizabeth FitzGerald (née Brown)

Full citation:
Brown, E., Stewart, C. & T. Brailsford (2006) Adapting for visual and verbal learning styles in AEH. Proceedings of the Sixth IEEE International Conference on Advanced Learning Technologies (ICALT 2006), Kerkrade, The Netherlands, 5-7 July 2006, pp1145-1146. [Awarded Best Paper Award in the Adaptive Web-Based Education and Learning Styles (AWELS) workshop]

This paper describes how visual and verbal learning styles have been successfully integrated into an adaptive... more This paper describes how visual and verbal learning styles have been successfully integrated into an adaptive educational environment. User trials of this system were carried out, to determine the effect of the adaptation, and although these user trials do not indicate any statistically significant differences, the qualitative information gleaned from the study indicated that students preferred using this environment over other traditional revision methods, and that they perceived personalized tuition to be better than a ‘one size fits all’ approach.

PhD thesis: The Use of Learning Styles in Adaptive Hypermedia

by Elizabeth FitzGerald (née Brown)

PhD thesis submitted to the University of Nottingham (School of Computer Science), October 2007.

Full citation:
Brown, Elizabeth (2007) The use of learning styles in adaptive hypermedia. PhD thesis. University of Nottingham: UK.

Computer-based learning has become a common phenomenon in the modern age. Many distance-learning systems distribute... more Computer-based learning has become a common phenomenon in the modern age. Many distance-learning systems distribute educational resources on the Internet and indeed entire study programmes are now widely available online. Such a large amount of content and information can be intimidating to learners, who may exhibit different individual characteristics, such as variation in goals, interests, motivation and/or learning preferences. This suggests that a uniform approach taken by learning environments to deliver materials and resources to students is not appropriate and that personalisation of such materials/resources should address users’ differences to provide a customised learning experience, thus enhancing its effectiveness, lowering drop-out rates and maintaining high student motivation.

This thesis addresses the latter issue of learning preferences, specifically investigating learning styles as an adaptation mechanism for personalised computer-based learning. A number of previous studies indicated the positive effect that this kind of adaptation provides, but under closer examination these were not conducted in a scientifically rigorous manner and thus their findings are somewhat limited. This research utilises a quantitative and highly objective approach to investigate visual/verbal and sequential/global learning styles in different user groups. Three user trials were carried out to discover whether there were any benefits to using these learning styles for studying in an adapted environment.

Overall, no statistically significant benefits were found and these findings now shed doubt as to whether learning styles are indeed an effective mechanism for personalised learning.

Reappraising cognitive styles in adaptive web applications

by Elizabeth FitzGerald (née Brown)

Full citation:
Brown, E., Brailsford, T., Fisher, T., Moore, A. and H. Ashman (2006) Reappraising cognitive styles in adaptive web applications. Proceedings of the 15th International World Wide Web Conference (WWW2006), Edinburgh, UK, 22-26 May 2006, pp327-335.

The mechanisms for personalisation used in web applications are currently the subject of much debate amongst... more The mechanisms for personalisation used in web applications are currently the subject of much debate amongst researchers from many diverse subject areas. One of the most contemporary ideas for user modelling in web applications is that of cognitive styles, where a user’s psychological preferences are assessed stored in a database and then used to provide personalised content and/or links. We describe user trials of a case study that utilises visual-verbal preferences in an adaptive web-based educational system (AWBES). Students in this trial were assessed by the Felder-Solomon Inventory of Learning Styles (ILS) instrument, and their preferences were used as a means of content personalisation.

Contrary to previous findings by other researchers, we found no significant differences in performance between matched and mismatched students. Conclusions are drawn about the value and validity of using cognitive styles as a way of modelling user preferences in educational web applications.

Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User Induced Adaptation

by Beat Signer

Bruno Dumas, Beat Signer and Denis Lalanne, Proceedings of EICS 2012, 4th International Conference on Engineering Interactive Computing Systems, Copenhagen, Denmark, June 2012

Multimodal interfaces have shown to be ideal candidates for interactive systems that adapt to a user either... more Multimodal interfaces have shown to be ideal candidates for interactive systems that adapt to a user either automatically or based on user-defined rules. However, user-based adaptation demands for the corresponding advanced software architectures and algorithms. We present a novel multimodal fusion algorithm for the development of adaptive interactive systems which is based on hidden Markov models~(HMM). In order to select relevant modalities at the semantic level, the algorithm is linked to temporal relationship properties. The presented algorithm has been evaluated in three use cases from which we were able to identify the main challenges involved in developing adaptive multimodal interfaces.

Using a Social Semiotic Approach to Multimodality: Researching Learning in Schools, Museums and Hospitals

by Jeff Bezemer

Jeff Bezemer, Sophia Diamantopoulou, Carey Jewitt, Gunther Kress and Diane Mavers.
MODE WORKING PAPER 1, MARCH 2012

The aim of this paper is to show how a substantive area of social research –learning– can be investigated using a... more The aim of this paper is to show how a substantive area of social research –learning– can be investigated using a multimodal social semiotic approach. We apply the approach to three different institutions – a school, a museum and a hospital, illustrating key concepts and addressing issues around pedagogy and technology in contemporary society. We sketch out a theoretical lens, a social semiotic ‘gaze’, introducing key concepts and using data excerpts taken from a number of different research projects. The paper illustrates how a social semiotic perspective on multimodality can illuminate learning by attending to multimodal designs for learning, to multimodal signs of learning, and to social and technological changes and their effects on pedagogy and curriculum.

Russian Da ‘yes’ and Net ‘no’: eye behavior

by Elena Grishina

Uncorrected draft. To be published in Proceedings of International Workshop Approaches to Slavic Interaction, March> 2011

The paper describes the most commonly recurring traits of eye behavior in Russian dialogue and their connection with... more The paper describes the most commonly recurring traits of eye behavior in Russian dialogue and their connection with the Russian words Da ‘yes’ and Net ‘no,’ which are the main modes of expressing assertion and negation in Russian. The eye behavior is analyzed from three main points of view: 1) gaze grammar, 2) eye closing, 3) blinking. The author arrives at the conclusion that 1) Da ‘yes’ and Net ‘no’ are connected to a specific gaze pattern, 2) are combined with eye closing such as special (embedded) gesture and 3) form the basis on which the usage of blinking as punctuation marks and as accent diacritics is founded. The research has been conducted using data from the Multimodal Russian Corpus (MURCO) and on purposefully collected samples.

A participatory design agenda for ubiquitous computing and multimodal interaction: a case study of dental practice

by Margot Brereton

Multimodal Interaction Design: Die multimodale Wechselwirkung und ihre Bedeutung für den Gestaltungsprozess

by Rainer Hirt

Seeing music performance: Visual influences on perception and experience

by Frank Russo

Thompson, W. F., Graham, P., & Russo, F. A. (2005). Seeing music performance: Visual influences on perception and experience. Semiotica, 156, 203-227. P

Audio-visual integration of emotional cues in song

by Frank Russo

Thompson, W. F., Russo, F. A., & Quinto, L. (2008). Audio-visual integration of emotional cues in song. Cognition & Emotion, 22, 1457-1470.