Gesture

Contours of time: Topographic construals of past, present, and future in the Yupno Valley of Papua New Guinea

by Kensy Cooperrider

Co-authored with Rafael Núñez, D Doan, and Jürg Wassmann

Time, an everyday yet fundamentally abstract domain, is conceptualized in terms of space throughout the world’s... more Time, an everyday yet fundamentally abstract domain, is conceptualized in terms of space throughout the world’s cultures. Linguists and psychologists have presented evidence of a widespread pattern in which deictic time—past, present, and future—is construed along the front/back axis, a construal that is linear and ego-based. To investigate the universality of this pattern, we studied the construal of deictic time among the Yupno, an indigenous group from the mountains of Papua New Guinea, whose language makes extensive use of allocentric topographic (uphill/downhill) terms for describing spatial relations. We measured the pointing direction of Yupno speakers’ gestures—produced naturally and without prompting—as they explained common expressions related to the past, present, and future. Results show that the Yupno spontaneously construe deictic time spatially in terms of allocentric topography: the past is construed as downhill, the present as co-located with the speaker, and the future as uphill. Moreover, the Yupno construal is not linear, but exhibits a particular geometry that appears to reflect the local terrain. The findings shed light on how, our universal human embodiment notwithstanding, linguistic, cultural, and environmental pressures come to shape abstract concepts.

Differential Use of Dominant and Nondominant Hands: A Window on Referential and Non-Referential Functions

by Nicla Rossini

To appear in Hudson, J. Magnusson, U. and Paradis, C. (Eds) The construal of spatial meaning: windows into conceptual space.  Oxford University Press: Oxford.

The gestures ASL signers use tell us when they are ready to learn math

by Daniel Lenzen

A prelinguistic gestural universal of human communication

by Connie de Vos

Liszkowski, U., Brown, P., Callaghan, T., Takada, A., & De Vos, C. (2012). A prelinguistic gestural universal of human communication. Cognitive Science, 36, 698-713.

Several cognitive accounts of human communication argue for a language-independent, prelinguistic basis of human... more Several cognitive accounts of human communication argue for a language-independent, prelinguistic basis of human communication and language. The current study provides evidence for the universality of a prelinguistic gestural basis for human communication. We used a standardized, semi-natural elicitation procedure in seven very different cultures around the world to test for the existence of preverbal pointing in infants and their caregivers. Results were that by 10–14 months of age, infants and their caregivers pointed in all cultures in the same basic situation with similar frequencies and the same proto-typical morphology of the extended index finger. Infants’ pointing was best predicted by age and caregiver pointing, but not by cultural group. Further analyses revealed a strong relation between the temporal unfolding of caregivers’ and infants’ pointing events, uncovering a structure of early prelinguistic gestural conversation. Findings support the existence of a gestural, language-independent universal of human communication that forms a culturally shared, prelinguistic basis for diversified linguistic communication.

Embodied conversational characters: representation formats for multimodal communicative behaviours

by Christopher Edward Peters

Krenn, B., Pelachaud, C., Pirker, H., and Peters, C. "Embodied conversational characters: representation formats for multimodal communicative behaviours". In P. Petta, C. Pelachaud and R. Cowie (Eds.), Emotion-Oriented Systems: The Humaine Handbook, pp. 389-415, Cognitive Technologies Series, Springer, January 2011
Bibtex available here: http://www.coventry.ac.uk/ec/~cpeters/bibtex/bibtex.html#HandbookRepresentation2011

This contribution deals with the requirements on representation languages employed in planning and displaying... more This contribution deals with the requirements on representation languages employed in planning and displaying communicative multimodal behaviour of Embodied Conversational Agents (ECAs). We focus on the role of behaviour representation frameworks as part of the processing chain from intent planning to the planning and generation of multimodal communicative behaviours. On the one hand, the field is fragmented, with almost
everybody working on ECAs developing their own tailor-made representations, which is amongst others reflected in the extensive references list. On the other hand, there are general aspects that need to be modelled in order to generate multimodal behaviour. Throughout the chapter we take different perspectives on existing representation languages and outline the fundament of a common framework.

Multimodal clusters in spoken Russian

by Elena Grishina

To be published at LREC'2012, workshops

The paper introduces the notion of multimodal cluster (MMC). MMC is a multicomponent spoken unit, which includes diads... more The paper introduces the notion of multimodal cluster (MMC). MMC is a multicomponent spoken unit, which includes diads “meaning + gesture”, “meaning + phonetic phenomenon” (double MMC) or triad “meaning + gesture + phonetic phenomenon” (triple MMC). All components of the same MMC are synchronized in the speech, gestural and phonetic components conveying the same idea as the semantic component (naturally, with available means). To put it another way, MMC is a combination of speech phenomena of different modi (semantic, visual, sound), which are tightly connected in the spoken language, and roughly speaking mean the same, i.e. convey the same idea by their own means. The paper describes some examples of double and triple MMCs, which are specific for the Spoken Russian.

Non-verbal persuasion and communication in an affective agent

by Christopher Edward Peters

Andre, E., Bevacqua, E., Heylen, D., Niewiadomski, R., Poggi, I., Pelachaud, C., Peters, C., and Rehm, M. "Non-verbal persuasion and communication in an affective agent". In P. Petta, C. Pelachaud and R. Cowie (Eds.), Emotion-Oriented Systems: The Humaine Handbook, pp. 585-608, Cognitive Technologies Series, Springer, January 2011 isbn:978-3-642-15183-5
Bibtex avaialble here: http://www.coventry.ac.uk/ec/~cpeters/bibtex/bibtex.html#HandbookPersuasion2011

This chapter deals with the communication of persuasion. Only a small percentage of communication involves words: as... more This chapter deals with the communication of persuasion. Only a small percentage of communication involves words: as the old saying goes, “it’s not what you say, it’s how you say it”. While this likely underestimates the importance of good verbal persuasion techniques, it is accurate in underlining the critical role of non-verbal behaviour during face-to-face communication. In this chapter we restrict the discussion to body language. We also consider embodied virtual agents. As is the case with humans, there are a number of fundamental factors to be considered when constructing persuasive agents. In particular, one who wishes to persuade must appear credible, trustworthy, confident and non-threatening. Knowing how not to behave is also a vital basis for effective persuasion. This includes resolving task constraints or other factors with the social perception considerations. These social virtual agents face many of the same problems as humans have in controlling and expressing themselves in an appropriate manner so as to establish and maintain persuasive interaction. All along the chapter, much of our discussion will handle concepts applicable both to agent and human behaviour.

Expressive copying behavior for socially interacting machines: A perceptual analysis

by Christopher Edward Peters

Castellano, G., Mancini, M., Peters, C., and McOwan, P. W. "Expressive copying behavior for socially interacting machines: A perceptual analysis", IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, to appear: 2012 doi: 10.1109 / TSMCA.2011.2172415

Successful human interaction commonly involves prototypical exchanges where interactors are engaged, synchronized and... more Successful human interaction commonly involves prototypical exchanges where interactors are engaged, synchronized and harmonious in their behaviors. The copying of aspects of the other’s behavior, at different levels, seems central to establishing and maintaining such empathic connections. Yet many questions remain unanswered, in particular, how it is possible to reflect the same affective content back to the other when the actual motion itself is not exactly the same as theirs. This work presents a perceptual study in which emotional gestures conducted by an actor were mapped onto synthesized versions generated by an embodied virtual agent. Copying is at the expressive level, where qualities such as the fluidity or expansiveness of gestures are considered, rather than exact lowlevel motion matching. Participants were later asked to rate the emotional content of video recordings of both the original and the synthesized gestures. A statistical analysis shows that, in most cases, participants associated the emotional content of the agent’s gestures with that intended to be expressed by the original actor. The results suggest that a combination of the type of movement performed and its quality are important for successfully communicating emotions.

Gestural Coupling and Social Cognition: Möbius Syndrome as a Case Study

by John Michael

Krueger J and Michael J (2012). Gestural Coupling and Social Cognition: Möbius Syndrome as a Case Study. Front. Hum. Neurosci. 6:81. doi: 10.3389/fnhum.2012.00081

Social cognition researchers have become increasingly interested in the ways that behavioral, physiological and neural... more Social cognition researchers have become increasingly interested in the ways that behavioral, physiological and neural coupling facilitate social interaction and interpersonal
understanding. We distinguish two ways of conceptualizing the role of such coupling processes in social cognition: strong and moderate interactionism. According to strong interactionism (SI), low-level coupling processes are alternatives to higher-level individual cognitive processes; the former at least sometimes render the latter superfluous. Moderate interactionism (MI) on the other hand, is an integrative approach. Its guiding assumption is that higher-level cognitive processes are likely to have been shaped by the need to coordinate, modulate and extract information from low-level coupling processes. In this paper, we present a case study on Möbius Syndrome (MS) in order to contrast SI and MI. We show how MS—a form of congenital bilateral facial paralysis—can be a fruitful source of insight for research exploring the relation between high-level cognition and low-level coupling. Lacking a capacity for facial expression, individuals with MS are deprived of a primary channel for gestural coupling. According to SI, they lack an essential enabling feature for social interaction and interpersonal understanding more generally and thus ought to exhibit severe deficits in these areas. We challenge SI’s  prediction  and  show  how  MS  cases  offer  compelling  reasons  for  instead  adopting  MI’s 
pluralistic model of social interaction and interpersonal understanding. We conclude that investigations of coupling processes within social interaction should inform rather than
marginaliz e or eliminate investigation of higher-level individual cognition.

Towards expressive musical robots: a cross-modal framework for emotional gesture, voice and music

by Angelica Lim

Angelica Lim*, Tetsuya Ogata and Hiroshi G Okuno

EURASIP Journal on Audio, Speech, and Music Processing 2012, 2012:3 doi:10.1186/1687-4722-2012-3

It has been long speculated that expression of emotions from different modalities have the same underlying 'code',... more It has been long speculated that expression of emotions from different modalities have the same underlying 'code', whether it be a dance step, musical phrase, or tone of voice. This is the first attempt to implement this theory across three modalities, inspired by the polyvalence and repeatability of robotics. We propose a unifying framework to generate emotions across voice, gesture, and music, by representing emotional states as a 4-parameter tuple of speed, intensity, regularity, and extent (SIRE). Our results show that a simple 4-tuple can capture four emotions recognizable at greater than chance across gesture and voice, and at least two emotions across all three modalities. An application for multi-modal, expressive music robots is discussed.

Keywords: affective computing; gesture; entertainment robots.

The gestures ASL signers use tell us when they are ready to learn math

by Aaron Shield

Published in Cognition, 2012.

The manual gestures that hearing children produce when explaining their answers to math problems predict whether they... more The manual gestures that hearing children produce when explaining their answers to math problems predict whether they will profit from instruction in those problems. We ask here whether gesture plays a similar role in deaf children, whose primary communication system is in the manual modality. Forty ASL-signing deaf children explained their solutions to math problems and were then given instruction in those problems. Children who produced many gestures conveying different information from their signs (gesture-sign mismatches) were more likely to succeed after instruction than children who produced few, suggesting that mismatch can occur within-modality, and paving the way for using gesture-based teaching strategies with deaf learners.

Russian Da ‘yes’ and Net ‘no’: eye behavior

by Elena Grishina

Uncorrected draft. To be published in Proceedings of International Workshop Approaches to Slavic Interaction, March> 2011

The paper describes the most commonly recurring traits of eye behavior in Russian dialogue and their connection with... more The paper describes the most commonly recurring traits of eye behavior in Russian dialogue and their connection with the Russian words Da ‘yes’ and Net ‘no,’ which are the main modes of expressing assertion and negation in Russian. The eye behavior is analyzed from three main points of view: 1) gaze grammar, 2) eye closing, 3) blinking. The author arrives at the conclusion that 1) Da ‘yes’ and Net ‘no’ are connected to a specific gaze pattern, 2) are combined with eye closing such as special (embedded) gesture and 3) form the basis on which the usage of blinking as punctuation marks and as accent diacritics is founded. The research has been conducted using data from the Multimodal Russian Corpus (MURCO) and on purposefully collected samples.

En busca de la precisión: análisis de una configuración manual en el Diccionario normativo de la lengua de signos española

by María del Carmen Cabeza-Pereiro

Illustrators and speech in intellectually-impaired individuals: some observations from a corpus-based study

by María del Carmen Cabeza-Pereiro

Hamlet in England, Hamlet in Exile: What's Hecuba to him, or Kupenga to them?

by Colette Gordon

Some Uses of Gestural Responsive Actions

by Israel Berger

Co-authored with John Rae, in press

A central focus of research on gestures in social interaction has been their relationship to the concurrent production... more A central focus of research on gestures in social interaction has been their relationship to the concurrent production of talk. This report concerns situations in which interactants build responsive actions using gestures. We establish analytically relevant differences in approaches to the analysis of concurrent and responsive gestures before demonstrating some uses of gesture in the building of responsive actions.
Previous research has established that gestures that are produced by recipients during the production of talk are often used to show stance. We show that, likewise, stance is often a salient aspect of gestures that occur following initiating actions that ordinarily make a spoken response relevant. Gestural responses may be used to do sensitive interactional work through recipients’ treatment of stance as salient and by exploiting constraints that conditional relevance imposes on responsive actions. Uses of gestural responsive actions include showing stance in a sequential position, using one action type to do another, and showing that talk is forthcoming. The current findings have implications for the treatment of gesture in conversation analysis and other sequentially oriented methodologies.

Keywords
conversation analysis, gesture, responsive action, second pair part, sequence organisation, action formation

How Do We Communicate About Pain? A Systematic Analysis of the Semantic Contribution of Co-speech Gestures in Pain-focused Conversations

by Samantha Rowbotham

Gestures in the Scrap Yard_Denis Côté’s Carcasses

by Scott Birdwise

Conducting gesture and McNeill's Continua

by Katharine Parton

ICOMCS 2007

The self-contextualisation of conductor gesture

by Katharine Parton

IPrA 2009