Embodied Cognition

Patterns of Synchronization of Non-verbal Cues and Speech in ECAs: Towards a More "Natural" Conversational Agent

by Nicla Rossini

In Esposito, A., Esposito, A. M., Martone, R. Mueller, V. C., Scarpetta, G. (Eds) Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces: Theoretical and Practical Issues. Pp. 97-104. Springer-Verlag: Berlin.

N.B.: THIS IS A PRE-PUBLICATION EARLY DRAFT. IT MAY CONTAIN ERROS AND DIFFER SIGNIFICANTLY FROM THE PUBLISHED VERSION. FOR QUOTATION PURPOSES, PLEASE ASK ME A COPY OF THE PUBLISHED VERSION.

Please find it on Springer online: http://www.springerlink.com/content/l410238375431524/

This paper presents an analysis of the verbal and non-verbal cues of Conversational Agents, with a special focus on... more This paper presents an analysis of the verbal and non-verbal cues of Conversational Agents, with a special focus on REA and GRETA, in order to allow further research aimed at correcting some traits of their performance still considered unnatural by their final users. Despite the striking performance of new generation ECA, some important features make these conversational agents unreliable to the users, who usually prefer interacting with a classical computer for information retrieval. The users’ preference can be due to several factors, such as the quality of speech synthesis, or the inevitable unnaturalness of the graphics animating the avatar. Apart from the unavoidable traits that can render ECAs unnatural to the ultimate users, instances of poor synchronization between verbal and non-verbal behaviour may contribute to unfavourable results. An instance of synchronization patterns between non-verbal cues and speech is here analysed and re-applied to the basic architecture of an ECA in order to improve the ECA’s verbal and non-verbal synchronization. A proposal for future inquiry aimed at creating alternative model for the ultimate Mp4 output is also proposed, for further development in this field.

Mimesis and language: a distributed view

by Stephen J. Cowley

This is a draft of a paper that appeared recently in a Special Issue of Interaction Studies that takes as its theme: "Language as social coordination: an evolutuionary perspective". The final version appeared as:
Cowley, S.J. (2012). Mimesis and language: A distributed view. Interaction Studies, 13/1: 17-40.

A growing number of scholars regard language as social co-ordination. Not only does this overcome stale debate about... more A growing number of scholars regard language as social co-ordination. Not only does this overcome stale debate about whether langauge is cognitive or communicative but it opens up new thinking about its evolutionary history. Focusing on populations, the paper outlines and critiques Merlin Donald’s view of the human mind. It endorses the view that the evolutionary emergence of language can be traced to mimesis or what, twenty years ago, was called the “ability to produce conscious self-initiated representational acts that are intentional” (Donald, 1991: 168). However, ecological critique rejects the original theory’s appeal to symbolic models. I therefore argue that, while Donald is regarded as likely to be correct about the evolutionary basis of language (and languaging), it is possible to simplify his account. Instead of positing 3 evolutionary thresholds (mimesis, language and writing), mimetic skills may themselves be sufficient to ground all the slow processes of human cognition. Indeed, like tool-making, they probably co-evolved with vocal coordination. As cultures developed tools together with ‘public language’, we gained skills in using distributed cognition –ways of living that were later extended by external resources such as, for example, writing, religions, laws and technologies.

Is a small apple more like an apple or more like a cherry? A study with real and modified sized objects

by Valentina Bazzarin

Co-authored with Anna Maria Borghi

In a categorization experiment we assessed whether seeingobjects automatically activates information on how... more In a categorization experiment we assessed whether seeingobjects automatically activates information on how tomanipulate them. The experiment also aims at investigatingthe role played in a categorization task by online, visualinformation (i.e., of information mediated by the dorsal system), and by information stored in memory (i.e.,information mediated by the ventral system). Participantscategorized photographs of objects manipulable either with apower or a precision grip into artifacts or natural kinds.Target-objects were preceded by primes consisting of photographs of hands in either grasping postures (precision orpower grip) or in a neutral posture (grip). Target-objects couldbe presented either in their real size or in modified size, sothat they activated a different kind of grip. For example, astrawberry was presented both in its real size and with the sizeof an apple, so that it activated a power grip. Results confirm that visual stimuli activate motor information. More importantly, they suggest a crucial role of online, visualinformation even in a categorization task. Results arediscussed in the framework of theories on the role of onlineand offline memory features.

Incidental picture exposure affects later reading: Evidence from the N400

by Leonora Coppens

in press, Brain and Language

Language comprehenders form a mental representation of the implied shape of objects mentioned in the text. In the... more Language comprehenders form a mental representation of the implied shape of objects mentioned in the text. In the present study, the influence of prior visual experience on subsequent reading was assessed. In two separate phases, participants saw a picture of an object and read a text about the object, suggesting the same or a different shape. When the shapes in the two phases mismatched, ERPs during reading showed a larger N400 amplitude than when the shapes matched, suggesting that a picture presented incidentally 15 minutes earlier affected reading. These results further strengthen the case for the interaction of language and visual experience during language comprehension.


Keywords: embodied cognition; reading comprehension; visual experience; ERP; N400

Spatial attention is driven by mental simulations.

by Diane Pecher

Van Dantzig, S. & Pecher, D. (2011). Spatial attention is driven by mental simulations. Frontiers in Cognition, 2:121.

Congruency between word position and meaning is caused by task induced spatial attention.

by Diane Pecher

Pecher, D., van Dantzig, S. Boot, I., Zanolie, K., & Huber, D. E. (2010). Congruency between word position and meaning is caused by task induced spatial attention. Frontiers in Cognition, 1:30.

Representation of categories: Metaphorical use of the container schema.

by Diane Pecher

Boot, I. & Pecher, D. (2011). Representation of categories: Metaphorical use of the container schema. Experimental Psychology, 58, 162-170.

In the present study we investigated whether the mental representation of the concept categories is represented by the... more In the present study we investigated whether the mental representation of the concept categories is represented by the container image schema (Lakoff & Johnson, 1980). In two experiments participants decided whether two pictures were from the same category (animal or vehicle). Pictures were presented inside or outside a frame that should activate the container schema. We found that performance to pictures was
influenced by the frame in congruence with the metaphorical mapping (same category – inside bounded region; different category – not in same bounded region). These results show that the concept categories is metaphorically represented by containers.

Numbers in Space: Differences between concrete and abstract situations.

by Diane Pecher

Pecher, D., & Boot, I. (2011). Numbers in Space: Differences between concrete and abstract situations. Frontiers in Cognition, 2:121.

Mighty metaphors: Behavioral and ERP evidence that power shifts attention on a vertical dimension.

by Diane Pecher

Zanolie, K., Van Dantzig, S., Boot, I., Wijnen, J., Schubert, T. W, Giessner, S., & Pecher, D. (2012). Mighty metaphors: Behavioral and ERP evidence that power shifts attention on a vertical dimension. Brain and Cognition, 78, 50-58.

Thinking about the abstract concept power may automatically activate the spatial up-down image schema (powerful up;... more Thinking about the abstract concept power may automatically activate the spatial up-down image schema (powerful up; powerless down) and consequently direct spatial attention to the image schema-congruent location. Participants indicated whether a word represented a powerful or powerless person (e.g ‘king’ or ‘servant’). Following each decision, they identified a target at the top or bottom of the visual field. In Experiment 1 participants identified the target faster when their spatial position was congruent with the perceived power of the preceding word than when it was incongruent. In Experiment 2 ERPs showed a higher N1 amplitude for congruent spatial positions. These results support the view that attention is driven to the image schema congruent location of a power word. Thus, power is partially understood in terms of vertical space, which demonstrates that abstract concepts are grounded in sensory-motor processing.

Distributed language: implications for volition

by Stephen J. Cowley

The attached paper is a draft for a Russian volume that explored new perspectives on language. It was translated and appeared in Russian as:
С. Дж. Коули. Понятие распределенности языка и его значение для волеизъявления // А.В.Кравченко (ред.). Наука о языке в изменяющейся парадигме знания (Studia linguistica cognitiva 2). Иркутск: БГУЭП, 2009. С. 192-227.

It can be cited as:
Cowley, S. J. (2009). Distributed language: implications for volition. (In Russian). In A, Kravchenko (ed.) New Perspectives on Language and Cognition, pp. 192-227, Irkutsk: Baikal University Press.

Most post-Cartesian views trace human agency to the organism and are thus obliged to either leave aside questions of... more Most post-Cartesian views trace human agency to the organism and are thus obliged to either leave aside questions of volition or, worse, seek explanations in the individual brain. By contrast, when language is recognised as distributed, human cognition is seen to arise as we adapt to life in a collective world. Since language is embodied AND non-local, learning emerges under dual or multiple control –babies learn to talk by participating in “distributed cognitive systems.” In relation to human volition, this opens a gap between tracing actions and feelings to a single brain and privileging the person ‘level’. Although behaviour emerges as people deal with circumstances together, language gives some control over what is not said or done. By focusing on the possible (and what we imagine), we can use the real duration associated with verbal and other thoughts. In short, it is because language is embodied and conventional that we can modulate action/perception: this enables individual organisms to act as living subjects who exert a degree of control over what they – and others – say and do.

Early hominins, utterance-activity, and niche construction

by Stephen J. Cowley

This commentary addresses issues in Falk’s “Prelinguistic communication in hominins: Whence motherese?” http://bortfeld.psy.uconn.edu/UCONNWeb/Bortfeld_BBS2004.pdf

It appeared as:
Cowley, S.J. (2004). Early hominins, utterance-activity and niche construction. Commentary on Falk, D. Behavioral and Brain Sciences, 27, 509-510

In line with mainstream linguistics, Falk assumes that language is based in knowledge of words. Specifically, we are... more In line with mainstream linguistics, Falk assumes that language is based in knowledge of words. Specifically, we are attributed with a genetic propensity for identifying, storing and producing verbal patterns. By challenging the assumption, I suggest that the positive thesis be brought in line with behavioural ecology. Specifically, seen as part of niche-construction, “putting the baby” down can engender skills in modulating how we speak and move; by extension, it is a possible basis for gaining the fine control over phonetic gestures that is necessary to language.

The Neuroscience of the Tragic Mask

by Peter Meineck

Arion 19.1 2011

The Greek tragic mask was the focus of the entire visual and emotional experience of ancient drama. The close... more The Greek tragic mask was the focus of the entire visual and emotional experience of ancient drama. The close coordination of masked movement with music, song, and spoken word allowed the ambiguity of the mask to provoke a highly personal response in the mind of each individual spectator. Their neural processing mechanisms would have been stimulated by the context of what was presented, and then fired to create a deeply personal emotional image. In this way, the visual ambiguity of the mask greatly enhanced the presentation of tragedy. Thus, the tragic mask was far more powerful that the real face of an actor, as it constantly changed, reflecting the emotional realities of each person sitting before its compelling gaze.

A Functional Role for Modality-Specific Perceptual Systems in Conceptual Representations

by Dermot Lynott

Alsmith & de Vignemont 2012 Embodying the mind and representing the body

by Adrian Alsmith

Co-authored with Frédérique de Vignemont. Intro to special issue of the Review of Philosophy & Psychology that we guest edited:
http://www.springerlink.com/content/1878-5158/3/1/

Does the existence of body representations undermine the explanatory role of the body? Or do certain types of... more Does the existence of body representations undermine the explanatory role of the body? Or do certain types of representation depend so closely upon the body that their involvement in a cognitive task implicates the body itself? In the introduction of this special issue we explore lines of tension and complement that might hold between the notions of embodiment and body representations, which remain too often neglected or obscure. To do so, we distinguish two conceptions of embodiment that either put weight on the explanatory role of the body itself or body representations. We further analyse how and to what extent body representations can be said to be embodied. Finally, we give an overview of the full volume articulated around foundational issues (How should we define the notion of embodiment? To what extent and in what sense is embodiment compatible with representationalism? To what extent and in what sense are sensorimotor approaches similar to behaviourism?) and their applications in several cognitive domains (perception, concepts, selfhood, social cognition).

Abstract concepts: Sensory-motor grounding, metaphors, and beyond.

by Diane Pecher

Pecher, D., Boot, I., & Van Dantzig, S. (2011). Abstract concepts: Sensory-motor grounding, metaphors, and beyond. In B. Ross (Ed.). The Psychology of Learning and Motivation, vol. 54 (pp. 217-248). Burlington: Academic Press.

In the last decade many researchers have obtained evidence for the idea that cognition shares processing mechanisms... more In the last decade many researchers have obtained evidence for the idea that cognition shares processing mechanisms with perception and action. Most of the evidence supporting the grounded cognition framework focused on representations of concrete concepts, which leaves open the question how abstract concepts are grounded in sensory-motor processing. One promising idea is that people simulate concrete situations and introspective experiences to represent abstract concepts [Barsalou, L. W., & Wiemer-Hastings, K. (2005). Situating abstract concepts. In D. Pecher, & R. A. Zwaan (Eds.), Grounding cognition: The role of perception and action in memory, language, and thinking (pp. 129–163). Cambridge: Cambridge University Press.], although this has not yet been investigated a lot. A second idea, which more researchers have investigated, is that people use metaphorical mappings from concrete to abstract concepts [Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago: Chicago University Press.]. According to this conceptual metaphor theory, image schemas structure and provide sensory-motor grounding for abstract concepts. Although there is evidence that people automatically activate image schemas when they process abstract concepts, we argue that situations are also needed to fully represent meaning.

Verifying Visual Properties in Sentence Verification Facilitates Picture Recognition Memory

by Diane Pecher

Pecher, D., Zanolie, K., & Zeelenberg, R. (2007). Verifying visual properties in sentence verification facilitates picture recognition memory. Experimental Psychology, 54, 173-179.

According to the perceptual symbols theory (Barsalou, 1999), sensorimotor simulations underlie the representation of... more According to the perceptual symbols theory (Barsalou, 1999), sensorimotor simulations underlie the representation of concepts. We investigated whether recognition memory for pictures of concepts was facilitated by earlier representation of visual properties of those concepts. During study, concept names (e.g., apple) were presented in a property verification task with a visual property (e.g., shiny) or with a nonvisual property (e.g., tart). Delayed picture recognition memory was better if the concept name had been presented with a visual property than if it had been presented with a nonvisual property. These results indicate that modality-specific simulations are used for concept representation.

Approach and avoidance as action effects

by Diane Pecher

Van Dantzig, S., Pecher, D., & Zwaan, R. A. (2008). Approach and avoidance as action effects. Quarterly Journal of Experimental Psychology, 61(9), 1298-1306. doi:10.1080/17470210802027987

Numerous studies use arm movements (arm flexion and extension) to investigate the interaction between emotional... more Numerous studies use arm movements (arm flexion and extension) to investigate the interaction between emotional stimuli and approach/avoidance behaviour. In many experiments, however, these arm movements are ambiguous. Arm flexion can be interpreted either as pulling (approach) or as withdrawing (avoidance). On the contrary, arm extension can be interpreted as reaching (approach) or as pushing (avoidance). This ambiguity can be resolved by regarding approach and avoidance as flexible action plans that are represented in terms of their effects. Approach actions reduce the distance between a stimulus and the self, whereas avoidance actions increase that distance. In this view, action effects are an integral part of the representation of an action. As a result, a neutral action can become an approach or avoidance reaction if it repeatedly results in decreasing or increasing the distance to a valenced stimulus. This hypothesis was tested in the current study. Participants responded to positive and negative words using key-presses. These "neutral" responses (not involving arm flexion or extension) were consistently followed by a stimulus movement toward or away from the participant. Responses to emotional words were faster when the response's effect was congruent with stimulus valence, suggesting that approach/avoidance actions are indeed defined in terms of their outcomes.

Perceptual Processing Affects Conceptual Processing

by Diane Pecher

Van Dantzig, S., Pecher, D., Zeelenberg, R., & Barsalou, L. W. (2008). Perceptual processing affects conceptual processing. Cognitive Science, 32, 579-590.

According to the Perceptual Symbols Theory of cognition (Barsalou, 1999), modality-specific simulations underlie the... more According to the Perceptual Symbols Theory of cognition (Barsalou, 1999), modality-specific simulations underlie the representation of concepts. A strong prediction of this view is that perceptual processing affects conceptual processing. In this study, participants performed a perceptual detection task and a conceptual property-verification task in alternation. Responses on the property-verification task were slower for those trials that were preceded by a perceptual trial in a different modality than for those that were preceded by a perceptual trial in the same modality. This finding of a modality-switch effect across perceptual processing and conceptual processing supports the hypothesis that perceptual and conceptual representations are partially based on the same systems.

Language comprehenders retain implied shape and orientation of objects

by Diane Pecher

Pecher, D., Van Dantzig, S., Zwaan, R. A., & Zeelenberg, R. (2009). Language comprehenders retain implied shape and orientation of objects. Quarterly Journal of Experimental Psychology, 62, 1108-1114.

According to theories of embodied cognition, language comprehenders simulate sensorimotor experiences to represent the... more According to theories of embodied cognition, language comprehenders simulate sensorimotor experiences to represent the meaning of what they read. Previous studies have shown that picture recognition is better if the object in the picture matches the orientation or shape implied by a preceding sentence. In order to test whether strategic imagery may explain previous findings, language comprehenders first read a list of sentences in which objects were mentioned. Only once the complete list had been read was recognition memory tested with pictures. Recognition performance was better if the orientation or shape of the object matched that implied by the sentence, both immediately after reading the complete list of sentences and after a 45-min delay. These results suggest that previously found match effects were not due to strategic imagery and show that details of sensorimotor simulations are retained over longer periods.

Unconstraining theories of embodied cognition

by Diane Pecher

Van Dantzig, S., Zeelenberg, R., & Pecher, D. (2009). Unconstraining theories of embodied cognition. Journal of Experimental Social Psychology, 45, 345-351.

The approach/avoidance effect refers to the finding that valenced stimuli trigger approach and avoidance actions.... more The approach/avoidance effect refers to the finding that valenced stimuli trigger approach and avoidance actions. Markman and Brendl [Markman, A. B., & Brendl, M. (2005). Constraining theories of embodied cognition. Psychological Science, 16, 6-16] argued that this effect is not a truly embodied phenomenon, but depends on participants' symbolic representation of the self. In their study, participants moved valenced words toward or away from their own name on the computer screen. This would induce participants to form a 'disembodied' self-representation at the location of their name, outside of the body. Approach/avoidance effects occurred with respect to the participant's name, rather than with respect to the body. In three experiments, we demonstrate that similar effects are found when the name is replaced by a positive word, a negative word or even when no word is presented at all. This suggests that the 'disembodied self' explanation of Markman and Brendl is incorrect, and that their findings do not necessarily constrain embodied theories of cognition.