Mind, Hands, Face and Body

Communication is multimodal. In everyday interaction we do not communicate only by words, but by our whole body. We talk by gestures, facial expression, gaze, body movements, posture, and these communicative modalities interact with each other in subtle and complex ways. But can we disentangle the different sounds in a symphony, the different pieces in a mosaic? This book claims that the communication scholar can write down the musical score of the communicative symphony by attributing a specific meaning to each single signal – to each gesture, gaze, facial expression – and by finding out lexicons of all communicative modalities. If Linguists have been writing dictionaries of verbal languages for millennia, why not start compiling a new type of dictionaries, and discover the lexicons and the alphabets of gestures, gaze, or touch? Part I of this book (Mind) presents a cognitive model of communication in terms of the notions of goal and belief; Parts II (Hands) and III (Face) analyse gestural and facial communication in detail, by distinguishing universal and cultural aspects in gesture and gaze, showing the differences between gestures that are codified in our mind and gestures that we create on the spot, and teaching how to make a dictionary of touch or how to find the meanings conveyed by the eyebrows. Part IV (Body) presents an annotation scheme to transcribe and analyse signals in all modalities and to capture the meaning of their interaction, that has proved useful for empirical research on multimodality and for its simulation in Embodied Conversational Agents; to illustrate the potentialities of this tool, multimodal discourses are analysed, taken from TV talk shows, political discourse, classroom interaction, speech-therapy sessions, judicial debates, university examinations and comic movies. The subtleties of multimodality are dissected, showing how the whole body can be a tool for indirect and contradictory messages, deception, joke, irony and other sophisticated uses of communication. Prof. Dr. Isabella Poggi teaches General Psychology and Psychology of Communication at Roma Tre University. She works to the construction of a cognitive model of mind, social interaction and communication, through conceptual analysis, observative research and simulation in Embodied Agents. After her first research about the teaching of Italian as a first language, she has published books and papers about emotions (guilt, shame, humiliation, pity, enthusiasm), deception, persuasion, verbal and multimodal communication in humans and machines.