Published by Banff Centre Press, Riverside Architectural Press Copyright © 2011 Banff Centre Press, Riverside Architectural Press All rights reserved. No part of this book may be used or reproduced in any manner without written permission from the publisher, except in the context of reviews. Every reasonable attempt has been made to identify owners of copyright. Errors or omissions would be corrected in subsequent editions. Publication Design and Production: Philip Beesley Architect Inc. Art Director: Hayley Isaacs Copy Editor: Claire Crighton eBook Development: WildElement.ca Riverside Architectural Press, www.riversidearchitecturalpress.com Banff Centre Press, www.banffcentre.ca/press Printing and binding by Regal Printing Limited This book is set in Zurich Lt BT and Garamond Library and Archives Canada Cataloguing in Publication Euphoria & Dystopia : the Banff New Media Institute dialogues / edited by Sarah Cook and Sara Diamond. Accompanied by HorizonZero DVD. Includes bibliographical references and index. ISBN 978-0-920159-71-2 ISDN 1-894773-24-1 1. Art and technology. 2. Banff New Media Institute. I. Cook, Sarah, 1974- II. Diamond, Sara, 1954- N72.T4E97 2011 700.1’05 C2011-905947-9 Black CMYK Pantone THE BANFF NEW MEDIA INSTITUTE DIALOGUES EUPHORIA & DYSTOPIA EDITED BY SARAH COOK & SARA DIAMOND Banff Centre Press . Riverside Architectural Press Contents  KELLOGG BOOTH & SIDNEY FELS 10 Foreword 13 Preface and Acknowledgements SARA DIAMOND WITH SARAH COOK 19 Introduction SUSAN KENNARD 978 Afterword 994 bnmi Events and Participants Listing 1056 Further Reading 1059 Biographies 1063 Notes on the HorizonZero dvd 1064 Index Northern Words: Visualizing the Evolution 1101 of 10 Years of New-Media Discourse 1 THE MATERIAL KNOWN AS DATA SARA DIAMOND & SARAH COOK 52 Introduction Navigating the Cognisphere: Meditations on N. KATHERINE HAYLES 72 Visualization, Memory, Database, and Narrative ANDREW SALWAY 84 Bodies in Play, 2005 W. BRADFORD PALEY 89 Quintessence, 2002 WARREN SACK 91 Living Architectures, 2000 THOMAS WHALEN 99 Living Architectures, 2000 PAMELA JENNINGS 106 Living Architectures, 2000 CHAR DAVIES & JOHN HARRISON 112 Living Architectures, 2000 Unforgiving Memory, 2001 HERVÉ FISCHER 120 Artificial Stupidity/Artificial Intelligence, 2002 JEAN-CLAUDE GUÉDON 129 Digital Media Management, 2001 “THE DIGITAL ARCHIVE & 136 Unforgiving Memory, 2001 THE DIGITAL DATABASE” Q&A TAPIO MÄKELÄ 140 Unforgiving Memory, 2001 PEDRO MEYER 147 Flesh Eating Technologies, 1997 NATHANIEL BOBBITT 149 Out of the Box, 1998 PETER LUNENFELD 151 Quintessence, 2002 R.G. GOEBEL 157 Artificial Stupidity/Artificial Intelligence, 2002 JOANNA BERZOWSKA 163 Outside/Inside, 2004 1 The Material Known as Data SARA DIAMOND & SARAH COOK  53 Data comprises a set of organized measurements created by instruments that cali- brate quantifiable qualities of an original source (natural, artificial, or recombinant). What kind of material is data? Is it tangible, solid, ephemeral, mediated? Data sets are shaped by prior decisions, such as the instruments chosen to collect the data, the structure of the database, the source and sampling methods, and the software used. What terms do we use to describe the qualities of data in the age of computation, and how do we image or represent it? How do we manage it? These concerns have driven several decades of consideration at the Banff New Media Institute (Bnmi). This book and this chapter start with metadata, in both senses of the term. Metadata is the data that is about data; it designates the structure of a data set and, at the same time, through description, creates the structure that contains that set— making it accessible, yet limiting it. Metadata provides a frame for each excerpt in the book (Speaker / Transcription information / Talk / Panel / Event / Date and time),1 thus establishing the archival nature of this publication’s foundational materials. Metadata discloses the context and structure of these excerpts. Metadata reminds us that there is other data not present, and draws our attention to the lim- its of the choices made—either consciously, by the editors, or unconsciously, due to the limitations and challenges presented by the source data.2 The front and back covers of the book you hold in your hands make reference to the keywords that dominated the discourse of the Bnmi. These words’ varying intensities are illustrated by glittering star points within the flickering power of the Northern Lights. To create this visualization, we worked with computer scientist 1 For example: Tapio Mäkelä / Transcribed: Full paper from author / Talk: “Understanding the Cultural Value of Memory” / Panel: “Memory, History and the Record in the Digital Era” / Event: Unforgiving Memory / Date: Thursday, August 23, 2001, 9:45–11:30 a.m. 2 In the case of this book these limitations and challenges include audio quality, gaps in recording, and problems with transcription. E U P H O R I A & DY S TO P I A : T H E B A N F F N E W M E D I A I N S T I T U T E D I A LO G U ES  Andrew Salway,3 who came to Banff in 2005 to talk about his work in compu- 54 tational semiotics and text extraction—in this case, the vocabulary of the Bnmi as it was parsed from the many documents included in the making of this book. This transcript of Salway’s talk is a key starting point, and it provides an effective entry into other discussions about the data produced by language and conversa- tion, including artist and computer scientist Warren Sack speaking at Living Architectures: Designing for Immersion and Interaction, on a panel entitled “Data Navigation: Architectures of Knowledge.” 4 Sack talks about scripting Conversation Map (2000), a Usenet newsgroup browser that analyzes the text included in an archive of newsgroup messages from large-scale conversations; the browser then outputs a graphical interface that can be used to search and read the archived mes- sages for social networks, discussion themes, and semantic networks.5 Large-and small-scale conversations formed the heart of Bnmi summits, which were carefully moderated affairs on which the exchange of knowledge in the field depended. As the cover image gives a snapshot of the book’s contents, so too does this chapter suggest the backbone of the “dialogues” as a whole—how science meets art, and how abstract data meets tangible visualization. W. Bradford Paley’s artwork and tool TextArc,6 discussed in this chapter, ties this together, again demonstrating how we can see information in a more aesthetic form. Data visualization in the era of computation serves as one of this chapter’s themes. To explore data visualization, the Bnmi held two summits: Quintessence: The Clumpy Matter of Art, Math and Science Visualization (2003), which considered artists’ and scientists’ differing and parallel approaches to creating models and visualizations, and The Shape of Conversation: Language Simulation, Sonification and Visualization (2005), which was purely dedicated to language visualization. As well, the Bnmi played host to the Association for Computing Machinery (ACM) Symposium on Virtual Reality Software and Technology (2001) and twice co-hosted 3 In 2005, Researcher, Department of Computing, University of Surrey. Of particular relevance to this visualization are Andrew Salway, “Computational Semiotics,” in Proceedings of the Conference on Integrating Information from Different Channels in Multimedia Contexts, eds. Kenneth Holmqvist, Peter Kuhnlein and Hannes Rieser, (Birmingham, UK: ESSLII, 2000), 1–11; and Andrew Salway and Chris Frehen, “Words for Pictures: Analyzing a Corpus of Art Texts,” in TKE 2002: Terminology and Knowledge Engineering, ed. Alan Melby (Nancy, France: GTW, 2002), 113–18. Each of these papers explores applying semiotic approaches to text extraction from expert corpus. 4 Software Designer and Media Theorist 5 See http://www.web.media.mit.edu/~lieber/IUI/Sack/Sack.html and http://www.people.ucsc. edu/~wsack/. 6 Artist and Interaction Designer. See http://www.didi.com/ to view a Paley work that visualizes monetary data. See http://www.textarc.org/ for TextArc, his text visualization software discussed in this book. SARA DIAMOND & SARAH COOK the International Symposium on Smart Graphics (2004, 2010); both events were concerned with making the invisible visible. At Emotional Architectures/Cognitive 55 Armatures/Cognitive Science in Interactive Design (2001), some panels and art- works linked data visualization and analysis with affective experiences. At Living Architectures: Designing for Immersion and Interaction (2000), participants dis- cussed the metaphors of architectural structure and data analysis. The Bnmi held conversations about subjects such as: the transition and translation of paper, photography, film archives, and archival finding aids into digital formats; decisions on what collections and objects to retain or dispose of; and the ways in which metadata has allowed and limited discovery throughout its history. Equally prevalent were concerns about the ephemeral nature of digital creation on one hand, and the overwhelming volume of information (or data) on the other. It was also important to distinguish between an “archive” and a “database.” Three events in 2001—prompted in part by the millennial preoccupation with history, memory, and loss—captured these concerns in different ways: Unforgiving Memory, Digital Media Management, and Media, Material, and Culture: Communicating Canada’s Heritage. These explorations of the technologies of memory occurred in relation to other dialogues concerning physical and cultural memory. Throughout the 10 years, sum- mits—such as Navigating Intelligence: A Banff Summit (1998) and Artificial Stupidity/ Artificial Intelligence (2002)—and panels on artificial intelligence (AI) contributed to this knowledge. Bodies in Play: Shaping and Mapping Mobile Applications (2005) explored context- and location-aware mobile devices, and Bodies in Motion: Memory, Personalization, Mobility and Design (2005) delved into the impact of new memory- archiving technologies, including shape metals and intelligent textiles. DATA A R C H I T ECT U R E A N D DATA V I S UA L I Z AT I O N From the first time The Banff Centre began to research the artistic possibilities of vir- tual reality through The Bioapparatus residency (1989) and through early explorations into authoring tools, the larger notion of visualization—expressing data as a digital visual form—has been a consistent one at the Bnmi. Recently, interest in these issues has grown into a more precise engagement with both information and scientific data visualization; that is, computer scientists have started working with actual data sets to create visualizations. Some of the same technologies and software systems used for virtual-reality research drove the emergence of data visualization and simulation— fields of creative inquiry for artists and designers in their own right, and fields in which an alliance between the sciences, computation, art, and design seemed T H E M AT E R I A L K N O W N A S D ATA  imminent. These emerging areas became themes of art making, research practice and 56 debate at the Bnmi. The Bnmi produced and studied visualizations that were immer- sive and/or interactive. Disciplinary knowledge from other fields—such as architec- ture (structure), literature (metaphor), geography (navigation), and cognitive science (perceptual processes)—has been brought to bear on the creation and study of data visualization, data extraction, and graphics computational knowledge. The Quintessence summit included physicists, such as co-moderator Anthony Zee, who did not use visualization, preferring instead to construct “mental mod- els” or algorithms. Scientists were riveted when artists divulged their conceptual schemas, displayed their visualizations, and argued for abstract modelling rather than indexical representations of their theories. At the same time, cultural theo- rist Peter Lunenfeld raised concerns about the power of the visual, noting that through the mediation of popular culture, “the particular configuration of the contemporary visual discourse is one in which the visual is understood as a reveal- ing: transparency, clarity, and truth.” 7 Rather than opposing visualization per se, Lunenfeld preferred the visualizations of alchemists, which required “domain- specific knowledge” for interpretation. Lunenfeld warned about the “sexiness” of scientific imagery and its uncritical use in popular culture, in policy circles, and even in scientific domains. Some projects involved very direct—almost literal—presentations of data. Emotional Architectures/Cognitive Armatures included a presentation by Derek Hales of the Media Centre in Huddersfield, United Kingdom. Hales gave an overview of Media Centre projects, which allowed public engagement with information at their centre. He described artist Jaap de Jonge’s Speaker’s Corner (2000–01) a 15-metre LED display linked to the web, voice recognition, and SMS street interface. As part of this display, the Media Centre had commissioned content that writers, artists and commercial partners and the public could input via the web, which would then appear on the outdoor screen. Text might “take the form of commenting on a particular discussion subject (which is a part of the site), reacting to what other people have said or some arbitrary idea or rant. Some texts are also provided by passersby who use the microphone on the street booth 7 Presentation given in response to Marc Rioux (National Research Council) and Ravin Balikrishnan (University of Toronto) on the panel “Visualization—Virtual and Enhanced Realities and The Human Experience: What Is The State Of The Current Research? Why Visualize? What Can We Learn From Visualization That We Might Not Otherwise Know?” SARA DIAMOND & SARAH COOK located across the junction opposite the text display.” 8 Hales wondered whether “we are no longer competing for attention, but being offered by computers and 57 programming ways of interacting in more forms and outlets.” 9 In 2005, the bnmi held The Shape of Conversation, a workshop in which partici- pants considered simulation, sonification, and visualization tools. Participants contributed their thoughts on the ways in which these forms of data representa- tion could add dimensions to mobile experiences of all kinds. For example, a panel entitled “Social Visualizations, Social Activities, and Collaboration,” co-moderated by Sara Diamond and Finnish cultural theorist Minna Tarkka,10 featured IBM scien- tist Thomas Erickson11 and database researcher Gerald Penn,12 both of whom had built software that represented knowledge-rich conversations and social processes. A number of speakers provided examples of the ways that language (such as SMS) could be visualized. Calling on the Bnmi’s interest in design, the summit examined experience design for the small screen in genres such as extended-reality gaming. Sheelagh Carpendale discussed strategies she had been developing for measuring emotion in text.13 The workshop echoed Bodies in Play and built on the consider- able research into mobile-experience design the Bnmi had conducted through the Mobile Digital Commons Network, bringing this knowledge face-to-face with the Bnmi’s interests and capabilities in visualization.14 The summit acknowledged the dynamic move into mobile platforms for games, entertainment, training, education, and fitness and sports applications. The summit looked at the next stages of a ubiq- uitous and immersive world, concentrating on emerging technologies that could extract data. Consequently, the challenge involved in representing data that would enable location-specific and highly personalized experiences within the mobile con- text was introduced. Technologies such as GPS and biometric sensors would bring added dimensions to location-based games. 8 From the description of Speaker’s Corner found at http://www.jaapdejonge.nl/portfolio/opdrachten/ speak.html, which offers further explanation of the project. 9 Derek Hales is quoted in the report on the conference by Mathew Kabatoff. 10 Director, m-cult centre for new media culture (Finland) 11 Research Staff Member, IBM Thomas J. Watson Research Center (New York) 12 Professor of Computer Science, Department of Computer Science, University of Toronto 13 Assistant Professor and Canada Research Chair, Innovations in Visualization Laboratory, University of Calgary 14 The summit was video-streamed live to universities and colleges in Canada and abroad as a learning resource as well as a prototype demonstrated through the AccessGrid, desktop audio, and video- conferencing software. T H E M AT E R I A L K N O W N A S D ATA  The summit introduced a conceptual spin-off of data visualization: the idea of “social 58 mapping,” which considers social networks and, with geographic information sys- tems (GIS), is able to place these networks in physical spaces and indicate presence to users.15 On a panel entitled “Understanding Mobility and Social Mapping—Mobile Communities, Affect and Mobility, Mobile and Locative Subcultures,” participants Jan-Christoph Zoels,16 Minna Tarkka, and Julie Zilber17 discussed the ways in which mobile technology created new forms of community and relationships that could be mapped through GPS data as well as patterns of SMS messaging. A number of artists demonstrated mobile projects that used social mapping. One such project was the work of Marc Tuters18 and programmer Karlis Kalnins,19 who had developed the concept of “locative media”—location- and time-specific media that rely on real-time and location-based data. Tuters and Kalnins demonstrated their space-tagging tech- nology Geograffiti (2002), which allows users to place messages into specific locations for other mobile users to find and annotate. Julie Andreyev 20 had worked with automobiles, remixing audio tracks on the fly by location and responding to the presence of other smart vehicles, thus transforming the city into an “interventionist canvas.” Michael Longford 21 and Barbara Crow 22 shared their analyses of the ways that users had interacted with mobile devices during a historical park walk—a stroll through an urban park that featured oral histories and stories that Longford and his colleagues had recorded. A session on “Expressive Texts and Interaction Models” anticipated the emergence of technolo- gies such as Twitter by asking, “How can we model participant behaviours? How can we provoke participation?” Other discussions included examples of data visualization on mobile telephones, or between mobile devices and screens. Moderated by Andrew Salway and Sara Diamond, the session featured the virtual graffiti tool NextText, created by Jason Lewis,23 15 See Janet Abrams and Peter Hall, eds., Else/Where: Mapping New Cartographies of Networks and Territories (Minneapolis: University of Minnesota Press, 2002) for an excellent analysis of the relationship between mapping social and geographic networks, as well as for examples of visualization work by artists and designers. 16 Senior Associate Professor, Interaction Design Institute Ivrea (Italy) 17 Research Leader, Mobile MUSE Research Network (British Columbia) and Co-Director, 7th Floor Media, Simon Fraser University 18 Principle Investigator, Mobile Digital Commons Network (Quebec) 19 Creative Director, Locative Media Lab (British Columbia) 20 Artist and Associate Professor, Emily Carr Institute (British Columbia) 21 Associate Professor, Department of Design Art, Concordia University (Quebec) 22 Associate Professor, Communication and Culture, York University (Toronto) 23 Assistant Professor, Digital Image, Sound and Fine Arts, Concordia University SARA DIAMOND & SARAH COOK which allowed participants to input messages through SMS; these messages were then transmitted to billboards and other screens in layered projections.24 Leena 59 Saarinen25 presented Accidental Lovers, a television series she was co-creating, in which she used sophisticated sampling software to drive narratives by extracting data derived from SMS text messages written by the audience.26 The Mobile Heart Rate Project was also showcased. Its first endeavour was Mimichi, a game designed to take place on the Hoodoo Trail in Banff National Park, which allowed users to explore the park’s ecology through engaging in individual and social game play on their mobile phones. In the game, participants adopted a forest creature that appeared as an animated character on their mobile device. Players were then tasked with protecting this creature’s species on the trail by solving puzzles in order to help them to gather resources, to rest, and to survive. Players downloaded short, location-specific video clips at points along the trail; these provided narrative clues. The game used sensor systems that responded to users’ gestures, which helped to propel the narrative. Data was analyzed in real time, indicating a user’s position on a trail map, resources in the vicinity, and other players. Explaining their approach to creating the project, Tom Donaldson27 and the design team discussed the itera- tive engineering strategy—in which technology was developed alongside iterative designs—and the participatory design process, which engaged users in imaging and testing the iterations and aesthetics of the project. Summit participants then tested the prototypes on the Hoodoo Trail. A panel entitled “Data Beauty—Aesthetics, Sound, Image and Social Visualization” included cognitive scientist Brian Fisher,28 who discussed “Cognition, Sound and Vision” within mobile experience and visualization, specifically examining the new challenges that audio, image, and physical interaction posed for cognitive science. Latvian 3D artist Jaanis Garancs 29 presented his beautiful, frenetic, immersive visu- alizations of news, and Matthew Sloly 30 shared his work on language and text. The summit also included analyses about the development of industrial applications and new markets for 3D mobile and immersive experiences. Industry consultant 24 See http://cspeak.net/station/index.php?id=66&lang=fr&option=com_content&task=view for a discussion of Cityspeak: “Cityspeak is an urban intervention designed to engage people in actively marking up public space.… Our targets are the large-scale LED screens that are increasingly found in Western urban cores. Cityspeak provides a means for urban inhabitants to talk back to these giant screens.” 25 Researcher, University of Art and Design Helsinki (Finland) 26 See http://www.crucible.mlog.taik.fi/productions/accidental-lovers/ for details about the production. 27 In 2005, Lead Faculty, A.R.T. Mobile Research Labs, Banff New Media Institute, The Banff Centre 28 Associate Professor, School of Interactive Arts + Technology, Simon Fraser University at Surrey 29 RIXC, the Center for New Media Culture (Latvia) 30 Visualization Artist, A.R.T. Labs, The Banff Centre, and Co-Designer of the cover visualization T H E M AT E R I A L K N O W N A S D ATA 60 Left Canmore and Banff high school students engage in a participatory-design workshop to create the mobile game Mimichi, Forest Creature during Mobile Digital Commons Network’s Global Heart Rate Project research, 2004. Courtesy of the BNMI. Right Tapio Mäkelä addresses human and machine memory. Unforgiving Memory, 2001. Courtesy of The Banff Centre. Hrad Hekimian 31 contributed his perspective as an inventor of systems and meth- ods for providing compressed digital teletext services and teletext support services. The summit included hands-on workshops in which participants imagined “experi- ences that combine social and physical maps, visualization strategies and location based experience design.” Participants then brainstormed ways to integrate these elements into collaborative mobile experiences. Once the technology to support 3D imaging and visualization moved to a PC platform in the early 21st century, The Banff Centre could afford to create a significant research facility. The Bnmi opened the Advanced Research Technology (A.R.T.) Labs in 2003 under the leadership of Dr. Maria Lantin, with support from the Canada Foundation for Innovation and WestGrid initiative (eventually a part of the Compute Canada network, a $60 million project to operate high-perfor- mance computing collaboration and visualization infrastructure across western Canada). Support was also provided by the Alberta Science and Research Authority (ASRA) and Western Economic Diversification (WD). The A.R.T. Mobile Research Laboratory, created later that year and led by Tom Donaldson, undertook research that included animation on mobile platforms. 31 Principal Consultant, Time Critical Research Inc. (Ontario) SARA DIAMOND & SARAH COOK Nathalie Prévost 68 and Maria Lantin 69 presented responsive abstract plush toys that help children with cognitive challenges by mimicking these children’s developing 69 personalities as expressed by their behaviours.70 Magdalena Wesolkowska 71 discussed aging and memory, suggesting that interactive art installations and virtual environ- ments that draw on photographs, home video, and Super 8-mm film can provide memory triggers for ageing individuals when these installations are embedded into individuals’ living quarters. At a public presentation entitled “Fashioning Memory: Wearing Emotional Systems,” the fashion designers Di Mainstone and Tomoko Hayashi (who were also part of the Am-I-Able Project) explained that they were fascinated by the intersection of smart technologies and fashion, due to the potential for clothing to create highly intimate and personalized experiences. Tom Donaldson and Tina Gonsalves demonstrated Medulla Intimata, jewellery that responds to conversa- tions between the wearer and others by displaying different images, thus indicat- ing the emotional state of the interlocutors. At the end of the event, participants spent a day in workshops brainstorming new opportunities for fields such as sport technology and embedded memory. Appropriately, for a summit about memory and ubiquity, Martha Ladly and Sara Diamond shared with participants several issues of HorizonZero: “DREAM!,” and “GHOST!” A R T I F I C I A L ST U PI D I T Y / A R T I F I C I A L I N T E L L I G E N C E Navigating Intelligence (1999) was a strategic think tank that explored the chal- lenges of representing information, new-media assets, data, and relationships. While it contained several panels on AI and artificial life (AL), it posed a larger set of questions, including those about the implication of search engines. How do we find what we want? How can we mobilize creative intelligence, software tools, and knowledge management to solve the needs of individual users, communities of interest, creators, content aggregators, the public, and commerce? This three-day event enabled participants to take a closer look at AL and AI algorithms, web-crawl- ers, search engines, and other creative and intelligent forms that make up the web and other digital media. The summit considered the spectra of 68 PhD, Cognitive Science, University of British Columbia 69 Visualization Researcher and Manager of A.R.T. Labs, The Banff Centre 70 See www.limbics.com for details about the history and current advancement of Limbics. 71 Lecturer and researcher, Concordia University Faculty of Fine Arts, and researcher, Université de Montréal T H E M AT E R I A L K N O W N A S D ATA 70 Left Control panel for Lucy, the orang-utan robot, created by Steve Grand. Artificial Stupidity/Artificial Intelligence, 2002. Courtesy of the BNMI. Right De-installation of David Rokeby and Tapio Mäkelä’s Steaming Media.org (2002), a Banff Centre co- production that streamed media between saunas in two discrete locations. Courtesy of The Banff Centre. information management from supercomputing to consumer environments. It fol- lowed several presentations at the 1998 Interactive Screen summit by theorist Janet H. Murray,72 who had discussed the conditions under which humans could suspend their disbelief during encounters with AI—for example, in AI–driven games such as Dogz and Catz, the first virtual-pet game. Drawing from performing arts and performance, Emotional Computing: Performing Arts, Fiction and Interactive Experience (2000) also reflected on artificial intel- ligence, focusing on robotics. It explored large-scale robotic performances with a panel entitled “As We Go Marching, Marching—Robots and Performance.” Christopher P. Csikszentmihalyi73 demonstrated DJ I, Robot, which was nomi- nated for the Best Artistic Software award at Berlin’s transmediale festival. Adrianne Wortzel74 shared her National Science Foundation–funded Tele/Robotics Theater Project and Camouflage Town, which produced a robot named Kiru that inhabited the first floor of the Whitney Museum of American Art, interacting with visitors in situ and online. The 2002 summit Artificial Stupidity/Artificial Intelligence (discussed in both this chap- ter and in Chapter 2), explored artificial intelligence, suggesting that AI functions best 72 Interactive Designer and Author of Hamlet on the Holodeck: The Future of Narrative in Cyberspace (Free Press, 1997; Cambridge, MA: MIT Press, 1998) 73 MIT Media Lab 74 Associate Professor, Advertising Design and Graphics Arts, New York Technical University SARA DIAMOND & SARAH COOK in the realm of algorithmic speed and machine intelligence rather than as a model of the human mind. This event also took up notions of identity, examining both the construc- 71 tion of identities using computational tools and human processes, and the construction of affect and other signifiers of identity within machines. Its brief read as follows: There is no intelligence, artificial or otherwise, to data if its quantitative mass is not shaped into qualitative experiences. How much can the machine do for us, and how much is left to our own representational devices? How can we negotiate the interface as boundary and inspiration? A hybrid of engineering and philosophy, of cognitive science, psychology, and physiology, the field of AI is constantly metamorphosing, generating fuzzy, blurred, generative forms through the indeterminacy of software and hardware, mechanism, and interpretation. The machine is who we imagine it to be. What are the boundaries of human and machine consciousness? What are the capabili- ties of affective computing? David Rokeby presented his work The Giver of Names at Artificial Intelligence/ Artificial Stupidity, alongside Steve Grand, inventor of Lucy, the orang-utan robot. Writing about the presentation, Nicholas Zaparyniuk stated: With a grammatical engine that chooses words to talk about the object, Rokeby extended the system’s range to engage and respond. The Giver of Names [includes] a camera pointed at a pedestal, so that when an object is placed on it, the camera and the system analysed its shape, color, texture, and outline, The Giver of Names could use this information to stimulate a knowledge base. This knowledge base has about 100,000 words and ideas that are cross-referenced to their stimulus based on the object’s properties. This stimulus brought about a type of “state of mind” where the system showed different degrees of stimulus based on the properties of the object. One of the most interesting things that Rokeby found from this piece is how he [and others] projected meaning onto the objects based on what the computer said.75 In the essay that begins this chapter, we have relied on N. Katherine Hayles to start untangling the various metaphors and approaches toward data, the archive, semiot- ics, and visualization that proliferate in our information-laden era. The excerpts that follow consider language, data visualization, and conversation and knowledge visualization, as well as emotion, cognition, and visualization. These are followed by a rich dialogue about the nature of memory, the database, and the archive, which pays additional practical attention to the changing nature of the archive and our responsibility toward the historical record. The chapter ends with considerations of new technologies that meld biometric data, the body, and digital tools, creating new opportunities and challenging practices and personal and collective liberties. 75 Nicholas Zaparyniuk, Artificial Stupidity/Artificial Intelligence Conference Research Report, 2007. T H E M AT E R I A L K N O W N A S D ATA 72 Navigating the Cognisphere: Meditations on Visualization, Memory, Database, and Narrative N. KATHERINE HAYLES A common theme uniting this section’s selections is the exponential proliferation of information since the advent of the World Wide Web, which really took off in 1995. The decade from 1995 to 2005 experienced a second flood, as epochal in its way as the one Noah faced—only this time, the world has been inundated with a global rain of bits rather than water droplets. We find ourselves swimming—or, as some would have it, drowning in information. As this ambiguity suggests, the meta- phors one chooses to represent the situation are consequential, whether enacted as computer interfaces, visualization techniques, or linguistic figures. We are enmeshed in a common difficulty, in which the tropes we use to represent our condition are pre-configured by assumptions that assert their power the very moment we try to describe our state. What follows, then, is as much an interrogation of metaphors as it is an exploration of the affordances, artworks, interfaces, and data-management strategies presented by the extraordinary collection of artists, scientists, program- mers, philosophers, cultural critics, humanists, and other researchers brought together by Sara Diamond and her colleagues at the Banff New Media Institute. As my title suggests, the overarching metaphor I have selected as a framework for this meditation is the cognisphere, a term suggested by Thomas Whalen in his presenta- tion on the construction of intelligent agents and interactive narratives.76 The term 76 Thomas Whalen, “Data Navigation, Architectures of Knowledge” (lecture, Living Architectures: Designing for Immersion and Interaction, Banff New Media Institute, September 23, 2000). SARA DIAMOND & SARAH COOK highlights a crucial issue explored in this section—the interplay and tension between information, knowledge, and wisdom. One way to negotiate the different values sug- 73 gested by these terms is to focus on cognition. Playing off a definition of “informa- tion” suggested by Gregory Bateson (and with a nod to Niklas Luhmann), I define “cognition” as “the ability to make distinctions that matter.” 77 The cognisphere, as I am using it, includes conscious thought within humans, as well as human somatic responses; emotional and visceral feedback loops (which constitute a crucial part of human cognition, as Antonio Damasio has convincingly argued);78 intelligent machines of all kinds that are networked and interconnected with each other and human users; embedded sensors and ubiquitous computing, which are tied into the cognisphere through wireless technology, RFID, and infrared communication channels; and of course the computer-mediated communications flowing globally through the Internet and the World Wide Web. Important questions highlighted in this section include how to integrate human perceptual systems with the cognisphere; how to incorporate, modify, and preserve diverse cultural traditions in digital media; how to negotiate between neocortical thought and the somatic and emotional realm of human cognition; how to archive and preserve the ever-changing and transforming content of the cognisphere; how to negotiate between human memory and the diverse kinds of machine memory; and how to think about the relation between data structures and the human activity of creating narratives as a way to understand the world. Given this large scope, it is perhaps not surprising that the selections in this section are more successful in raising questions than in providing definitive answers. Let us begin our interrogation of metaphors, then, with visualization. Broadly consid- ered, visualization can be seen as a technical strategy that enlists the highly evolved 77 Gregory Bateson comments that “difference is a very peculiar and obscure concept. It is certainly not a thing or an event. This piece of paper is different than the wood of this lecture. There are many differences between them—of colour, texture, shape, etc. Of this infinitude, we select a very limited number which become information. In fact, what we mean by information—the elementary unit of information—is a difference that makes a difference.” In Steps to an Ecology of Mind (Chicago: University of Chicago Press, 2000), 457–59. Niklas Luhmann modified Humberto Maturana’s theory of autopoiesis by focusing on the act of making distinctions as the defining act that separates a system from its environment; see “The Cognitive Program of Constructivism and a Reality That Remains Unknown” in Self-Organization: Portrait of a Scientific Revolution, eds. Wolfgang Krohn, Guenter Kueppes, and Helga Noswotny (Dordrecht: Kluwer Academic Publishers, 1990), 64–85. 78 Antonio R. Damasio, Descartes’ Error: Emotion, Reason, and the Human Brain (New York: Quill, 1995); The Feeling of What Happens: Body and Emotion in the Making of Consciousness (New York: Harvest, 2000). T H E M AT E R I A L K N O W N A S D ATA 74 Left Models wearing Spotty Dresses (2004) created by Joanna Berzowska and Sara Diamond for CodeZebra. Courtesy of The Banff Centre. Right An example of a typical summit multi-screen session in the Rice Studio, Jeanne and Peter Lougheed Building during Simulation and Other Re-enactments: Modeling the Unseen, 2004. Courtesy of The Banff Centre. human capacity to discern visual patterns in the service of understanding abstract data flows that would be virtually incomprehensible if presented as numerical tables. In evolutionary terms, visual-pattern recognition was associated with immediate sensory stimuli in the environment, and the ability to discern patterns appropriately often had life-and-death consequences for early humans. Far from being left behind as we vault into the information age, this evolutionary heritage persists, investing visualizations with emotional content that has the effect not only of reifying visual representations but also of making them resonant with qualities we may not con- sciously recognize but to which we respond nevertheless. The very responsiveness to visualizations that makes them effective as analytical tools also makes us vulnerable to implications evoked by them that may not be justified by the data alone. These issues were raised in two Banff discussions: by Jayanne English, in relation to images from astronomical instruments, and by Peter Lunenfeld, in the context of brain-scanning technologies.79 As English noted, some attributes of scientific visualiza- tions are not determined by the data but are added for aesthetic effect—for example, the colourizing of black-and-white images from the Hubble telescope. Lunenfeld notes that this is a practice also common with nano-scale images constructed from data gathered by instruments such as the scanning tunnelling microscope. Technically speaking, there is no colour in the nano realm, because the wavelength at which these 79 Jayanne English, “Quintessence Embodied: Visualization, Physics, Astronomy” (lecture, Quintessence: The Clumpy Matter of Art, Math and Science Visualization, Banff New Media Institute, September 12, 2002); Peter Lunenfeld, “Trendy Science” (lecture, Quintessence: The Clumpy Matter of Art, Math and Science Visualization, Banff New Media Institute, September 12, 2002). N . K AT H E R I N E H AY L ES phenomena take place is shorter than the wavelength at which colour is perceived. In responsible visualizations, there are usually valid criteria for the addition of colour; 75 for example, the wavelengths of various colours may be chosen to coordinate with the quantum interactions of the represented atoms. Such subtleties often escape the non- technical viewer, however, who nevertheless responds to the colours in ways condi- tioned by macro-scale experience, investing the image with qualities that the data do not possess. The effects of such backward feedback loops are consequential when the responses of non-technical viewers are important—for example, where the develop- ment of the technology is heavily dependent on state and federal funding. These considerations provide a context for the visualizations Warren Sack created to represent the content of e-mail messages and to construct meta-categories for that content.80 In his presentation, Sack seemed to waver between apologizing for the un-aesthetic quality of his visualization and defending it as appropriate to the func- tionalities with which he was concerned. Although the interface may have appeared unappealing to a designer’s eye, it clearly displayed the relationships at stake in his analysis—should this be sufficient? The playful animal patterns of CodeZebra—an impressive language game and social-network site created by Sara Diamond and her collaborators—raises different questions.81 Designed to facilitate interactions, the sophisticated software challenges participants to move beyond their existing assump- tions and into higher levels of dialogue. Although players can add games of their own design and edit how their interactions are interpreted, the basic parameters of the site are set by its creators. As Joanna Berzowska later noted, collaborative projects are necessarily haunted by a tension between the structure necessary to generate produc- tive dialogue and a design that overdetermines outcomes.82 Speaking as a designer, Berzowska voiced a concern about the possibility that software design might pre-empt how collaborative dynamics proceed. Perhaps this is a necessary risk, for some kind of generative framework is generally necessary in order to move past received ideas and open up the dialogue to new ground. A different set of issues emerges when the focus turns to the relation between human and machine memory. Hervé Fischer highlighted the flexible and subtle nature 80 Warren Sack, “Data Navigation, Architectures of Knowledge” (lecture, Living Architectures: Designing for Immersion and Knowledge, September 23, 2000). 81 Sara Diamond, “CodeZebra.net” (lecture, Quintessence: The Clumpy Matter of Art, Math and Science Visualization, Banff New Media Institute, September 14, 2002). 82 Joanna Berzowska, “Time, Memory, Emotion–Time as Medium Roundtable” (lecture, Outside/ Inside: Boundary Crossings, a Wearable Design Workshop, Banff New Media Institute, August 4, 2004). T H E M AT E R I A L K N O W N A S D ATA 1 TRANSCRIPTS ANDREW SALWAY Bodies in Play, 2005 W. BRADFORD PALEY Quintessence, 2002 WARREN SACK Living Architectures, 2000 THOMAS WHALEN Living Architectures, 2000 PAMELA JENNINGS Living Architectures, 2000 CHAR DAVIES & JOHN HARRISON Living Architectures, 2000 HERVÉ FISCHER Unforgiving Memory, 2001 Artificial Stupidity/Artificial Intelligence, 2002 JEAN-CLAUDE GUÉDON Digital Media Management, 2001 “THE DIGITAL ARCHIVE & THE DIGITAL DATABASE” Q&A Unforgiving Memory, 2001 TAPIO MÄKELÄ Unforgiving Memory, 2001 PEDRO MEYER Flesh Eating Technologies, 1997 NATHANIEL BOBBITT Out of the Box, 1998 PETER LUNENFELD Quintessence, 2002 R.G. GOEBEL Artificial Stupidity/Artificial Intelligence, 2002 JOANNA BERZOWSKA Outside/Inside, 2004 84 ANDREW SALWAY Bodies in Play, 2005 Andrew Salway / Transcribed: Not noted / Talk: Not noted / Panel: “Visualizing Communication— Language, System, Expression” / Event: Bodies in Play: Shaping and Mapping Mobile Applications / Date: Thursday, May 19, 2005, 11:30 a.m.–1:00 p.m. Andrew Salway’s research at the Department of Computing, University of Surrey, United Kingdom, was concerned with developing intelligent multimedia information systems—like digital image and video libraries—particularly by providing knowledge-rich representations of media content. Salway has an MA in Linguistics (University of Edinburgh), an MSc in Neural Computation (University of Stirling), and a PhD in Computing (University of Surrey). Two major themes of Salway’s research are collateral media and narrative. He has worked with audio description (a spoken account of on-screen action), which is created for an increasing number of television broadcasts and films—in effect, the story told by the moving image is retold in words. Salway’s focus has been on extracting information about characters’ emotions and other frequently occurring story elements, as a way to understand more about stories in computational terms. Andrew Salway: I want to begin by taking a step backward to look at the process of the text analysis that will get us data about language use, so that we can gener- ate a keyword visualization. The texts I have been analyzing are a collection of documents from the Banff New Media Institute over the last 10 years, and the interest—particularly for me, at least—has been to see the development of con- cepts of knowledge in this cross-disciplinary setting. I am going to be talking you through the process, which is actually something that can be one hundred percent automated, so what you are seeing is step-by-step what the machine is doing. I will take each point and show the results as they develop. I am aware that show- ing you lots of numbers and data is not a great strategy for an interesting talk, but if you are not convinced of the need for data visualization in the next 20 minutes, you never will be. I should say: this is a work-in-progress. I have been sent the text by Sarah Cook and the editorial team of the BNMI 10th anniversary project. I have sent this data back, and right now, there is a team of visualization artists looking at the data and making the pictures. I am afraid I can just show you words and numbers today, but I think it is important to give some sense of what the machines can and cannot do by way of getting data about languages.98 98 The resulting visualization appears on the front and back covers of this book. A N D R E W S A LWAY [Showing slides.] I have data for the years of the BNMI, and this is how many words 85 are in all the documents. So, it is slightly biased, but that is not too much of a problem. So, over the last seven years, we have got this number of words being analyzed. A very simple first step is then just to count up how often each word occurs. We put it into a software system to do just that. This tells us that for the text from the year 2001, we have one hundred thousand words (altogether about 90,000 different words). These are the various BNMI events from which the texts come. If you look at it now, on the left side, it shows the frequency of the word. As almost always in the English language, the word “the” is there at the top, and the words “or,” “and,” and so forth. These are not especially interesting. We can scroll down a little way, and the word “p.m.” appears; there are more than a few meeting agendas in there, no doubt. I guess more happens in the afternoon, because “p.m.” comes above “a.m.,” and is more frequent in the texts. Scrolling down the list further, now we start to see some words like “media” and “space”—that perhaps is more charac- teristic of the domain of the BNMI. We can see that the problem is: if we wanted the machine to work with this data, these character strings, it is not going to see the difference between the word “that” and the word “media” and so forth. We cannot presume any prior linguistic knowl- edge. So we need another statistic to help us extract more topical words, and the other statistic that we use is on the right-hand side here. What we do is we take the frequency of the word (the word “the” comes up 6,000-odd times in these texts) and we divide it by the total number of words (about one hundred thousand words), and that gives us a ratio recording. This is a “special language ratio.” This [showing slide] is a special subset of language in the BNMI texts. This number tells us how relatively often the word “the” comes in these texts. In the database, we have got the relative frequency of the word “the” in about one hundred million words of English—as determined by the British National Corpus.99 As what we are doing at the BNMI is in Canadian English, we have used the British National Corpus as a guide. This number is the ratio we have observed against the database, and it actually tells us, when it is near one, that the word “the” is being used pretty much normally in these docu- ments (as are almost all of these words). What we want to do is scroll down and start looking for some big values in the right- hand column: the word “media” occurs here 70 times more often than normal in English. That is reassuring—so we have been talking about the right thing for 99 The British National Corpus is a collection of samples of written and spoken English, numbering more than one hundred million words, freely distributed online. See http://www.natcorp.ox.ac. uk/. T H E M AT E R I A L K N O W N A S D ATA What follows is an extract from the Q&A after W. Bradford Paley’s presentation. In this extract, he is showing a print-out version of his software and art work TextArc, which ana- 89 lyzes word frequency and placement in a text—in this case, the book Alice’s Adventures in Wonderland by Lewis Carroll. Paley presented numerous times at the BNMI—in each case, adopting an improvisational/workshop style of presentation, difficult to reproduce in text. W. BRADFORD PALEY Quintessence, 2002 W. Bradford Paley / Transcribed: 0:00–3:00 Q&A / Talk: “Digital Image Design from the Stock Market to Literature” / Panel: “Data Visualization—Information Architectures and Visualization—Methods and Metaphors” / Event: Quintessence: The Clumpy Matter of Art, Math and Science Visualization / Date: Saturday, September 14, 2002, 1:45–3:45 p.m. W. Bradford Paley is an interaction designer and artist whose focus in both worlds is the readable, clear, and engaging expression of complex data. His visual representations are inspired by the calm, richly layered information in natural scenes. Paley made his first computer graphics in 1973, founded Digital Image Design Incorporated (didi.com) in 1982, and started doing financial and statistical data visualization in 1986. He has exhibited at the Museum of Modern Art, has work included in the permanent collection of the Whitney Museum of American Art, and has received multiple grants and awards for both art and design. Paley’s designs are at work every day in the hands of brokers on the floor of the New York Stock Exchange. He is an adjunct associate professor at Columbia University. Sara Diamond: First of all, I think you should show people the prints you made because they are really beautiful. W. Bradford Paley: Oh, yes, thank you. Having 1,024 x 768 pixels may seem like a lot, but it’s incredibly information-poor. To the man who was talking about looking at a camera obscura and was amazed at the beauty of the density of information: we shouldn’t be amazed at that beauty. That’s what’s in the real world! We are only amazed at that beauty and density because we’ve been living with such information poverty for so long, and we’ve focused our lives on it. So … pardon my rants but it’s [laughs] who I am. I made this because I was so sad about the lack of information density that I had on the screen and so excited about the fact that people like Bill Buxton, a Toronto researcher of user interfaces, was saying that displays like this are going to be avail- able and flexible and an eighth of an inch thick for about five to seven dollars per square foot in about five to 10 years—cheaper than the tapestry that I’m displaying T H E M AT E R I A L K N O W N A S D ATA What follows are three talks all from the same panel—“Data Navigation: Architectures of Knowledge,” from the summit Living Architectures: Designing for Immersion 91 and Interaction, held in 2000—in which important works of computer programming designed to deal with human interaction were presented: Warren Sack’s mapping of e-mail conversations and Thomas Whalen’s creation of Alice, a natural language interac- tive system. The transcript of this panel is followed by a talk from earlier in the same summit, in which artists Char Davies and John Harrison present a more visual way of navigating through a digital landscape. WARREN SACK Living Architectures, 2000 Warren Sack / Transcribed: OCRed from archival transcript / Talk: Not Noted / Panel: “Data Navigation, Architectures of Knowledge” / Event: Living Architectures: Designing for Immersion and Interaction / Date: Saturday, September 23, 2000, 1:30–3:00 p.m. Warren Sack is a software designer and media theorist whose work explores theories and designs for online public space and public discussion. Before joining the faculty at the University of California, Santa Cruz, in the Film & Digital Media Department, Sack was an assistant profes- sor at University of California, Berkeley, a research scientist at the MIT Media Lab, and a research collaborator in the Interrogative Design Group at the MIT Center for Advanced Visual Studies. He earned a BA from Yale College and an SM and PhD from the MIT Media Lab. He worked with artist and designer Sawad Brooks on The Translation Map, a network art-research project that interrogates “translation,” funded in part by the Arts Technology Center of the University of New Mexico, with grants from the Rockefeller Foundation and the National Endowment for the Arts; see translationmap.walkerart.org. More information about Sack’s current work can be found at people.ucsc.edu/~wsack. I’m going to demo you a piece of software that I’ve been working on. The software is more or less a fancy newsgroup e-mail browser—like Netscape Messenger—that you use to read your e-mail. In principle, it can be used like that, but it has some extra features. Before I demo it for you, I want to really give you an explanation of why I did it the way I did it, and get your feedback on these extra features. Specifically what I’m aiming for is to elicit a critique from you. I want to take advantage of the fact that there is a great mix of artists and designers here, and in order to elicit that critique, I need to contextualize the work a little bit, put it in perspective. What I’m interested in specifically is public space and public discourse, especially the new sorts of public spaces that are getting created on the Internet, which arguably are engendering new social and political formations. Things like the open-source movement arguably T H E M AT E R I A L K N O W N A S D ATA  couldn’t have happened without the Internet supplementing new social movements, 92 environmental movements, and so forth. Communications that are being supported by the Internet—that’s my interest in public space and public discourse. That said, what I’m trying to do is write a tool that can allow one to participate in a phe- nomenon that I am simply referring to as a very large-scale conversation. I’m begging the question by using that term. I’m saying that there are these places on the Internet— for example, Usenet—where you’ve got hundreds of thousands of people sending each other thousands of messages. You’ve had this new phenomenon that’s many-to-many, and the existing tools and theories that we have for trying to understand and participate in these things are oftentimes inadequate. If you think of discourse theory or conversa- tion analysis from social sciences, or other kinds of summarization technologies from computer science, they’re just not up to the task. I’m trying to develop some means of navigating and understanding these large-scale conversations that are happening in public spaces. I want to talk about the interface, and you just got a peek at it. Most people’s first reaction is “God, why is it so ugly?” So I want to explain my rationale for doing this. What I’m trying to get the machine to do is to take a bunch of e-mail messages, analyze them—have the machine come up with some notion of who’s talking to whom, what are they talking about, what are some of the emergent themes of discussion, what are some of the emergent meta- phors or definitions if we aggregate across everybody’s language—and then diagram them out. But I have a very specific notion of “diagram” when I say that. Yesterday, Scott [Snibbe] provided us with a beautiful example of a diagram; they don’t have to be ugly. There’s a difference between a diagram and a representation, and I just want to put that up front, that’s the difference Scott pointed out, when he was describing his installation at NTT. 101 If you walked up to his installation and you stood back from it and looked at it, and you looked for representation (as one would if one were stand- ing in front of a painting or something like that), you’d see nothing. That was sort of the point: you only saw something once you stepped into the installation, and then suddenly there was a line that was associated with you, and if someone else got in, there was a line associated with them, and so forth. And those lines were really indi- ces of you. If you step out of the installation, it goes away. If you move, it moves with you. So the difference between this aspect of the diagram—an index of you rather than a symbol—is that it’s connected to you in some way. The common philosophical example of this is that smoke is an index of fire; if the fire goes out, the smoke goes 101 Scott Snibbe’s exhibition took place in 1999 at NTT InterCommunication Center, a venue for media arts and technology in Tokyo, Japan. WA R R E N S A C K away. Or a weathervane points in the direction of the wind, and if the wind changes direction, it changes direction. 93 There’s a difference between coming up with icon-based interfaces or a symbol—as when we talk about computers that have symbolic systems—and what I think is a new kind of approach, which is indexical systems and thinking about producing diagrams and indices rather than representations in symbols. Within the history of art and the history of architecture, there’s a long discussion about diagrams and indices. Rosalind Krauss wrote a nice essay in the ’70s where she reviewed a show at MoMA PS1 in which there was a lot of work that was akin to Scott’s work. Krauss called it something like the “The Art of the Index,” and that gets at some notion of gesture and space and motion and trace. Krauss argues that this goes back at least to [Marcel] Duchamp.102 More recently, last year in Architecture, for example, there was a whole issue on architecture in New York that was devoted to the diagram—things about the dia- gram and how you could supplement the weaknesses of conventional architectural representations with a diagram. We had a discussion about that here yesterday, when Kristine Woolsey 103 was showing how the gestural and movement were get- ting lost if we just looked at plans or models or these other representations. Also, Marcos Novak’s exploration of pulling apart the representations of architecture and supplementing these with a kind of trans-architecture shows how these diagrams or figures might be useful for supplementing these other things.104 The point is: I’m trying to do diagrams. I want to demo for you, so I’m just going to walk through the interface and explain to you what these diagrams are supposed to be doing. The system is called Conversation Map. You give it 5,000 messages and it does a variety of computational linguistics and quantitative sociology analyses on the e-mail messages. It tries to come up with the notion of who’s talking to whom. The upper left-hand corner is a social network, and it’s created in the following way: If Sara sends an e-mail to a group and I respond to it, and then, later in the discus- sion, I send a message and Sara responds to it, we’re said to be reciprocating in the discussion, and we both end up as nodes in the social network. We get a line 102 See Rosalind Krauss, “Notes on the Index: Seventies Art in America,” October Spring and Fall (1977), 210–19. 103 Architect, http://www.woolseystudio.com 104 Marcos Novak, an Architect and Professor, School of the Arts and Architecture, University of California, Los Angeles (UCLA), spoke with Kristine Woolsey on a panel about smart buildings and wired spaces. See http://www.centrifuge.org/marcos. T H E M AT E R I A L K N O W N A S D ATA THOMAS WHALEN 99 Living Architectures, 2000 Thomas Whalen / Transcribed: OCRed from archival transcript / Talk: Not noted / Panel: “Data Navigation, Architectures of Knowledge” / Event: Living Architectures: Designing for Immersion and Interaction / Date: Saturday, September 23, 2000, 1:30–3:00 p.m. Thomas Whalen studied experimental psychology at the University of California, San Diego, the University of British Columbia, and Dalhousie University. In 1979, he moved to Ottawa to take a postdoctoral position with the Department of Communications of the Government of Canada, where he conducted research in human-computer interactions. In addition to continu- ing his position as a scientist working for the Canadian government, Whalen has also been an adjunct professor in the Psychology Department of Carleton University and in the Department of Management Science and Finance at St. Mary’s University. Thomas Whalen: What I’m going to talk about today is natural language. The primary purpose of automobiles is to let people look at drive-in movies. That’s a foolish statement. You know it’s foolish; I know it’s foolish—basically, because it’s wrong. The primary purpose of the web is to let people get information. That doesn’t sound quite as foolish, but I think that it’s equally wrong. Let me tell you why. Just as the primary purpose of an automobile is to navigate through the earth’s geosphere, I think that the primary purpose of the web is to let people navigate through the earth’s knowledge sphere. The earth provides us with an atmosphere or hydro- sphere or biosphere. We have provided the earth with a knowledge sphere; for the sake of aesthetics, I think we should call it the cognisphere. The cognisphere didn’t begin with the web, by any means. It began the first time someone spoke to another person, using some sort of reasonable symbolic language, many millions of years ago. The web just happens to be one of the latest innovations that are adding to the richness of the cognisphere and that make it more easily navigable. So what are these “cognitons” then—these things that are known in the cogni- sphere? Different people have different definitions of knowledge. We heard a few today, and we heard a few yesterday. I’ll give you the definition that I like to use. Knowledge certainly involves text and graphics in some way. It involves audio, video, and animations. It involves any method of presenting information—but we shouldn’t get hung up on that. That’s really losing the forest for the trees. The ways of presenting information are the trees. Knowledge is the forest. It’s the shape of the mass, the overall pattern that emerges from the mass of information and the paths that we have through that mass of information. It’s been said by others that knowledge is what remains when we’ve forgotten all of the facts. I think that’s a bit T H E M AT E R I A L K N O W N A S D ATA  of an overstatement, but in the right direction. Let’s consider a concrete example 100 that’s fairly easy. If we’ve seen or read Arthur Miller’s Death of a Salesman, we know Willy Loman quite well even though we have not memorized every line in that play. But we can still know Willy Loman’s personality, his goals, his flaws, and his ambi- tions. Arthur Miller could have written a completely different play with completely different speeches and still given us the same knowledge about Willy Loman that he did in Death of a Salesman. Our knowledge about Willy is not about the informa- tion and lines of the play; it’s the overall pattern of those lines. It is, in effect, our interpretation of those lines. Okay, let’s change track and show you something completely different. [Whalen begins demonstrating his project Alice.] Just look at the last three lines, where it says: “You enter a classroom; it is empty except for a woman sitting at one of the stu- dent’s desks. There’s a parka draped over the back of her chair. She speaks to you: ‘Hi, are you looking for the psychology class?’” In that situation, what would you say to that woman? You might say “Yes;” you might say “No;” you might say, “Are you the professor?” She laughs, “No, hardly, this is the first university class that I’ve ever taken. Do you take many classes here?” You might reply, “Yes, a lot.” You might reply, “Psychology is my major.” She replies, “I don’t know anything about psychology. I just thought it would be a lark. My calendar says it’s about control of human behaviour.” And on it goes. What I’m showing you is just one of the things that might be in a cognisphere. I’m showing this particular one to you because you may not have seen the computer before it carries on a conversation with you in text, where you can type unconstrained natural language.108 It’s kind of a blind-man-and-the-elephant problem,109 because I can’t possibly, in a few minutes, show you everything that human knowledge has represented. However, I am confident that you know most of them—you’ve lived with most of them your whole life; you’ve lived in a cognisphere. I show you this one—this little toenail of the elephant—because, as I say, it might be interesting to you. I would like to talk a little bit about where this came from—how this was developed. It is software, obviously; there’s a program in there somewhere. The program was one that I wrote, which was really designed for non-fiction. It was designed to 108 Meaning that we may encounter a user interface and believe we are typing to another human being when in fact it is a computer behind the interface. This is often referred to as the Turing test for artificial intelligence. 109 Whalen is making reference to a folk tale in which different blind men all touching different parts of an elephant may describe it as a tree or a snake. THOMAS WHALEN