P. Jones, UIUC and S. Kasif, UIC, Co-Leaders

Group Members: M. Ackerman, R. Altman, T. DeFanti, P. Dewan, S. Dumais, J. Flanagan, C. Judice, C. Kamm, J. Mariani, R. Nakatsu, G. Olson, R. Picard, L. Rabiner, E. Roth, A. Silberschatz

Preface

As we are heading into the 21st century, we are presented with unprecedented technological advances in computation and communication. These current and future advances create many opportunities for enhancing the quality of our lives at work/home; substantially improving the quality (e.g., cost, reliability, effectiveness) of critical services such as health, transportation, environment, education; as well as making major impacts on the productivity and effectiveness of the business and industrial sectors. We are in fact facing the emergence of a new reality where almost every human activity may be intimately affected by, supported, monitored and sometimes even controlled by ubiquitous computer and communication technology. This suggests an urgent and immediate need to develop scientific and engineering methodologies (methods, solutions, frameworks) for designing, building, and analyzing complex systems that center on fundamental forms of human activity supported by computer and communication technology.

The NSF activity in the area of Human Centered Systems is a broad research area that addresses the development of scientific and engineering methods to support the construction and evaluation of complex technological systems that support fundamental human activities such as communication, interaction, visualization, planning and management, creating, monitoring, collaboration, information extraction, education/training, business, etc.

There are four complementary goals:

a) to scale up current technology in order to support (reliably and cost effectively) human centered activities

b) to develop new and revolutionary technology that expands the space of current human activities

c) to expand our understanding of human behavior and needs in view of the changing environments

d) to increase the understanding of the effect of technology on human life

This effort expands and substantially generalizes existing notions of "human-computer interaction " and "user interface design" as core activities of Human-Centered Systems. While still important, this initiative also includes many fundamental topics in computing, communications, epistemology, and language that emerge from the need to develop complex computational/communication frameworks for supporting diverse human activities. In fact, one of the main goals of this initiative is to create an interdisciplinary program that fuses ideas and methods from engineering (e.g., computer science and electrical engineering) and behavioral sciences (e.g., psychology, economics, social informatics)

Initiatives in the area of Human-Centered Systems can be organized around "Grand Problems" in substantive areas such as education, health care, aviation, transportation, collaborative research and development, and political activism/participative democracy. Part of the solution is the engineering of human-centered information technology (i.e., building such complex systems as digital libraries, cockpits, and large-scale information systems). The analysis, design, construction, and evaluation of such human-centered engineered systems rests on three interrelated and equally important activities: (1) human-centered design methodologies that incorporate principles and methods for the modeling, design, and evaluation of open, adaptive, flexible, and effective human-machine systems; (2) technological developments that enable key capabilities (e.g., text to speech technologies for natural and effective auditory verbal feedback); and (3) behavioral and social science advances in theory and method for the analysis and modeling of human performance and behavior from physical, cognitive, affective, social, and organizational perspectives.

General characteristics of technological systems that are human-centered include:

They take into account human perceptual and motor capabilities and limitations

They support actual practice (real behavior in real tasks) effectively

They are flexible rather than rigid -- can be used in a variety of ways and do not

unnecessarily constrain the user(s)

 They are adaptive and context-sensitive to the changing needs of the user(s)

 They are open and inspectable so that they can be understood by user(s)

They are engaging and enjoyable

Design and evaluation is fundamentally iterative and longitudinal; new technology

fundamentally changes the nature of tasks and needs to be examined carefully in the

context of real practice over time

In this section of this report, we focus on communication and collaboration in the context of human-centered systems. Indeed, it can be argued that collaboration is fundamental to a human-centered design stance, or even that collaboration is fundamental to intelligence (Goody, 1995). Here we discuss both human-computer interaction as a type of collaboration and information technology as a medium for human collaboration.

1. Introduction

Communication and collaboration are important components of a comprehensive approach to the analysis, design, and evaluation of Human-Centered Systems. Indeed, it may be argued that collaboration is a fundamental part of effective decision making and problem solving in complex environments. We loosely define communication as the exchange of messages or information among multiple agents and collaboration as the creation of shared understanding (Schrage, 1990) or joint progress towards one or more goals shared by multiple agents. The "agents" of interest are humans and computers (in particular, software programs that interact directly with people). A wide variety of disciplines are relevant in addressing these issues, including linguistics, artificial intelligence, psychology, sociology, information systems, networking, multimedia, and organizational behavior and communication. Distributed artificial intelligence, multi-agent systems, human-computer interaction, computer-supported cooperative work, and computational and mathematical organizational modeling are relevant interdisciplinary specializations that have achieved recognition recently in the academic community.

1.1. Varieties and Characteristics of Communication and Collaboration

Why do people collaborate? Schmidt (1994) offers three fundamental reasons for cooperative work: augmentative (there is too much work for one agent; e.g., lifting a heavy object), integrative (integration of different techniques and expertise, as in concurrent engineering), and debative (debate among different perspectives, as in scientific discourse). There are also social incentives for group work, such as (a) knowing that others depend on you may motivate you to do your part or (b) working in a team is more fun and offers possibilities for friendship and so on. There is also a vast social science literature on critical mass theory, the diffusion of technology, and cultural appropriation of technological artifacts that is relevant for analyzing how and why communication technologies are appropriated and absorbed into practice. For example, "Web presence" is increasingly important as a means for advertising, information exchange, and participating in a community of practice.

The "overhead" of collaboration and communication involves several interrelated facets of behavior. First, the "compute versus communicate" tradeoff as articulated in computer science is relevant here: the very act of communication itself requires resources on the part of an agent to design and send a message to others, to share or publish information or data. Second, the "invisible meshing" of activity in collaborative systems that has been termed "articulation work" (Schmidt, 1994) includes direction of attention (e.g., verbal, gestural or other behaviors which mean "look over there") and task allocation (e.g., "you do that") as activities in which agents engage to perform coordinated activity with others. Third, nonverbal cues form part of the resources used to create shared meaning and make inferences about another's intentions, including aspects of social presence, emotional state, and so on.

Communication and collaboration can take place over varying dimensions of time and space; participants can work together synchronously or asynchronously and can be physically co-located or remote (Baecker, 1991; Schmidt, 1994). Collaborating agents can vary in their degree of interdependence; for example, they may be semi-autonomous or may be more tightly coupled or "collective" (Schmidt, 1994).

There are several ways in which we conceive of human-computer communication and collaboration that can be framed as the respective roles of the human and computer in the interaction:

1.2. Human actors in a shared virtual environment

The role of the computer-as-environment is to mediate and support interaction among multiple humans. In virtual environments, information technology generates, provides, and captures rich and natural sensory signals to and from the human. Varieties of embedded computing or augmented virtual reality systems mix together aspects of the material and virtual. A human-centered approach to the design of virtual environments includes multimodal and multimedia systems that handle some combination of visual, auditory, voice, and haptic inputs and outputs that are intended to capitalize on and be shaped to accommodate human perceptual and response capabilities and limitations. This may be to mimic real-world interactions or to expand human perceptual, intellectual, and motor capabilities.

Furthermore, such environments may include explicit representation of oneself and other humans; such avatars include representation of cognitive, affective, social and organizational aspects of the human actors "behind" the avatar (e.g., natural and expressive faces and gestures, representing and reasoning about others' places in organizational systems, social relationships, and "who knows what"). Related issues include the blurring of boundaries between the 'real world', augmented reality, embedded, and virtual environments; and diversity in terms of assistive technologies for special populations and multicultural issues (including cultures defined by academic disciplines or community of practice as well as cultures defined by ethnicity or country of origin).

1.3. Human collaboration with intelligent systems

In this paradigm, the computer-as-other engages in dialog and joint problem solving with human actors. A human-centered approach to the design of intelligent systems can be conceptualized in several different ways: (1) The intelligent system as a team player that is reliable, predictable, trustworthy and engages in cooperative problem solving with human practitioners; (2) Representation aiding in which context-sensitive visualizations support problem solving by humans; and (3) Cognitive tools that assist humans in decision making and problem solving (Roth, Malin, Schreckenghost, 1996). The extent to which an intelligent system is perceived as another agent by the user(s) varies and is an important research and design issue. Approach (1) in the above list is closer to the notion of computer-as-other than Approaches (2) or (3). On the one hand, metaphors of human communication influence conceptualizations and design of technological systems for human use (e.g., we speak of dialog design (Gaines and Shaw, 1983; Giachin, 1996) and of computer-generated avatars that speak and emote). On the other hand, anthropomorphizing technological systems raises many issues related to ethics and identity; and furthermore, effective computational support for activity does not necessarily have to be human-like to be useful. Intelligent visualizations that assist in highlighting relevant features of data, for example, can be wonderfully useful without an explicit sense of the software as another agent in the interaction.

There are real dangers in attempting to make systems appear too human-like in cases where in fact they have very limited 'intelligence' and are brittle in their interaction. People have trouble assessing the bounds of the system's capability and this leads to trouble ranging from, at one extreme, over-reliance on the system in cases where it is inappropriate to do so (e.g., Guerlain, et al., 1994; Parasuraman, Molloy, and Singh, 1993), and at the other extreme, loss of 'trust' in the system, and lack of acceptance even in situations where it performs well (e.g., Muir, 1987; Lee and Moray, 1992). Moreover, there are 'human-centered' paradigms that are being advanced that provide alternatives to the development of 'human-like' systems, e.g., 'intelligent' design environments (e.g., Fischer, 1994); intelligent information visualization (Roth et al., 1994); and ecological interface design (e.g., Vicente & Rasmussen, 1992; Pawlak & Vicente, 1996.)

Related to this are issues of automation, authority, power, control, and responsibility: human-centered means in part that humans are an integral part of the problem solving process and not automated out of the system . This is not to say that automation is 'bad', but rather, that the choice of what to automate and how to design the automation itself (whether or not the automation is "intelligent") should take into account human capabilities and limitations, allow humans to override it, and so on (Billings, 1996).

Another aspect of intelligent systems related to collaboration is the notion of intelligent systems that assist people in collaborating (e.g., an "intelligent facilitator" for electronic meetings or "intellect amplifier").

2. General Themes and Issues

Context and evolution are two underlying issues. What is context? How can technology be "contextually engineered"? (For example, gesture and speech recognition are greatly assisted by contextual knowledge.) What are appropriate contextual measures of evaluation? How can technology be adaptive to support changing contexts? What does it mean to model and support the co-evolution of human performance and practice with the possibilities offered by new technologies?

Good technology (functional, reliable, available, etc.) is necessary but not sufficient for effective communication and collaboration among human actors. Instead, joint consideration of technological advances, human use in the context of the domain of practice, and social and organizational aspects is needed.

Supporting effective collaboration in the real world is complex: practice is fluid, dynamic, and consists both of formal work that can be modeled computationally and informal or "invisible" work that may not be modelable but must be possible. As information moves around a distributed cognitive system, it is continually recontextualized. What something "means" is emergent and produced through interaction among human actors. This again emphasizes the importance of flexibility, openness, and adaptiveness in the design of information technology to support collaboration. Yet there are serious design tradeoffs between control and predictability on the one hand and flexibility on the other; the 'right' answer depends on the context of the task at hand.

A focus on context also includes consideration of history; history of the digital contexts of use can be an important part of asynchronous collaboration. Consideration of history and existing infrastructure both in terms of technological systems (legacy systems, archival information), existing work practices and organizational structures, and so on are all part of the context as well.

One particular aspect of collaborative systems is mutual awareness (i.e., knowing who else is here or who else is on the team with you). In particular, the capabilities to represent and support presence ("who is here?"), attentiveness ("what are they doing?"), knowledge ("what do they know?"), affect ("how are they feeling?"), social or organizational position ("what is their status?"), and workload ("how busy are they?") are important for collaborative activity. However, this also raises issues of user control and privacy (e.g., does a person want to make explicit "how I feel" or "how busy I am"?). A flexible notification service embedded in technology can support users in easily expressing chosen attributes on-the-fly.

Cooperative work is not only the delegation of tasks among agents and information sharing; it also has affective, social, and cultural dimensions that cannot be ignored. These dimensions likewise form part of the context of problem solving and collaboration that influences behavior and performance in (presumably) systematic ways. In human communication research, a fundamental tenet is "what is meant is more than what is said". Goffman (1959) distinguished between impressions that are given (the "narrow" view of communication as the words or messages that are said) and those that are given off (in terms of setting, appearance, and manner of the actor and his/her current situation). Similarly in technological projects there is a lot of work on verbal and non-verbal capture, representation, interpretation, and generation that is tied to affective effects and inferences.

A human-centered design philosophy is intimately tied to questions of ethics and values (what is trustworthy technology? how can we design and deploy technology that respects its users? how can we ensure that the human remains the ultimate authority in complex automated systems?). Design can be viewed as a collaborative social process between designers and practitioners, where those boundaries are deliberately blurred; design products can be viewed as a medium for communication between designers and practitioners and as tentative hypotheses that need empirical validation. Yet in being practice-centered we also want to capitalize on new technological advances ("the envisioned world problem") and maybe it is part of the designers' job to communicate these hypotheses in context of practice.

Several high-level principles of human-centered systems include human locus of control, flexibility, openness, adaptiveness, mutual intelligibility. Hence, measures of evaluation are oriented around these constructs in addition to, maybe in place of, speed and cost. For example, in speech recognition, error rates are an important metric, but now, measures oriented around task models and accomplishment of practical activity are critical as well (Oviatt, 1996). As another example, in collaboration technologies we can measure the number or rate of transactions and response time, but new questions arise now in how to measure "meaningful" and "good" collaboration. Another implication is that measures are fundamentally interactive (not 'batch') and evolutionary; hence longitudinal studies (i.e., those that follow the evolution of the system over time) are important.

3. State of the Art

In this section, a brief summary of current research is provided. This discussion is organized around several fundamental components of collaboration: sharing (and filtering) information, coordination of activity, communication, awareness of others, and more broadly, building communities. Next, human interaction with intelligent systems is discussed as a related but separate tradition. Finally, current research in computational modeling of users and organizations, design and evaluation issues in collaborative systems, and issues oriented around several technical foci (security, networking and operating system issues, and rich multi-modal interaction) are discussed as well.

One organizing framework for discussion through all these issues is the "Virtual Ad-Hoc Team" for knowledge work. This means that in complex dynamic environments, one needs to put together a team quickly to address a particular issue. Examples include military teams for crisis management; tiger teams in business; and task forces for education. By definition, the team is formed dynamically, has a particular purpose and 'lifespan', and may not be composed of people in the same geographic space but must have resources and infrastructure for remote collaboration. Many complex issues surround this kind of scenario: How do we know who is available for our team? How do we choose the best people? In fact, how do we define 'best'? How do we structure the team? How can we facilitate 'rapid socialization' in these contexts? How do we preserve and reuse organizational memory when our organization is so transient? How do we cope with very different heterogeneous knowledge and skills and technological infrastructure among team members?

3.1. Sharing and Filtering Information

The notion of joint creation of a shared information space is the basis for a great deal of work in CSCW. People can share information by talk (face-to-face, via telephone, electronic chat facilities, electronic mail) and by collaborative drawing and writing with mechanisms as varied as publishing on the Web, electronic mailing of documents, document management systems, and shared electronic whiteboards and synchronous collaborative writing tools. With the vast amount of information available, benefits of allowing all to have a voice are balanced by costs of one's own need to search and filter information for relevance. Search engine technologies and information retrieval systems exist and are fairly good. Much current work is being done on indexing schemes, concept searches, and content-based retrieval. One current wave of work is a blend of social systems and information retrieval -- the notion of 'recommender systems' or collaborative filtering in which one discovers information based on the recommendations of others (e.g., March 1997 issue of Communications of the ACM, including Resnick and Varian, 1997; Terveen et al., 1997; Kautz, Selman, and Shah, 1997; Balabanovic and Shoham, 1997; and Konstan et al., 1997).

3.2. Coordination of Activity

Another part of collaboration is coordinated activity, which relates to issues of redirection of attention, allocation of tasks, "knowing who is doing what when", planning, and articulation work (Schmidt, 1994). Coordination theory has focused on the varieties of interdependencies among activity (Malone and Crowston, 1990). Related work has proposed the Process Interchange Format (PIF) as a standard for sharing data about coordinated processes. Indeed, one prominent overarching metaphor for the modeling and analysis of cooperative work views an organization as a distributed information processing system (Morgan, 1986; also see Jones and Jasek, 1997). Common "conceptual primitives" that are represented computationally include goals, activities, actors or agents, resources, decisions, constraints, rationale, and data that can be analyzed at various levels of abstraction (Jones and Jasek, 1997). Research in the context of this metaphor emphasizes formal modeling and reasoning algorithms and performance measures of consistency, efficiency, and correctness with respect to system goals.

In contrast to formal modeling approaches, a 'sociocultural view' focuses on how shared meanings emerge in practice. For example, Geertz (1973) views culture as "essentially a semiotic [concept]" and its analysis as "not an experimental science in search of law but an interpretive one in search of meaning [that consists of] sorting out the structures of significance ... and determining their social ground and import" (Geertz, 1973, p. 5 and 9). Suchman's well-known critique of the "strong" view of planning and the concomitant view of action as inherently situated in the local context of particular material and social circumstances similarly results in an emphasis on mutual intelligibility in context (Suchman, 1987). Issues of responsibility, authority, power, status, and "visible" versus "invisible" work have been of particular interest to other researchers (cf. Gerson and Star, 1986). Greenbaum and Kyng's (1991) view of the cooperative approach to computer systems design emphasizes situations, breakdowns, tacit knowledge, and group work in contrast to a traditional software development focus on tasks, explicit knowledge, formalization, and individual work. Another aspect of the social/cultural view is a focus on management, labor and industrial relations, and the like (cf. Schmidt, 1990).

In summary, cooperative work can be studied from both an "engineering" and "social science" perspective, and these perspectives are complementary and mutually beneficial to the understanding of organizational systems. Thus, a well-rounded analysis of cooperative work should include the articulation of the formal structures and mechanisms of interaction as well as the local contingencies that emerge in practice.

With respect to technical infrastructure for coordinated activity, three basic frameworks are conflict prevention (don't let interdependent activities clash), conflict management (let things clash and provide support for sorting things out), and process enactment/workflow. Current work in conflict prevention supports pessimistic serializable transactions and shared views of others' actions ("what you see is what I see"). More recent work is in flexible transactions (e.g., reflective transactions). Current work in conflict management includes versioning (e.g., as in Lotus Notes™), semi-automatic merging (e.g., CODA), and simple "diffing" (e.g., PREP editor). More recent work looks at more automated merging and how to provide users with increased control and more natural and expressive ways to handle conflict management. Finally, current workflow technologies allow users to express activities or tasks, the nature of their dependencies, their time constraints (duration, deadlines, etc.), and assignment of tasks to individuals or teams. Relevant examples include Microsoft Project™, Coordinator™, and USA CERL's KnowledgeWorker™. These rely on explicit representations of activity and action sequencing. More recent work looks at more flexible and malleable representations of activity, process augmentation, and exception handling.

3.3. Communication

Communication is fundamental in many senses: network protocols for data communication over computer networks, social protocols (practices) for polite communication in face-to-face interaction, and user interface design in human-computer communication to mention just three ways of framing "communication".

With respect to human communication, McCarthy et al. (1990) consider four "generic" communication tasks that must be supported in any communication system: synchronization, coherence, repair, and shared focus. That is, synchronous talk needs to be synchronized (people need to take turns in conversation); effective communication is coherent, "makes sense", "follows"; people need to be able to engage in repair when communicative breakdowns occur (e.g., by engaging in metatalk or alignment talk); and people need to be able to express and identify shared focus of attention. Human communication research has for years studied natural conversation to model how conversational turns or 'moves' are constructed, analyzed conversational coherence, inventoried a variety of repair strategies, and looked at issues of reference, redirection of attention, and the like (e.g., Haslett, 1988).

In the context of the effects of technology on communicative practices between people, McGrath and Hollingshead (1993) provide a good summary of work on empirical findings and modeling efforts. More recently, computational organizational models and theories have been applied to these problems (see Section 3.7).

Social network and organizational communication theorists have evolved a vast range of theories to account for social behavior. Monge and Contractor (in press) identify eleven classes of generative mechanisms or underlying logics that explain the manner in which networks enable and constrain social attitudes and behavior in general. These include: (1) exchange and dependency theories (social exchange and resource dependency), (2) contagion theories (social information processing, social learning theory, institutional theory, structural theory of action), (3) cognitive theories (semantic networks, cognitive social structures), (4) consistency theories (balance theory, theory of cognitive dissonance), (5) theories of homophily (social comparison theory, social identity theory), (6) theories of social capital (theory of structural holes, strength of weak ties theory), (7) theories of proximity (physical and electronic proximity), (8) uncertainty reduction theories, (9) social support theories, (10) collective action theories, and (11) theories of network and organizational forms (contingency theory, transaction cost theory, and theories of network organizations).

3.4. Awareness of Others, Team Membership, Organizational Knowledge

Part of collaboration and communication involves awareness of others. A great deal of research in social psychology, sociology, and human communication research has looked at topics such as how first impressions are formed, what kinds of inferences are made from manner, appearance, talk, and setting (Goffman, 1959), social status and hierarchy, and so on. Identity theory, social presence theory, and other positions exist in this arena.

In terms of technology support for awareness of others, anonymity versus identity has been a critical research issue. Experimental work has looked explicitly at how anonymity affects group communication and problem solving (e.g., see McGrath and Hollingshead's summary). Collaboration technologies have made a variety of assumptions about anonymity versus identity: some technologies let participants set this as a feature; some technologies build in identity but allow participants to construct this very flexibly (e.g., in MUDs and MOOs; Turkle, 1995). Video teleconferencing, "porthole" systems (e.g., Xerox, Nynex), and the like show "one's real self" over a video link. Thus, the notion of identity is an important design consideration in collaborative systems; tradeoffs exist between allowing participants to flexibly construct one or more identities versus authenticating "real people" for security and other reasons. This is an issue also with respect to rapid socialization of new team members.

Related to the issue of knowing "who is here" or who is a member of the team or organization are questions about the diffusion of knowledge and organizational memory. Where does knowledge reside in the sociotechnical system, how is it organized, and how can it be retrieved and used effectively? Requirements, specification, and implementation of collaboration policies with respect to persistence, security, notification, concurrency control, and merging will affect how organizations construct and use knowledge.

Thus, the issue of sharing ontologies is critical. Two basic computational approaches to sharing knowledge are (1) agreeing on a common ontology and (2) translating between different ontologies (as in much work at the Stanford Knowledge Systems Laboratory (http://www-ksl.stanford.edu/)). As Davenport (1994) points out, an inevitable tension exists between the global (all participants have to express themselves in a common language which may not be optimal for their local problem solving needs) and the local (different sub-groups have their own local representations tailored for their own use, but these are not necessarily shared with other groups). Furthermore, to support asynchronous collaboration and organizational learning, the capture, representation, and reuse of the contexts of use of information and knowledge is another critical area of research.

The previous paragraph presumes that 'knowledge' is about work tasks, organizational roles, and the like. Another facet of knowledge that is usually neglected is knowledge of the affective states of other participants. Here, questions of how to recognize the affective states of users, how to understand affect in different situational contexts, and how affect is conveyed to others.

3.5. Fostering Communities

Another aspect of collaboration technology is the potential for the building and sustainment of communities. Technologies such as newsgroups, email, MUDS, and MOOs offer forums for interaction that have been subject to a variety of praise and criticism (e.g., Rheingold, 1993; Jones, 1995; Turkle, 1995).

3.6. Intelligent Systems, Intelligent Software Agents, Human Interaction

with Intelligent Systems

Current empirical studies and theory, design, and evaluation of intelligent systems have focused on notions such as cooperative problem solving, mixed-initiative interaction, and mutual intelligibility (e.g., Roth, Jones, Woods). Rather than black-boxing authority into an intelligent system and forcing the person to act as a data gatherer and solution filter for the system, instead we should design joint cognitive systems in which humans have clear authority, can intervene flexibly and appropriately, and are engaged actively in problem solving (Woods, 1986; Roth). Current prototype systems that work in these ways include systems that provide tools for on-the-fly replanning, interactive visualizations of an activity model as a resource for action, and the like (Roth, Malin, Schrekenghost, 1996).

With respect to degrees of automation of group support, a tension exists between routine, procedural, formal things that it may be desirable to automate versus the novel, improvisational, informal which is not amenable to automation. This related to workflow technologies described previously: we would like to capitalize on the routine but not rigidify practice or make necessary 'invisible work' impossible. Current approaches to this include both the design of more flexible representations and the design of the interaction paradigm itself to be "a system on the side" rather than "the system that is a user's only interface".

3.7. Computational Modeling of Users and Organizations

"User modeling" has been a topic of research for many years; indeed there is an entire academic journal devoted to this issue. Researchers on intelligent user interfaces and intelligent tutoring systems have modeled users in terms of current goals and activities, interests (in particular objects in the world), preferences for information displays, and the like. Student modeling in intelligent tutoring systems have been of two basic varieties: overlay models in which student knowledge is presumed to be a subset of domain expert knowledge, and buggy models in which student knowledge is represented as systematic deviations from expert knowledge.

More recently, computational organizational modeling has come into its own as an area of research (Hanneman, 1988) and there is now an entire journal devoted to it as well (Computational and Mathematical Organizational Theory). For example, Contractor and Seibold (1993) propose self-organizing systems theory as an approach to modeling organizational adaptation to and with technology. This emergent perspective assumes that the uses and effects of communication technology emerge from complex social interactions among users (Contractor and Eisenberg, 1990; Contractor and Seibold, 1993; Contractor, 1994). In particular, models of the evolution of activity represent the articulation of reciprocal and dynamic relationships among social norms, affordances provided by technologies, and actors' roles (Contractor and Eisenberg, 1990).

3.8. Design and Evaluation of Collaborative Systems

The most effective design strategy for achieving human-centered systems is to use an iterative approach with an empirical feedback loop (Landauer, 1995). This means that valid, reliable measures of the human use of systems are needed that can be incorporated into this process. But there are many issues about evaluation itself that are in need of further research.

There are many reasons why the evaluation of collaborative systems is difficult. First, there are multiple levels of analysis of such systems, such as individual, group, organization, and industry. It is well-known that improving productivity at one level of analysis (e.g., individuals) does not necessarily mean that it will be improved at another level of analysis (group, organization) (Harris, 1994). A second problem is that many important and substantive effects are long-term. For instance, while changes can be shown in the productivity of an individual worker in a few hours or days of using a new system, productivity effects for an organization or an industry may take months or years to appear. Indeed, such lag effects are one of the possible explanations of the so-called "productivity paradox" (Brynjolfsson, 1993). Third, many of the effects that are most determinate of performance at any level of aggregation are embodied as cognitive skills (Anderson, 1982) or organizational routines (Cohen & Bacdayan, 1994). Knowledge in this form is less accessible and therefore more difficult to extract and analyze. Fourth, human systems are notoriously reactive to the introduction of new information tools (Sproull & Kiesler, 1991; DeSanctis & Poole, 1994). Changing the information environment changes what people do, and as a result measures which made sense under the old environment may not make sense in the new one. This is especially true of the kinds of efficiency measures that are most often the focus of information technology interventions (Sproull & Kiesler, 1991).

It is also well-known that different evaluation measures may show different things. For instance, in evaluations of various kinds of group technology it has often been found that the technologies improve task performance but lead to longer times to task completion and decrease user satisfaction (Olson & Olson, 1997). In other words, there are trade-offs in performance that need to be taken into account in any overall evaluation of the effectiveness of the technology.

There is no shortage of kinds of things that can be measured. But often researchers investigating information systems are ignorant of the large methodological literatures that can guide the design, collection, and analysis of evaluations (see Olson & Olson, 1997, for details). For instance, questionnaires and interviews must be designed with great care to avoid well-understood framing effects.

As the level of aggregation of the collaborative system increases the cost of doing a careful evaluation goes up. Groups are more difficult to evaluate than individuals, organizations more difficult than groups, and so forth. Indeed, ecologically valid assessments of group or organizational systems require full-scale deployments or testbeds. A high priority research need is to develop cost-effective measures that are appropriate for meaningful group and organizational outcomes. Another priority is for studies that are themselves collaborative, where different research groups accumulate cases using agreed-upon benchmarks so that definitive results can be obtained in reasonable time periods. Such collaborative studies are widespread in the biological and health sciences.

Careful empirical evaluation of systems in terms of processes and outcomes defined by human purposes need to become common elements of development projects. A brief scan of the journal and conference literature shows that such evaluations are very rare. As a result, numerous demonstration systems exist with little collective understanding of how their characteristics relate to important human-centered outcomes.

3.9. Security

Issues related to security include access control, methods for encryption, and authentication. Current technologies include operating systems such as UNIX which support logins and passwords and allow owners of files to specify read, write, and execute permission for themselves, groups, and others; public key methods for cryptography, and Kerberos for authentication.

3.10. Networking and Operating Systems Issues in Distributed Collaboration

Many technical issues arise in the context of providing flexible collaboration support for multiple users. Coping with heterogeneous resources (transmodal, multimedia) is a critical issue; e.g., automatic presentation of information that takes into account the hardware/output device of the receiver. Importantly, the ways in which information is presented at lower resolutions and so on must preserve the important aspects of the context to be useful. Policies and algorithms for such information sharing are also a key research area (e.g., pre- or post-filtering of data to accomodate different devices, resolutions, etc.).

Many quality of service issues arise in hardware, networking, and operating systems. Shared intelligence among these layers is important. The "end to end quality of service parameters" must be clearly articulated and appropriate tradeoffs invoked based on the context of use.

Other issues include policies and algorithsm for notification, priorities, goals, and intermedia translation. Interoperability of software tools is likewise critical (e.g., we each use of own text editors but are able to easily share things) in order to preserve local competence in individual tools but be able to seamlessly share information.

An emerging distinction is collaboration-aware versus collaboration-transparent applications. Collaboration-aware applications explicitly monitor and represent multiple users. Collaboration-transparent applications do not explicitly represent multiple users (are "collaboration-oblivious") but require collaboration-aware environments in order to have sharing take place. A related issue is sharing windows versus sharing screens on the desktop.

A number of distributed networking issues also arise (e.g., migration, replication). For example, in the context of the "Virtual Ad-Hoc Team", if team leadership is dynamic, then a potential implication is that the application should migrate to the leader's desktop because he/she requires the fastest and best interaction.

3.11. Rich Multi-Modal Interaction and Tele-Immersive Collaborative Virtual Environments

Current tele-immersion technologies support fairly good visual resolution, some auditory and haptic input and output. Examples are the projection-based virtual reality displays (e.g., the CAVE) and head-mounted displays. A method and interface has been developed by Carlos Ricci (NCSA) for navigating virtual spaces and manipulating virtual objects in the CAVE through natural walking and leaning motions. A sensing device, which is strapped to the user's shoes, translates foot pressure into signals whose patterns are recognizable in the host graphics computer system. The system is considered to be more natural, more interactive, and less confining than the traditional treadmill or stepper type of walking interface. A software driver has been written to identify natural walking patterns and derive from them a velocity value which, in turn, may be integrated with CAVE applications as a control parameter. Pattern recognition in the driver was implemented using fast-executing artificial intelligence methods. The driver is expandable to identify other dynamic patterns, such as those associated with "mime" or "in-place" walking, or static patterns, such as leaning, and infer corresponding control parameters from each of these.

Current CRT technology seems limited by market forces and development to 2048x2048 pixels. LCD screen sizes and resolutions seem driven by market needs for laptop computers. In terms of haptic devices, keyboards and mice injure without the help of force feedback; devices capable of providing substantial feedback could do real injury. Some heavy earth-moving equipment designs are now fly-by-wire; force feedback is being simulated to give the operator the feel once transmitted by mechanical linkage.

In terms of I/O device connectivity, currently the PC-clone is the universal I/O adapter because of its open architecture and the availability of cheap mail-order I/O devices, but a stack of PC's each doing one filtering task, trying to communicate with one another on serial lines is not directly adaptable to the ECI need set. Custom chip sets will drive the cost down to consumer level; adapting video game I/O devices where possible will help in achieving similar price performance improvements as computing itself.

Vibrafloor is a virtual audio real-time experience. Participants can interact with and create unique soundscapes in 3D space using a computer-controlled head and hand tracking device. The virtual audio environment produces 3D localized sounds in 4-channel surround-sound creating a totally immersive audio environment enabling participants to "forget themselves" while standing, sitting, or lying down on the sonic wave floor. Participants can control what sounds are played, placed, mixed, and/or composed. The sonic wave floor consists of tactile sound transducers that provide sounds that participants not only hear, but feel. The sonic wave floor is carpeted, allowing attendees to sit, lie, or stand while receiving some degree of audio-induced "messages."

The literature on empirical, generalizable evaluation of human-computer systems with spoken-dialogue and multimodal interfaces is sparse. The basic issue of how to evaluate spoken-dialogue systems effectively is still unresolved and requires further research (Danieli and Gerbino, 1995; Sparck-Jones and Galliers, 1992; Walker, Litman, Kamm, and Abella, 1997).

4. Further Future Directions

An immense array of issues confront the researcher in collaboration and communication technologies. The previous section has indicated current areas of work and some promising future directions. Overarching issues include (1) coping with context: how can contexts be systematized, formally modeled, and used in principled ways to design technologies and (2) linking social science to technology: new languages, computational methods, ways of embedding semantics/context/meaning from social science theory into technology design. Human-centered systems may be seen as a new field that addresses these issues.

4.1. Sharing Information in Context

While existing types of computer-mediated communication (CMC) systems permit the conversational interaction of two or more parties, the enabled interactions lack any computational support. On the other hand, artificial intelligence and other types of computer systems provide only rudimentary collaborative capabilities, compared to the rich and nuanced interactions among human actors. (For example, access control mechanisms for information are primitive compared to how people weigh when and to whom to release sensitive personal or organizational information.) Thus, we are caught between CMC systems with little augmentation of collaboration (although with a rich set of conversational and social mechanisms) and traditional AI or CS systems with little nuance to their collaborative interactions (but with strong augmentation capabilities). What is needed is to bridge the two, providing the nuance of normal social interactions as well as the computational augmentation of collaborative interactions. Both are necessary in a true synthesis of capabilities if we wish to construct human-centered systems that help and aid human capabilities.

This task requires basic research into:

The nature of human collaboration in real settings. We need to know more about what people actually do, and we need to know this in the context of system construction (i.e., in order to properly understand the requirements of future systems). We also need to develop lightweight methods of obtaining the social requirements for commercial and organizational applications.

How to bridge the gap between providing human communication capabilities and our capability to emulate human activities in systems. Research needs to be done on how to provide nuanced and contextualized activity through computational systems or how to provide approximations that are workable for the humans involved. This research might be done in the context of information access, information retrieval, access control, security, privacy, or other parts of collaborative or user activity.

What type of augmentations to social interactions and the social networks of people would be useful (and doable). These augmentations might include providing better access to other people on an expertise network, providing the right information (formal or informal) on demand, finding others with whom to have a collaboration, and so on.

4.2. Tele-Immersion and Collaborative Virtual Environments

The goal of "Tele-Immersion" research is to extend the "human/computer interaction" paradigm to "human/computer/human collaboration," with the computer providing real-time data in shared, collaborative environments, to enable interaction between human actors (the "tele-conferencing" paradigm) as well as computational models, over distance; and to provide easy access to integrated heterogeneous distributed computing environments, whether supercomputers, remote instrumentation, networks, or mass storage devices using advanced real-time 3D immersive interfaces.

Part of the research agenda in this arena is to focus on issues of networking (to stress-test network bandwidth and latency), data distribution, computational heterogeneity, next-generation graphics engines (e.g., increase polygons per second; real-time volume visualization), and effective human-computer interfaces and support for human sharing and collaboration.

An important evaluation criteria is that these systems be sought after and used regardless of distance; if they are discretionary (i.e., users can choose to use the technologies or not) and do become absorbed into collaborative practices, then that is one measure of success.

Several specific research agenda items are as follows:

Image resolution to match human visual capabilities. In particular, provide enough anti-aliased image resolution to match human vision (minimally 5000x5000 pixels at >50Hz). 20-20 vision is roughly 5000 pixels (at 90-degree angle of view), less is needed at the angle we normally view television or workstation screens, more for wide-angle VR applications. A reasonable benchmark is 8000 pixels (the size of a typical magazine advertisement). More resolution can be used to facilitate simple panning or zooming, both of which can be digitally realized with processing and memory without requiring more resolution out of the display device. Certain quality enhancements may be achieved with higher refresh rates (e.g. 100Hz) including less strobing during panning or the capability of doing stereo visuals by sending two 50Hz images, one for each eye. Low latency, not currently a feature of LCD displays, is needed for 100Hz or greater devices. Micromirror projectors show promise in this area. Desirable, of course, would be wall-sized screens with very high resolution (>20,000 pixels) whose fidelity would be matched to our vision even when closely examined. Multiple projectors tiled together may achieve such an effect where warranted; monitors and LCD screens do not lend themselves to tiling because the borders around the individual displays do not allow seamless configurations. Truly borderless, flat displays are desirable as a way to build truly high-resolution displays.

Universal and safe input devices that capitalize on human capabilities for vocal and motor output. This requires a comprehensive compilation of existing bio-engineering and medical published research on human performance measurement techniques, filtering for the instrumentation modalities that the human subjects can use to willfully generate continuous or binary output. Modalities should be ranked according to quality/repeatability of output, comfort, intrusiveness, cost, durability, portability, and power consumption. Note that much is known about human input capacity, by contrast.

In the development of haptic input support, a critical issue is safe force feedback devices capable of delivering fine touch sensations under computer control. Development of fail-safe mechanisms and fundamental advances in hardware and software technology are needed.

The development of universal methods for I/O device connectivity is another key component of infrastructure for tele-immersion. "Plug and play" open architectures and standards are needed.

Research is needed to understand how (and in what contexts) humans combine speech and gesture to communicate effectively in multi-modal environments (Bellik, 1996; Cohen & Oviatt, 1994).

Audio output matched to the dynamic range of human hearing. Digital sound synthesis is in its infancy. Given the speed of currently available high-end microprocessors, advances are sought in software tools and in creating of contextual "soundscapes" .

Metaphors and navigation. This is akin to understanding the functional transitions in moving around within the WIMP desktop metaphor. What are appropriate metaphors for the design of virtual environments (e.g., shopping mall)? And how does the choice of metaphor interact with navigation possibilities? Directional surround-sound audio and tactile feedback rich enough to assist a vision-impaired person in navigation would also likely help a fully sighted person. Develop schematic means to display the shopping mall metaphor on conventional desktop computers, small video projectors, and embedded displays.

 Storage and retrieval of visualization/VR sessions. One would like to play back and edit visualization/VR sessions in ways akin to the revision mode in word processors. A key technology development here is the extension of the video-server concept to visualization/VR capture and playback.

5. Summary

In this section, we offer a few programmatic suggestions for the National Science Foundation to follow for human-centered systems research. These may be summarized as follows:

Establishment of one or more Human-Centered Systems Collaboratories to foster multidisciplinary collaboration.

Encouragement of multidisciplinary teams of technologists, social scientists, and practitioners. This includes aspects of the nature of the research project and educational and outreach activities.

 The development of relevant, contextual measures of evaluation is critical.

Evaluation should be longitudinal and ongoing (rather than simply a 'post-test' at the end of the project).

 Routine duration for projects should be at least five years, not the current three. The time necessary to analyze practice, construct design artifacts, and do longitudinal evaluation demands longer-term efforts.

6. Bibliography

Alty, J. L. and Coombs, M. J. (1980). Face-to-face guidance of university computer users-I: A study of advisory services. International Journal of Man-Machine Studies, 12, 390-406.

Anderson, J.R. (1982) Acquisition of cognitive skill. Psychological Review, 89, 369-406.

Baecker, R. (Ed). (1993). Readings in groupware and computer-supported cooperative work. Morgan Kaufman.

Balabanovic, M. and Shoham, Y. (1997). Fab: Content-based, collaborative recommendation. Communications of the ACM, 40, 3, March 1997, 66-72.

Bellik, Y. (1996). Modality integration: Speech and gesture. In R. A. Cole, J. Mariani, H. Uszkoreit, A. Zaenen, and V. Zue (Eds.), Survey of the state of the art in human language technology. http://www.cse.ogi.edu/CSLU/HLTsurvey/HLTsurvey.html

Billings, C. E. (1991). Human-centered aircraft automation: A concept and guidelines (NASA Technical Memorandum 103885). Moffett Field, CA: Ames Research Center.

Bond, A. H. and Gasser, L. (1988). An analysis of problems and research in DAI. In A. H. Bond and L. Gasser (Eds.), Readings in distributed artificial intelligence. San Mateo, CA: Morgan Kaufmann.

Bourdieu, P. (1990). The logic of practice (translated by Richard Nice).Stanford University Press.

Boy, G. A. (1987). Operator assistant systems. International Journal of Man-Machine Studies, 27, 541-554.

Brown, J. S. (1986). From cognitive to social ergonomics and beyond. In D. Norman and S. Draper (Eds.), User centered system design (457-486). Hillsdale, NJ: Lawrence Erlbaum Associates.

Brynjolfsson, E. (1993) The productivity paradox of information technology. Communications of the ACM, 36(12), 67-77.

Burt, R. S. (1982). Toward a structural theory of action: Network models of stratification, perception and action. New York: Academic Press.

Burt, R. S. (1987). Social contagion and innovation: Cohesion versus structural equivalence. American Journal of Sociology, 92, 1287-1335.

Carley, K. and Prietula, M. (Eds.) (1994). Computational organization theory. Hillsdale NJ: Lawrence Erlbaum Associates.

Clark, H. H. and Brennan, S. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, and S. D. Teasley (Eds.) Perspectives on socially shared cognition (pp. 127-149). Washington DC.: American Psychological Association.

Cohen, M.D., & Bacdayan, P. (1994) Organizational routines are stored as procedural memory: Evidence from a laboratory study. Organizational Science, 5, 554-568.

Cohen, P. R. and Oviatt, S. L. (1994). The role of voice in human-machine communication. In D. B. Roe and J. Wilpon (Eds.), Voice communication between humans and machines, 34-75. National Academy of Sciences Press.

Cole, R., Mariani, J., Uszkoreit, H., Zaenen, A., and Zue, V. (1996). Survey of the state of the art in human language technology. Research report sponsored by the National Science Foundation and European Commission. http://www.cse.ogi.edu/CSLU/HLTsurvey/HLTsurvey.html

Connolly, T., & Thorn, B.K. (1990). Discretionary databases: Theory, data, and implications. In J. Fulk & C. Steinfield (Eds.), Organizations and communication technology (pp. 219-233). Newbury Park, CA: Sage.

Contractor, N. S. (1994). Self-organizing systems perspective in the study of organizational communication. In B. Kovacic (Ed.), New approaches to organizational communication (pp. 39-66). Albany, NY: SUNY Press.

Contractor, N. S., & Eisenberg, E. M. (1990). Communication networks and new media in organizations. In J. Fulk & C.W. Steinfield (Eds.), Organizations and communication technology (pp. 143-172). Newbury park, CA: Sage.

Contractor, N. S., & Grant, S. (1996). The emergence of shared interpretations in organizations: A self-organizing systems perspective. In J. H. Watt & C. A. VanLear (eds.), Dynamic patterns in communication processes. Newbury Park, CA: Sage.

Contractor, N. S., & Seibold, D. R. (1993). Theoretical frameworks for the study of structuring processes in group decision support systems: Adaptive structuration theory and self-organizing systems theory. Human Communication Research, 19, 528-563.

Danieli, M. and Gerbino, E. (1995). Metrics for evaluating dialogue strategies in a spoke language system. Proceedings of the 1995 AAAI Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, 34-39.

DeSanctis, G., & Poole, M.S. (1994) Capturing the complexity in advanced technology use: Adaptive structuration theory. Organizational Science, 5, 121-147.

Fischer, G. (1994). Domain-Oriented Design Environments. Automated Software Engineering, 1, 177-203.

Fischer, G., Lemke, A. C., Mastaglio, T., and Morch, A. I. (1991). The role of critiquing in cooperative problem solving. ACM Transactions on Information Systems, 9, 123-151.

Giachin, E. (1996). Spoken Language Dialogue. In R. A. Cole, J. Mariani, H. Uszkoreit,

A. Zaenen, and V. Zue (eds.), Survey of the State of the Art in Human

Language Technology. (http://www.cse.ogi.edu/CSLU/HLTsurvey/HLTsurvey.html).

Goffman, E. (1974). Frame analysis: An essay on the organization of experience. New York: Harper and Row.

Goffman, E. (1959). The presentation of self in everyday life. New York: Anchor / Doubleday.

Goody , E. (Ed.) (1995). Social intelligence and interaction . Cambridge: Cambridge University Press.

Guerlain, S., Smith, P. J., Gross, S. M., Miller, T. E., Smith, J. W., Svirbely, J. R., Rudmann, S., & Strohm, P. (1994). Critiquing vs. partial automation: How the role of the computer affects human-coomputer cooperative problem solving. In M. Mouloua & R. Parasuraman (Eds.), Human performance in automated systems: Current research and trends (p. 73-80). Hillsdale, New Jersey: LEA.

Hanneman, R. A. (1988). Computer-assisted theory building: Modeling dynamic social systems. Newbury Park, CA: Sage.

Harris, D.H. (1994) Productivity linkages in computer-aided design. In D.H. Harris (Ed.), Organizational linkages: Understanding the productivity paradox. Washington: National Academy Press.

Haslett, B. J. (1987). Communication: Strategic action in context. Hillsdale, NJ: Lawrence Erlbaum.

Hutchins, E. (1995). Cognition in the wild. Cambridge, MA: The MIT Press.

Hyatt, A., Contractor, N. and Jones, P. M. (1996). Computational organizational network modeling: Strategies and an example. To appear in Computational and Mathematical Organizational Theory.

Jones, P. M. (1995). Cooperative work in mission operations: Analysis and implications for computer support. Computer-Supported Cooperative Work, 3, 1, 103-145.

Jones, P. M. and Jasek, C. A. (1997). Intelligent support for activity management (ISAM): An architecture to support distributed supervisory control. IEEE Transactions on Systems, Man, and Cybernetics, 27, No. 5, 274-288.

Jones, P. M., and Mitchell, C. M. (1995). Human-computer cooperative problem solving: Theory, design and evaluation of an intelligent associate system. IEEE Transactions on Systems, Man, and Cybernetics, 25, 1039-1053.

Jones, S. G. (Ed.) (1995). Cybersociety: Computer-mediated communication and community. Sage Publications.

Kamm, C., Walker, M. and Rabiner, L. (1997). The role of speech processing in human-computer intelligent communication. Report prepared for NSF Workshop on Human-Centered Systems, February 1997.

Kautz, H., Selman, B., and Shah, M. (1997). ReferralWeb: Combining social networks and collaborative filtering. Communications of the ACM, 40, 3, March 1997, 63-65.

Klein, G., Orasanu, J., Calderwood, R. and Zsambok, C. (Eds.) (1993). Decision making in action: Models and methods. Academic Press.

Konstan, J. A., Miller, B. N., Maltz, D., Herlocker, J. L., Gordon, L. R., and Riedl, J. (1997). GroupLens: Applying collaborative filtering to Usenet news. Communications of the ACM, 40, 3, March 1997, 77-87.

Landauer, T.K. (1995) The trouble with computers: Usefulness, usability and productivity. Cambridge, MA: MIT Press.

Larson, C. E. and LaFasto, F. M. J. (1989). Teamwork: What must go right/What can go wrong. Newbury Park, CA: Sage.

Layton, C. Smith, P. J. and McCoy, E. (1994). Design of a cooperative problem-solving system for en-route flight planning: An empirical evaluation. Human Factors, 36, 94-119.

Lee, J., & Moray, N. (1992). Trust, control, strategies and allocation of function in human-machine systems. Ergonomics, 35, 1243-1270.

Malin, J. T., Schreckenghost, D. L., Woods, D. D., Potter, S. S., Johannesen, L., Holloway, M., and Forbus. K. D. (1991). Making Intelligent Systems Team Players: Case Studies and Design Issues. Vol 1: Human-computer Interaction Design; Vol 2: Fault Management System cases. (NASA Technical Memorandum 104738). Houston, TX: NASA Johnson Space Center.

Malone, T. and Crowston, K. (1990). What is coordination theory and how can it help design cooperative work systems? Proceedings of the 1990 ACM Conference on Computer-Supported Cooperative Work, 357-370.

Mayberry, Mark (Ed.) (1993). Intelligent multimedia interfaces. AAAI/MIT Press.

McCarthy, J. C., Miles, V. C., Monk, A. F., Harrison, M. D., Dix, A. J. and Wright, P. C. (1991). Four generic communication tasks which must be supported in electronic conferencing. Departments of Psychology and Computer Science, University of York, U. K. ACM SIGCHI Bulletin, January 1991.

McGrath, J. and Hollingshead, A. (1993). Groups interacting with technology. Sage.

Monge, P. R., & Contractor, N. S. (in press). Emergence of communication networks. In F. Jablin & L. Putnam (Eds.), Handbook of Organizational Communication (2nd edition). Newbury Park, CA: Sage.

Morgan, G. (1986). Images of organization. Sage.

Morningstar, C. and Farmer, F. R. (1991). The lessons of Lucasfilm's Habitat. In M. Benedikt (Ed.), Cyberspace: First steps. Cambridge MA: MIT Press.

Muir, B. (1987). Trust between humans and machines, and the design of decision aids. International Journal of Man-Machine Studies, 27, 527-539.

Oberquelle, H., Kupka, I. and Maass, S. (1983). A view of human-machine communication and co-operation. International Journal of Man-Machine Studies, 19, 309-333.

Olson, G.M., & Olson, J.S. (1997). Research on computer supported cooperative work. In M.G. Helander, T.K. Landauer, & P. Prabhu, (Eds.). Handbook of Human-Computer Interaction (2nd Ed). Amsterdam: Elsevier.

Oviatt, S. (1996). Usability and Interface Design. In R. A. Cole, J. Mariani, H.

Uszkoreit, A. Zaenen, and V. Zue (eds.), Survey of the State of the Art in Human

Language Technology. (http://www.cse.ogi.edu/CSLU/HLTsurvey/HLTsurvey.html).

Parasuraman, R., Mollow, R. and Singh, I (1993). Performance consequences of automation-induced 'complacency'. International Journal of Aviation Psychology, 3, 1-23.

Pawklak, W. S. & Vicente, K. J. (1996). Inducing effective operator control through ecological interface design. Int. Journal of Human-Computer Studies, 44, 653-688.

Picard, R. W. (1996). Affective computing. MIT Media Lab Technical Report 321, Massachusetts Institute of Technology.

Poole, M. S., & DeSanctis, G. (1990). Understanding the use of group decision support systems: The theory of adaptive structuration. In C. Steinfield & J. Fulk (Eds.), Organizations and communication technology (pp. 175-195). Newbury Park, CA: Sage.

Reichman, R. (1985). Getting computers to talk like you and me. Cambridge, MA: MIT Press.

Resnick, P. and Varian, H. (1997). Introduction to the Special Section on Recommender Systems. Communications of the ACM, 40, 3, March 1997, 56-58.

Rheingold, H. (1993). The virtual community: Homesteading on the electronic frontier. Reading MA: Addison-Wesley.

Roth, E. M., Bennett, K. B., and Woods, D. D. (1987). Human interaction with an "intelligent" machine. International Journal of Man-Machine Studies, 27, 479-525.

Roth, E. M., Mumaw, R. J. and Lewis, P.M. (1994). An empirical investigation of operator performance in cognitively demanding simulated emergencies. (NUREG/CR-6208). Washington D. C.: U. S. Nuclear Regulatory Commission.

Roth, E. M., Malin, J. T., and Schreckenghost, D. L. (1996). Paradigms for intelligent interface design.

Roth, S. F., Kolojejchick, J., Mattis, J. and Goldstein, J. (1994). Interactive graphic design using automatic presentation knowledge. In Human Factors in Computing Systems CHI'94 Conference Proceedings (pp. 112-117). New York, NY: ACM/SIGCHI.

Schmidt, K. (1994). Modes and mechanisms for cooperative work. Riso Technical Report, Riso National Laboratory, Roskilde, Denmark.

Schmidt, K. and Bannon, L. (1992). Taking CSCW seriously: Supporting articulation work. Computer-Supported Cooperative Work, 1, Nos. 1-2, 7-40.

Schrage, M. (1990). Shared minds: The new technologies for collaboration. Random House.

Sparck-Jones, K. and Galliers, J. R. (1992). Evaluating natural language processing systems. Springer-Verlag.

Sperber, D. and Wilson, D. (1995). Relevance: Communication and cognition (Second edition). Oxford: Blackwell.

Sproull, L., & Kiesler, S. (1991) Connections: New ways of working in the networked organization. Cambridge, MA: MIT Press.

Star, S. L. and Griesemer, J. R. (1989). Institutional ecology: 'Translations' and boundary objects: Amateurs and professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39. Social Studies of Science, 19, 387-420.

Stuart, R. (1995). The design of virtual environments. New York: McGraw-Hill.

Suchman, L. (1987). Plans and situated actions: The problem of human-machine communication. Cambridge: Cambridge University Press.

Suchman, L. (1990). What is human-machine interaction. In S. P. Robertson, W. Zachary, and J. B. Black (Eds.), Cognition, computing, and cooperation (25 -55). Norwood, NJ: Ablex.

Terveen, L. G. (1993). Intelligent systems as cooperative systems. Journal of Intelligent Systems, 3 , 217-249.

Terveen, L. G. (1995). An overview of human-computer collaboration. Knowledge-Based Systems, 8, 67-81.

Terveen, L., Hill, W., Amento, B., McDonald, D. and Creter, J. (1997). PHOAKS: A system for sharing recommendations. Communications of the ACM, 40, 3, March 1997, 59-62.

Turkle, S. (1995). Life on the screen: Identity in the age of the Internet. New York: Simon & Schuster.

Vicente, K. J. & Rasmussen, J. (1992). Ecological interface design: Theoretical Foundations. IEEE Transactions on Systems, Man, and Cybernetics, 22, July/August, 1992, 589-606.

Walker, M., Litman, D., Kamm, C. and Abella, A. (1997). PARADISE: A framework for evaluating spoken dialogue agents. To appear in Proceedings of ACL/EACL.

Watt, J. H. and VanLear, C. A. (Eds.) (1996). Dynamic patterns in communication processes. Sage.

Watts, J., Woods, D. D., Corban, J., Patterson, E., Kerr, R. L. and Hicks, L. C. (1996). Voice loops as cooperative aids in space shuttle mission control. Proceedings of the 1996 ACM Conference on Computer-Supported Cooperative Work, 48-57.

Winograd, T. and Flores, F. (1986). Understanding computers and cognition: A new foundation for design. Norwood, NJ: Ablex.

Woods, D. D. (1986). Cognitive Technologies: the design of joint human machine cognitive systems. AI Magazine, 6, 86-92.

Woods, D. D., Roth, E. M., and Bennett, K B. (1990). Explorations in joint human-machine cognitive systems. In S. Robertson, W. Zachary, and J. Black (Eds.), Cognition, computing, and cooperation (pp. 123-158). Norwood, NJ: Ablex.