Human-Centered Software Agents:
Lessons from Clumsy Automation
David D. Woods
Cognitive Systems Engineering Laboratory
Ohio State University
The Cognitive Systems Engineering Laboratory (CSEL) has been studying the
actual impact of capable autonomous machine agents on human performance in
a variety of domains. The data shows that "strong, silent, difficult to
direct automation is not a team player" (Woods, 1996). The results of such
studies have led to an understanding of the importance of human-centered
technology development and to principles for making intelligent and
automated agents team players (Billings, 1996). These results have been
obtained in the crucible of complex settings such as aircraft cockpits,
space mission control centers, and operating rooms. These results can be
used to help developers of human-centered software agents for digital
information worlds avoid common pitfalls and classic design errors.
The powers of technology continue to explode around us. The latest focus
of technologists is the power of very large interconnected networks such as
the World Wide Web and digital libraries. The potential of such technology
is balanced with concern that such systems overwhelm users with data,
options and sites. The solution, we are told, is software agents that will
alleviate the burdens faced by consumers in managing information and
interfaces. Promises are being made that agents will hide the complexity
associated with the Web or other large digital worlds. This will be
accomplished by automating many complex or tedious tasks. Agents will help
us to search, browse, manage email, schedule meetings, shop, monitor news,
and so forth. They will filter information for us and tailor it to our
context-specific needs. Some will also help us to collaborate with others.
By assisting with such tasks, agents will reduce our work and information
overload. They will enable a more customized, rewarding, and efficient
experience on the Web. Given this vision, current efforts have focused on
developing powerful autonomous software agents in the faith that "if we
build them, the benefits will come."
In contrast to these dreams and promises is data from a variety of domains
where capable machine agents have already been at work -- highly automated
flight decks in aviation, space mission control centers, operating rooms
and critical care settings in medicine. These machine agents often are
called automation, and they were built in part in the hope that they would
improve human performance by off loading work, freeing up attention, hiding
complexity -- the same kinds of justifications touted for the benefits of
software agents (Table 1 contrasts typical designer hopes for the impact of
their systems on cognition with the results of studies).
The pattern that emerged is that strong but silent and difficult to direct
machine agents create new operational complexities. In these studies we
interacted with many different operational people and organizations,
* through their descriptions of incidents where automated systems behaved
in surprising ways,
* through their behavior in incidents that occurred on the job,
* through their cognitive activities as analyzed in simulator studies that
examined the coordination between practitioner and automated systems in
specific task contexts,
* unfortunately, through the analysis of accidents where people
misunderstood what their automated partners were doing until disaster
One way to see the pattern is simply to listen to the voices that we heard
in our investigations. Users described and revealed clumsiness and
complexity. They described aspects of automation that were strong but
sometimes silent and difficult to direct when resources are limited and
pressure to perform is greatest. We saw and heard how they face new
challenges imposed by the tools that are supposed to serve them and provide
"added functionality." The complexity created when automated systems are
not human- or practice-centered is best expressed by the questions they
posed when working with "clumsy" machine agents:
* "What is it doing now?"
* "What will do next?"
* "How did I get into this mode/state?"
* "Why did it do this?"
* "Why won't it do what I want?"
* "Stop interrupting me while I am busy."
* "I know there is some way to get it to do what I want."
* "How do I stop this machine from doing this?"
These are evidence for automation surprises (Sarter, Woods and Billings, in
press). These are situations where users are surprised by actions taken
(or not taken) by automated agents. Automation surprises begin with
miscommunication and misassessments between the automation and users which
lead to a gap between the user's understanding of what the automated
systems are set up to do, what they are doing, and what they are going to
The evidence shows strongly that the potential for automation surprises is
the greatest when three factors converge:
1. the automated systems can act on their own without immediately
preceding directions from their human partner (this kind of behavior arises
in particular through interactions among multiple automated subsystems),
2. gaps in users mental models of how their machine partners work in
different situations, and
3. weak feedback about the activities and future behavior of the agent
relative to the state of the world.
Designing Agents as Team Players
The dangers can, however, be predicted and reduced. The research results
also point to directions for developing more successful human-centered
automated systems. The key elements are:
* avoid operational complexity
* evaluate new systems in terms of their potential to create specific kinds
of human error and system failure,
* increase awareness and error detection by improved observability of
automation activities (provide feedback about current and future agent
* analyze the impact of new machine agents in terms of coordination demands
placed on the human user (make agents team players),
* give users the ability to direct the machine agent as a resource in the
process of meeting their (practitioners' goals),
* promote the growth of human expertise in understanding how agents work
and how to work agents in different kinds of situations.
Developers of the new breed of agents can avoid the pitfalls and exploit
the opportunities by using the hard won principles and techniques of
human-centered and practice-centered design.
Previous work has established that black box systems are not team players,
create new burdens and complexities, and lead to new errors and failures.
Some level of visibility of agent activities is required; some level of
understanding of how agents carry out their functions is required; some
level of management (delegation and re-direction) of agent activities is
needed. On the other hand, all of the most detailed data about systems may
overwhelm users; complete flexibility may create too many burdens leading
users to just do the job themselves. The key to research on human-centered
software agents is to find levels and types of feedback and coordination
that support team play between machine subordinates and human supervisor
that helps the human user achieve their goals in context.
For example, a common finding in studies that assess the impact of new
automation is that increasing the autonomy, authority and complexity of
machine agents creates the need increased feedback about agent activities
as they handle various situations or what has been termed, observability
(e.g., Norman, 1990; Sarter and Woods, 1995; Woods, 1996). Observability
is the technical term that refers to the cognitive work needed to extract
meaning from available data. This term captures the fundamental
relationship among data, observer and context of observation that is
fundamental to effective feedback. Observability is distinct from data
availability, which refers to the mere presence of data in some form in
some location (Sarter, Woods and Billings, in press). If "strong" software
agents are to be team players, they require new forms of feedback
emphasizing an integrated dynamic picture of the current situation, agent
activities, and how these may evolve in the future. Increasing autonomy
and authority of machine agents without an increase in observability create
Another example concerns a common joint system architecture where the
human's role is to monitor the automated agent. When users determine that
the machine agent is not solving a problem adequately, they interrupt the
automated agent and take over the problem in its entirety. Thus, the human
is cast into the role of critiquing the machine, and the joint system
operates in essentially two modes - fully automatic or fully manual.
Previous work in several domains and with different types of machine agents
has shown that this is a poor cooperative architecture (e.g., Roth et al.,
1987; Layton et al., 1994; Sarter et al., in press). Either the machine
does all the job without any benefits of practitioners' information and
knowledge, despite the brittleness of the machine agents, or the user takes
over in the middle of a deteriorating or challenging situation without the
support of cognitive tools. One can summarize some of the results from
research in this area as, "it's not cooperation, if either you do it all
or I do it all." Cooperative problem solving occurs when the agents
coordinate activity in the process of solving the problem. Cooperating
agents have access to partial, overlapping information and knowledge
relevant to the problem at hand.
New user- and practice-oriented design philosophies and concepts are being
developed to address deficiencies in human-machine coordination. Their
common goal is to provide the basis to design integrated human-machine
teams that cooperate and communicate effectively as situations escalate in
tempo, demands, and difficulty. Another goal is to help developers
identify where problems can arise when new automation projects are
considered and therefore help mobilize the design resources to prevent
Table 1. Designer's eye view of apparent benefits of new automation
contrasted with the real experience of operational personnel.
When new automation is introduced into a system or when there is an
increase in the autonomy of automated systems, developers often assume that
adding "automation" is a simple substitution of a machine activity for
human activity -- the substitution myth. Empirical data on the
relationship of people and technology suggests that is not the case (in
part this is because tasks and activities are highly interdependent or
coupled in real fields of practice). Instead, adding or expanding the
machine's role changes the cooperative architecture, changing the human's
role often in profound ways. New types or levels of automation shift the
human role to one of monitor, exception handler, and manager of automated
Putative benefit Real complexity
better results, transforms practice, the roles of people change
same system (substitution)
frees up resources: 1. create new kinds of cognitive work, often at
offloads work the wrong times
frees up resources: 2. more threads to track; makes it harder for
focus user attention practitioners to remain aware of and integrate
on the right answer all of the activities and changes around them
less knowledge new knowledge/skill demands
autonomous machine team play with people is critical to success
same feedback new levels and types of feedback are needed to support
peoples' new roles
generic flexibility explosion of features, options and modes create new
demands, types of errors, and paths towards failure
reduce human error both machines and people are fallible; new problems
associated with human-machine coordination breakdowns
Creating partially autonomous machine agents is, in part, like adding a new
team member. One result is the introduction of new coordination demands.
When it is hard to direct the machine agents and hard to see their
activities and intentions, it is difficult for human supervisorsto
coordinate activities. This is one factor that may explain why people
"escape" from clumsy automation as task demands escalate.
Norman, D.A. (1990). The 'problem' of automation: Inappropriate feedback
and interaction, not 'over-automation.' Philosophical Transactions of the
Royal Society of London, B 327:585--593.
Hutchins, E. (1995). Cognition in the Wild. MIT press.
CSEL References on Human-Centered Systems
N. Sarter, D.D. Woods and C. Billings. Automation Surprises. In G.
Salvendy, editor, Handbook of Human Factors/Ergonomics, second edition,
Wiley, New York, in press.
D.D. Woods and J.C. Watts. How Not To Have To Navigate Through Too Many
Displays. In Helander, M.G., Landauer, T.K. and Prabhu, P. (Eds.) Handbook
of Human-Computer Interaction, 2nd edition. Amsterdam, The Netherlands:
Elsevier Science, 1997.
D.D. Woods, E.S. Patterson, J. Corban and J.C. Watts. Bridging the Gap
between User-Centered Intentions and Actual Design Practice. Proceedings
of the Human Factors and Ergonomics Society, September, 1996.
D.D. Woods. Decomposing Automation: Apparent Simplicity, Real Complexity,
In R. Parasuraman and M. Mouloula, editors, Automation Technology and Human
Performance, Erlbaum, p. 3-17, 1996.
L. Johannesen. The Interactions of Alicyn in Cyberland, Interactions, 1(4),
D.D. Woods. The price of flexibility in intelligent interfaces.
Knowledge-Based Systems, 6:1-8, 1993.
D.D. Woods, E.M. Roth, and K.B. Bennett. Explorations in joint
human-machine cognitive systems. In S. Robertson, W. Zachary, and J. Black,
editors, Cognition, Computing and Cooperation, Ablex Publishing, Norwood,
R.I. Cook and D.D. Woods. Adapting to new technology in the operating
room. Human Factors, 38(4), 593-613, 1996.
J.H. Obradovich and D.D. Woods. Users as designers: How people cope with
poor HCI design in computer-based medical devices. Human Factors, 38(4),
R.I. Cook and D.D. Woods. Implications of automation surprises in aviation
for the future of total intravenous anesthesia (TIVA). Journal of Clinical
Anesthesia, 8:29s-37s, 1996.
E. Moll van Charante, R.I. Cook, D.D. Woods, L. Yue and M.B. Howie.
Human-computer interaction in context: Physician interaction with automated
intravenous controllers in the heart room. In H.G. Stassen, editor,
Analysis, Design and Evaluation of Man-Machine Systems 1992, Pergamon
Press, 1993, p. 263-274.
N. Sarter and D.D. Woods. Teamplay with a Powerful and Independent Agent:
A Corpus of Operational Experiences and Automation Surprises on the Airbus
A-320. Manuscript submitted for publication, 1997.
Billings, C.E. (1996). Aviation Automation: The Search For A
Human-Centered Approach. Hillsdale, N.J.: Lawrence Erlbaum Associates.
N. Sarter and D.D. Woods. "How in the world did we get into that mode?"
Mode error and awareness in supervisory control. Human Factors, 37: 5-19,
N. Sarter and D.D. Woods. 'Strong, Silent and Out of the Loop:' Properties
of Advanced (Cockpit) Automation and their Impact on Human-Automation
Interaction, Cognitive Systems Engineering Laboratory Report, CSEL
95-TR-01, The Ohio State University, Columbus OH, March 1995. Prepared for
NASA Ames Research Center.
N. Sarter and D.D. Woods. Pilot Interaction with Cockpit Automation II: An
Experimental Study of Pilot's Model and Awareness of the Flight Management
System. International Journal of Aviation Psychology, 4:1-28, 1994.
N. Sarter and D.D. Woods. Pilot Interaction with Cockpit Automation I:
Operational Experiences with the Flight Management System. International
Journal of Aviation Psychology, 2:303--321, 1992.
E.M. Roth, K. Bennett, and D.D. Woods. Human interaction with an
'intelligent' machine. International Journal of Man-Machine Studies,
Space Systems and Process Control
J. Malin, D. Schreckenghost, D. Woods, S. Potter, L. Johannesen, M.
Holloway and K. Forbus. Making Intelligent Systems Team Players. NASA
Technical Report 104738, Johnson Space Center, Houston TX, 1991.
D. Ranson and D.D. Woods. Animating Computer Agents. Proceedings of Human
Interaction with Complex Systems, IEEE Computer Society Press, Los
Alamitos, CA, 1996.
D. Ranson and D.D. Woods. Opening Up Black Boxes: Visualizing Automation
Activity. Cognitive Systems Engineering Laboratory Report, CSEL 97-TR-01,
The Ohio State University, Columbus OH, January 1997.
J.C. Watts, D.D. Woods, J.M. Corban, E.S. Patterson, R. Kerr, and L. Hicks.
Voice Loops as Cooperative Aids in Space Shuttle Mission Control. In
Proceedings of Conputer-Supoprted Coopertaive Work. Boston MA, 1996.
J.C. Watts, D.D. Woods, E.S. Patterson. Functionally Distributed
Coordination during Anomaly Response in Space Shuttle Mission Control.
Proceedings of Human Interaction with Complex Systems, IEEE Computer
Society Press, Los Alamitos, CA, 1996.
L. Johannesen, R.I. Cook, and D.D. Woods. Cooperative communications in
dynamic fault management. In Proceedings of the 38th Annual Meeting of the
Human Factors and Ergonomics Society, October, Nashville TN, 1994.
S.S. Potter and D.D. Woods. Event-driven timeline displays: Beyond message
lists in human-intelligent system interaction. In Proceedings of IEEE
International Conference on Systems, Man, and Cybernetics, IEEE, 1991.
David D. Woods
Cognitive Systems Engineering Laboratory
210 Baker Systems
The Ohio State University
1971 Neil Avenue, Columbus, Ohio 43210