The AgendaManager

Agenda Management

Our theory of CTM, as originally formulated, failed to address two important issues. First, human pilots are coming to depend more and more on automated aids, such as autopilots and centralized monitoring and alerting systems, to aid them in the monitoring and control of the aircraft and its subsystems. As machines perform certain goal-directed flightdeck activities, it is more appropriate to speak of those activities as functions since, technically speaking, a task is a function performed by a human. Second, with both humans and machines performing flightdeck functions, there is a potential for conflicting goals. Two recent aircraft accidents illustrate such goal conflicts. In 1994 in a China Airlines Airbus A300 on approach to Nagoya, Japan, the flightcrew inadvertently initiated an autoflight system go-around maneuver while trying to continue the landing. The goal conflict between the flightcrew and the autoflight system caused an out-of-trim condition that resulted in a stall and crash which killed 264 persons. In an American Airlines Boeing 757 on approach to Cali, Columbia in 1995, the flightcrew accepted an air traffic control clearance to fly direct to a designated navigational fix. They inadvertently configured the aircraft's flight management system to fly the airplane to a different fix. This goal conflict was not detected in time to prevent the aircraft from crashing into mountainous terrain, killing 159 persons.

To address these issues, which were clearly related to the original theory of CTM, we expanded the theory. Since an 'agenda' is a list of things to do, we called the new concept Agenda Management (AMgt). To formalize the concept, we developed a model of AMgt using IDEF0, a functional modeling language. IDEF0, whose name stands for ICAM (Integrated Computer Aided Manufacturing) DEFinition language 0, is a graphical modeling language. IDEF0 diagrams consist of boxes representing activities and arrows representing inputs and outputs to and from those activities, controls or constraints on the activities, and mechanisms that perform the activities. In an IDEF0 model of a process, each box represents an activity or function, which transforms its inputs to its outputs, subject to certain controls or constraints, by means of a set of mechanisms. The following summary theory of AMgt is based on the model.

An actor is an entity that does something in that it can control or change the state of the aircraft and/or its subsystems. Pilots are human actors; machine actors include autoflight and flight management systems. A goal is a representation (mental, electronic, or even mechanical) of an actor's intent to change the state of the aircraft or one of its subsystems in some significant way, or to maintain or keep the aircraft or one of its subsystems in some state. For example, a pilot might have a goal to descend to an altitude of 9,000 ft, a goal to maintain the current heading of 270E , and a goal to crossfeed fuel to correct a fuel system imbalance. If configured properly, the autoflight system in this example would also have a goal to descend to 9,000 ft and a goal to hold 270E . Goals come about as a result of planning and decision making in the case of human actors, and computation or human input, in the case of machine actors.

A function is an activity performed by an actor to achieve a goal. That activity may directly achieve the goal or it may produce sub-goals which, when achieved by performing sub-functions, satisfy the conditions of the original goal. Actors use resources to perform functions. Human actor resources include eyes, hands, memory, and attention; machine actor resources include input and output channels, memory, and processor cycles. Other machine resources include flight controls, electronic flight instrument system displays, and radios. In general, several goals might exist at any time, so several functions must be performed concurrently to achieve them. Actors must be assigned to perform those functions and resources must be allocated to enable them. An agenda then is a set of goals to be achieved and a set of functions to achieve those goals.

Agenda Management (AMgt) is a high-level flightdeck function performed cooperatively by flightdeck actors, which involves two sub-functions:

1. Goal management is the process of
1. recognizing or inferring the goals of all flightdeck actors;
2. canceling goals that have been achieved or are no longer relevant;
3. identifying and resolving conflicts between goals; and
4. prioritizing goals consistently with safe and effective aircraft operation.
2. Function management is the process of
1. initiating functions to achieve goals;
2. assigning actors to perform functions;
3. assessing the status of each function (whether or not it is being performed satisfactorily and on time);
4. prioritizing those functions based on goal priority and function status; and
5. allocating resources to be used to perform functions based on function priority.

At any point in time, AMgt performance is satisfactory if and only if:

1. there are no goal conflicts;
2. all goals and functions are properly prioritized; and
3. either
1. performance of all functions is satisfactory, or
2. if that is not possible, actors are actively engaged in bringing the highest priority unsatisfactory functions up to a satisfactory level of performance.

The AgendaManager

From the results of our CTM studies and our analysis of the Nagoya, Cali, and other aircraft accidents, we have concluded that AMgt -- and specifically the failure to perform AMgt satisfactorily -- is a significant factor in flight safety. The objectives of our most recent research task were to develop and to evaluate an experimental computational aid to facilitate AMgt. We call this aid the AgendaManager (AMgr).

Simulator

The part-task flight simulator that provides the context for the AMgr models a generic, twin engine transport aircraft. It is built from components developed at the NASA Langley and NASA Ames Research centers and in our own lab. It runs on one or two Silicon Graphics Indigo 2 computers and provides a simplified aerodynamic model (Langley), autoflight system (Langley), Flight Management System (Langley), primary flight displays (Ames), Mode Control Panel (Ames), and system models and system synoptic displays (OSU). The software is written in C, FORTRAN, and Smalltalk (VisualWorks 2.5).

Architecture and Function

From the IDEF0 model of AMgt we generated a data dictionary consisting of the entities that are the inputs, outputs, and controls of the activities in the model. We used this information to define the object-oriented architecture of the AMgr and the functions of its components. Major AMgr objects include System Agents, Actor Agents, Goal Agents, Function Agents, an Agenda Agent, and an Agenda Manager Interface. Each Agent is a simple knowledge-based object representing the corresponding elements of the cockpit environment. As a representative of such an element, the Agent's purpose is to maintain timely information about it and to perform processing that will facilitate AMgt. An Agent's declarative knowledge is represented using instance variables. Its procedural knowledge is represented using Smalltalk methods.

System Agents (SAs) represent systems modeled in the flight simulator, remembering their state and recognizing abnormal conditions such as malfunctions. System Agents provide situation information to the other AMgr Agents. Actor Agents (AAs) recognize actor (pilot or autoflight system) goals and instantiate Goal Agents. The Flightcrew Agent recognizes pilot goals by means of a Verbex VAT31 automatic speech recognition system as the pilot acknowledges air traffic control clearances. Goal Agents (GAs) represent actor goals. They detect conflicts and determine when goals are achieved. Function Agents (FAs) monitor the progress of activities directed towards the goals, noting whether that progress is satisfactory or unsatisfactory. The single Agenda Agent contains and coordinates the other Agents, introducing new Agents to its collections, checking GAs against each other to identify conflicts, and ordering Goal and Function Agents by priority. The AgendaManager Interface displays AMgt information to the pilot.

Operation

As the simulator runs it sends state data to the AMgr, whose SAs maintain a situation model of the simulated aircraft and its environment. AAs monitor real or simulated actors, detect or infer goals, and instantiate GAs. GAs look for conflicts with each other and monitor SAs to see if the goals are achieved. FAs monitor the progress -- if any -- made in achieving their associated goals. The Agenda Agent prioritizes GAs and FAs and keeps track of goal conflicts. The AgendaManager Interface presents this agenda information to the pilot.

Evaluation

Objective

The purpose of the experiment was to determine any differences in AMgt performance between the use of the AMgr and the use of a model (developed in our lab) of a conventional monitoring and alerting system called the Engine Indication and Crew Alerting System (EICAS).

Method

Subjects

A total of ten airline pilots participated in the experiment, with the first two being used to refine the scenarios and identify and correct problems with software and procedures.

Apparatus

The apparatus consisted of the following components

• part-task flight simulator, running on two Silicon Graphics Indigo 2 workstations,
• the AgendaManager running on one of the two workstations,
• an experimenter's console running on a third workstation,
• and a Verbex VAT31 automatic speech recognition system on a 486 personal computer, connected to the workstation running the AMgr by an RS-232 serial connection.

Procedure

Prior to the experiment each subject was given a brief introduction to the study, filled out a pre-experiment questionnaire, and read and signed an informed consent document. The following forty minutes were used to train the Verbex speech recognition system to recognize the subject's voice so that altitude, speed, and heading goals could be determined from ATC clearance acknowledgements. After a short break the subject learned how to fly the flight simulator using the Mode Control Panel (MCP -- the autoflight system interface), recognize and correct experimenter-induced goal conflicts and subsystem faults, interpret EICAS and AMgr displays, and alter programmed flightpaths. After a lunch break, the subject flew two 30 minute scenarios (one with EICAS, one with the AMgr), separated by a five minute break. Upon the completion of the experiment the subject answered a post-experiment questionnaire.

Experimental Design

The primary factor investigated in the experiment was monitoring and alerting system condition (whether AMgr or EICAS was used). The experimental design was balanced in regard to the monitoring and alerting system used and the scenario (1 or 2).

Data Collection

We collected data for each subject on:

1. how correctly the subject prioritized within concurrent subsystem functions;
2. the average subsystem fault correction time;
3. the average time to properly program the autoflight system;
4. the percentage of goal conflicts detected and corrected;
5. the average time to resolve goal conflicts;
6. how correctly the subject prioritized concurrent subsystem and aviate functions;
7. the average number of unsatisfactory functions at any time;
8. the percentage of time all functions were satisfactory; and
9. the subject's rating of the effectiveness of each monitoring and alerting system: -5 (great hindrance) to +5 (great help).

The raw data for variables 1 - 8 were recorded by the AMgr itself. GoalConflict objects recorded goal conflicts and FunctionAgents, which assess function status as part of their roles, recorded function performance data.

Results

The data were analyzed using Analysis of Variance and the following table summarizes the results obtained for each of these variables, with links to histograms.

The first three variables, within subsystem correct prioritization, subsystem fault correction time, and autoflight programming time, show no statistically significant differences (p-values > 0.05) across the AMgr/EICAS conditions. This is critical for the interpretation of the results in that it supports the hypothesis of the AMgr being the only cause of significant differences. For example, within subsystem prioritization performance does not differ between the two conditions. Also, once a subsystem fault is detected, the process of correcting it is identical between the two conditions. Programming the autoflight system is identical in both conditions. However, we did observe a minor practice effect for each subject between the two scenarios, i.e., they showed significant improvement in programming the autoflight system.

A key objective of the AMgr is to support the pilot in recognizing goal conflicts and to help resolve those in a timely manner. The next two variables, goal conflicts corrected percentage and goal conflict resolution time, directly reflect this, and the results indicate how successful the AMgr condition achieved it (suggestive evidence of differences, with 0.05 < p < 0.10). Any time a goal conflict existed, the AMgr helped the subject identify this conflict (100%) whereas with EICAS, the subjects only identified 70% of the conflicts (a statistically significant difference, with p < 0.05). Also, with the AMgr the subjects were able to resolve the conflict nearly 19 seconds faster. This may have helped them achieve an overall lower level of unsatisfactory functions (AMgr: 0.64; EICAS: 0.85; a statistically significant difference) by making more time available to them.

It is crucial for the pilot to recognize that primary flight control functions (i.e., aviate functions) are usually more critical than subsystem related functions. The AMgr clearly showed its strength by helping the pilots in 72% of the cases to correctly prioritize. With EICAS the pilots only achieved 46% (a statistically significant difference). Last, but not least, with the AMgr the subjects were able to achieve a significantly higher percentage of time where all functions were performed satisfactorily (AMgr: 65%; EICAS: 52%; a statistically significant difference).

Independent of how well an individual can perform under a given condition, it is also important that subjectively he or she finds this condition acceptable. Based on our results, the subjects' effectiveness ratings strongly support the AMgr (4.8 vs. 2, a statistically significant difference).

Discussion

The first set of findings (that there was no difference in measures related to functionally similar capabilities) is suggestive evidence that there was no experimenter-induced bias in favor of the AMgr. The second set of findings is strong evidence that the AMgr actually facilitated AMgt in the context of this experiment.

We must, however, be cautious concerning any inferences made from this finding. The fidelity of the simulator was fairly low and the fact that we observed a period effect (which could include learning) is an indication that perhaps the subjects did not receive adequate training. The simulator was a one-pilot version whereas all of our subjects fly on a two-pilot flightdeck. Finally, the success of the AMgr depends to a very large extent on its ability to correctly recognize the pilot's goals. In five to 10 percent of our subjects' goals the automatic speech recognition system (an old model) did not recognize the goal from the subject's utterance and the Goal Agent had to be instantiated by the experimenter.

Nevertheless, our findings are suggestive that AMgt performance, which is significant to flight safety, can be enhanced by means of a computational aid. Especially in light of recent advances in automatic speech recognition technology and the Federal Aviation Administration's plans to introduce datalink technology to deliver clearances to aircraft, we believe that further development of the AMgr is warranted.

Related Systems

The relationship of the AMgr to several existing aiding systems should be noted. First, the AMgr can be considered a logical extension of the Engine Indication and Crew Alerting System (EICAS) used in present-generation Boeing aircraft, and similar centralized monitoring and alerting systems in other aircraft. EICAS and related systems have been very successful and well received by the operational community. However, they are limited in the extent to which they can tailor the information to the phase of flight and they are not capable of merging the information in case of multiple failures. Of much greater significance is that little or no effort is made to consider the flightcrew's intent at any given moment. The AMgr builds on the success of EICAS by adopting EICAS display philosophy and coding and overcomes the latter limitation by basing its operation on the pilot's declared goals.

The AMgr also has some affinity to Pilot's Associate, Rotorcraft Pilot's Associate, and CASSY (Cockpit Assistant System), all of which are aiding systems designed to offer integrative and active assistance to the pilot. The AMgr is distinguished from these and similar systems in that it does not attempt to be a general, active aid. Rather, the AMgr focuses on passively assisting the flightcrew in performing AMgt by supplementing human memory and attention, not action.

Acknowledgements

The AgendaManager was developed under NASA Ames Research Center grant NAG 2-875. Kevin Corker and Barbara Kanki, of the Ames Aviation Operations Branch, were our technical monitors, and we greatly appreciate their support and encouragement. Greg Pisanich, formerly of Sterling Software and at that time Dr. Corker's assistant, was a constant and essential source of technical and moral support. And we especially appreciate the participation of our subjects. They not only helped us evaluate the AMgr, but also gave us many valuable insights into line operations, which we hope will greatly benefit our future efforts.

What's New

Listed below, most recent first, are changes made to this website since its creation.

3 Jun 99

• original page, from The Agenda Management website