Preliminary Cockpit Task Management Research at Oregon State University
Preliminary Cockpit Task Management Research at Oregon State University |
|
| Synopsis: | This page describes several early studies of Cockpit Task Management (CTM) conducted at Oregon State University. | ||
| Keywords: | cockpit task management, workload, aircraft accident reports, aircraft incident reports | ||
| Authors: | |||
| Ken Funk | <funkk@engr.orst.edu> | Department of Industrial and Manufacturing Engineering, Oregon State University, Corvallis, Oregon, USA | |
| Chung-di Chou | <chouc@hopi.dtcc.edu> | Distributed Systems Department, MBNA America | |
| Das Madhavan | formerly, Department of Industrial and Manufacturing Engineering, Oregon State University, Corvallis, Oregon, USA | ||
| Last Update: | 23 Dec 99 | This is a Work in Progress and its contents are subject to continual revision. | |
To determine the nature and significance of CTM in flight operations, we set out to
1. develop a taxonomy of CTM errors;
2. study CTM behavior in operational settings by means of accident and incident reports;
3. study CTM behavior under controlled, laboratory conditions; and
4. make recommendations to improve CTM behavior through training and design.
The descriptions of the following three studies is based on Chou et al (1996).
We developed an initial CTM error taxonomy consisting of seven general CTM error categories corresponding to the functions of CTM described above (Chou & Funk, 1990). Each category was further described in terms of specific error classes. Use of the initial taxonomy in preliminary analyses of accident and incident reports showed some of the error classes to be redundant and the taxonomy, as a whole, to be difficult to apply consistently.
As a result we revised the taxonomy to include the following CTM error categories:
This revised taxonomy served as the basis for our accident and incident studies, described below.
The underlying causes of aircraft accidents usually fall into the three broad categories of mechanical factors, weather, and pilot error. However, these labels should not be used to mark the end of further analyses for human and other system performance errors since aircraft accidents are usually the outcomes of a number of contributing factors. In an effort to determine if some instances of pilot error could be explained in terms of CTM, and thereby begin to understand the significance of CTM to flight safety, we reviewed a set of aircraft accident reports (Chou, 1991).
Our analysis reflects the examination of the abstracts of 324 NTSB aircraft accident reports concerning accidents occurring between 1960 and 1989. After reviewing the 324 National Technical Information Service (NTIS) abstracts of these reports, accidents that were obviously unrelated to this study were removed from the screening process. For example, this included accidents due primarily to weather and mechanical failures. This elimination process left 76 accident reports for further analysis.
Following the initial screening, we selected a representative set of cases for further study, based on the following considerations. First, we chose the cases so as to include a complete set of CTM errors as described above. Secondly, we chose cases involving conditions we felt we could reconstruct in a simulated environment. For each accident in this set, we carefully studied the data and conclusions of the NTSB investigators and constructed an operational task context. Each context was a graphical representation of cockpit activities during the time leading up to the accident. It included the number and type of concurrent tasks competing for the flightcrew's resources, the state of each task (pending, active, interrupted, or terminated), and selected system state variables (e.g., aircraft altitude, speed, etc.). With the insights gained from this detailed analysis and using the data and conclusions in the accident abstracts and full reports, we identified and classified 80 CTM errors in 76 of the 324 accident reports. That is, we found that CTM errors occurred in about 23 per cent of the accidents reviewed. These errors, summarized by category, are presented in Table 1.
Table 1:
CTM Errors Identified and Classified in 76 (23%) of 324 NTSB Accident Reports.
| CTM error | Number of accidents |
Per cent of CTM accidents |
Number of CTM errors |
Per cent of all CTM errors |
| Task initiation | 35 |
46 |
35 |
44 |
| Task prioritization | 24 |
32 |
24 |
30 |
| Task termination | 21 |
28 |
21 |
26 |
| Total | -- |
-- |
80 |
100 |
Although we cannot state categorically that CTM errors were the sole or even primary causes of these accidents, we do believe that they played significant roles. Had the errors been prevented, the accidents probably would have not occurred. We conclude that the moderately high incidence of CTM errors in the accidents -- 76 (23 per cent) of 324 accidents -- is supportive evidence that CTM is a significant factor in flight safety.
Fortunately, aircraft accidents are very rare events. Unfortunately, a set of accidents like the one we studied might be a very biased sample of the operating environment. Therefore, inferences made from a set of accidents may have little relevance to reducing the likelihood of future accidents. For that reason, we next turned our attention to aircraft incidents (Madhavan, 1993).
Incident means an occurrence other than an accident, associated with the operation of an aircraft, which affects or could affect the safety of operations (Federal Aviation Regulations, 1994). Although incidents by definition do not involve death, serious injury or substantial aircraft damage, in retrospect, most airline accidents were foreshadowed by clear evidence that the problems existed long before, as incidents. Our specific objective in analyzing aircraft incidents was to determine the significance of CTM in flight operations more representative of normal conditions.We used as a source of aircraft incident information NASA's Aviation Safety Reporting System (ASRS). The ASRS database consists of anonymous reports filed by pilots and air traffic controllers describing events in which accidents nearly occurred or in which flight safety was seriously compromised.
Our preliminary analysis of CTM errors focussed on aircraft incident reports relating to in-flight engine emergencies (99 reports) and controlled flight toward terrain (CFTT, 205 reports). We found CTM errors in 19 per cent and 54 per cent respectively of these reports. The high incidence of CTM errors in the CFTT reports as well as the fact that over 49 per cent of all airline accidents occur during approach and landing (Boeing, 1993), caused us to focus further attention on the terminal phases of flight. At our request the ASRS office furnished us with 243 additional reports pertaining to these phases.
As in most ASRS studies, we used the narrative section of the reports for our analysis. The narrative is the section of the report in which the reporter states in his/her own words what happened and why it happened.
In the narratives, we focussed on activities directly related to task management only. Incidents involving crew personality differences and other sociological factors were excluded. Where narratives were unclear about the specific errors committed (i.e. no categoric admission of the errors by the reporters), some inferences were made about the errors based on our knowledge of standard operating procedures, as gleaned from aircraft operations manuals, accident reports, incident reports, and other aviation literature. Explicit statements in the narratives such as "... forgot ...", "... omitted ...", "... memory lapse ...", "... oversight ..." etc., enabled us to home in quickly on the error classification.
From the 540 ASRS incident reports we obtained, we eliminated duplicates. We then applied the CTM error taxonomy to the remaining 470 unique reports. We found CTM errors in 231 (49 per cent) of the 470 ASRS incident reports. The results of the analysis are presented in Table 2.
Table 2: CTM Errors Identified and Classified in 231 (49%) of 470 ASRS Incident Reports.
| CTM error | Number of incidents |
Per cent of CTM incidents |
Number of CTM errors |
Per cent of all CTM errors |
| Task initiation | 137 |
59 |
145 |
42 |
| Task prioritization | 133 |
58 |
122 |
35 |
| Task termination | 83 |
36 |
82 |
23 |
| Total | -- |
-- |
349 |
100 |
Task initiation appears to be the most significant CTM error category, accounting for 42 per cent of the CTM errors identified. Task initiation errors included early descents, late configurations, and failures to tune navigation and communication radios. Task prioritization errors accounted for 35 per cent of the CTM errors and included distractions by weather and traffic watches. The remaining 23 per cent of the CTM errors were in the task termination category. These included early autopilot disengagements, altitude overshoots, and improperly continued landings under unsafe conditions.
While task initiation appears to be the largest CTM error category, that may be somewhat misleading. The failure to start a task on time (or at all) or the decision to start a task too early may often be explained as misprioritization. That is, excessive priority placed on one task may delay the start of a second task or cause the flightcrew to start the first task before they should. Similar arguments can be made for task prioritization verses task termination. Although the initiation and termination categories are useful for understanding errors, their causes, and their consequences, task prioritization should perhaps draw our greatest attention for the development of countermeasures.
We conclude that the high incidence of CTM errors in the incident reports -- 231 (49 per cent) of 470 reports -- is supportive evidence that CTM is a significant factor in flight safety.
From our accident and incident studies, we determined that CTM is significant enough to warrant further study. However, we felt that a different approach was needed to better understand the nature of CTM behavior. Aircraft accidents are rare events, thus providing few opportunities for developing insights into error processes, which are, in any case, very difficult to reconstruct. By the same token, though ASRS incident reports can provide first-hand information on abnormal cockpit operations, they are subject to self-reporting biases and other problems. Therefore, controlled experimentation provides a useful alternative, serving to compensate for the drawbacks noted above and to provide an opportunity for objective observations. An additional advantage of the simulation method is that it enables observation of how human operators manage tasks under normal conditions.
The main objectives of our experiment were to elicit and observe CTM errors similar to those identified in the accident and incident analyses and to identify the factors leading to such errors. Our approach was to have subject pilots fly a low fidelity flight simulator in several flight scenarios and observe and analyze their behavior in managing and performing concurrent flight tasks.
Our flight simulator consisted of three networked personal computers. The system simulated a generic, two-engine commercial transport aircraft. One computer simulated aircraft dynamics using a very simple aerodynamic model and produced a simple primary flight display showing heading, altitude, airspeed, pitch, and roll. The subject controlled the simulated aircraft by means of a joystick. A second computer simulated the navigation system and presented a moving map display. The subject could use the navigation display for planning and navigating purposes and could control map scale and orientation (north up or track up) by means of mouse-activated controls. The third computer simulated aircraft subsystems, including engines and hydraulic system, and generated a simplified Engine Indicating and Crew Alerting System (EICAS) display. Aircraft subsystem models included failure modes that could be triggered by script files and which required subject interaction by mouse-activated controls to correct.
Twenty-four unpaid subjects from Oregon State University participated in the experiment. The subjects included two engineering faculty members, three undergraduate engineering students, and 19 engineering graduate students. Two of the subjects had private pilot licenses with 120 to 150 hours flight time. The other subjects had no flight experience. Sixteen subjects participated in two pilot studies, and the remaining eight subjects participated in the data collection runs. The pilot studies were used for refining training procedures and flight scenarios.
Subjects received a 60-minute training session prior to each experiment. This session included viewing a training video tape and running a simplified scenario. The scenarios were categorized into 6 different levels by the following independent variables: resource requirements, maximum number of concurrent tasks, and flight path complexity. Following concepts from multiple resource theory (Wickens, 1992) and W/INDEX (North & Riley, 1989), scenarios were created and rated according to the requirements for visual resources (to acquire needed information from simulated visual displays), manual resources (to manipulate simulated controls), and mental resources (to recognize, remember, calculate, and decide). Each scenario received an aggregate resource requirements rating (low or high). The number of concurrent tasks was defined as the maximum number of tasks requiring subject attention at any point in the scenario (three or six). Flightpath complexity (easy or hard) was varied by adjusting the sharpness of turns at waypoints in the flightpath.
A split-plot design was used for the experiment. The latter factors (number of concurrent tasks and flightpath complexity) were crossed to provide four levels for whole unit factors. These four whole unit factors were then crossed with the subunit levels (resource requirements) to provide eight treatments. Given this design, eight subjects were used to provide two responses for each treatment. Each subject performed two levels of the subunit factor (low and high resource requirements), and the assignment of treatments to subjects was randomized to control learning effect. That is, four subjects started with the high resource requirements treatment and then performed the low resource requirement treatment, whereas the other four performed their treatments in the reverse order.
The following performance measures were used:
The response time to a system fault was defined as the time from the occurrence of the fault (such as an electrical bus fault) until a compensating response was initiated. This corresponded to task initiation. The task prioritization score was determined from paired comparisons between tasks, and was used for measuring task prioritization performance. A score of +1 was assigned when a correct prioritization was made by the subject (i.e., attention was first given to the higher priority task), otherwise a -1 was assigned. Scores for the remaining tasks were set to zero. Finally, a task was said to be initiated late if the subject did not respond to the task 60 seconds after it had been activated. This was used to measure task initiation performance.
The analysis of variance (ANOVA) results for factors with significant effects are summarized in Table 3. We found the resource requirements level to have a significant effect on the average task response time. That is, higher resource requirements increased delays in initiating a task. However, neither combination of flightpath complexity nor maximum number of concurrent tasks (alone or in combination) had a significant effect on task response time.
Table 3: Summary of Experimental Results: F and p Values, Statistical Significance.
Experimental factors |
||
| Response variables |
Number of concurrent tasks and flight path complexity (degrees of freedom = 3, 4) | Resource requirements (degrees of freedom = 1, 4) |
| Task initiation (average response time) | F = 5.85 p = 0.060 not significant |
F = 14.65 p = 0.019 significant at alpha = 0.05 |
| Task initiation (late task initiation) | F < 6.59* p > 0.05* not significant |
F = 27.00 p = 0.007 significant at alpha = 0.01 |
| Task prioritization
|
F = 32.08 p = 0.003 significant at alpha = 0.01 |
F = 34.13 p = 0.004 significant at alpha = 0.01 |
| RMS flight parameter errors | F = 1.26 p = 0.400 not significant |
F = 3.04 p = 0.156 not significant |
| not significant: not statistically significant significant at alpha = 0.05: marginally significant significant at alpha = 0.01: highly significant * Exact F and p values were not recorded. |
||
During the experiments, subjects were warned if 60 seconds passed after the occurrence of a system fault and no actions were taken. Thus, the definition of a late initiation was failure to initiate the task within one minute following fault occurrence. The analysis of variance results show that resource requirements had a significant effect on late task initiation.
Results from the ANOVA show that both resource requirements and the combination of flightpath complexity and number of concurrent tasks created significant effects on task prioritization. Therefore, task prioritization degrades as either one of these factors increase.
We calculated the RMS of deviations in flight parameters using data obtained from whole mission information. Heading deviations were significantly affected by the combination of flightpath complexity and the number of tasks; changes in mental resource requirements were significant to the altitude deviation. None of the other RMS deviations were significantly affected by either the resource requirements or the combination of flightpath complexity and the number of concurrent tasks.
We developed a normative theory of Cockpit Task Management and a taxonomy of CTM errors, based on that theory and applied the latter in the analysis of National Transportation Safety Board aircraft accident reports and Aviation Safety Reporting System incident reports. We found CTM errors in 76 (23 per cent) of the 324 accident reports analyzed and in 231 (49 per cent) of the 470 incident reports. In a low fidelity simulator study, we found that resource requirements (visual, manual, and mental) had a statistically significant effect on task initiation and task prioritization performance, and that the number of concurrent tasks coupled with flight path complexity had a statistically significant effect on task prioritization performance.
From our studies of aircraft accidents and incidents, we conclude that CTM is a significant factor in flight safety. Furthermore, as Raby and Wickens' (1994) results implied, our experiments confirm that increased resource requirements increase the likelihood of CTM errors, specifically, late task initiation and incorrect task prioritization errors.
We offer four recommendations. First, we recommend that pilots receive instruction concerning CTM and how to avoid CTM errors. More specifically, pilots should be made aware that in periods of high workload, when large numbers of concurrent tasks are competing for their attention, there is danger that they will not initiate important tasks promptly and/or that their attention will be drawn away from safety-critical tasks. Presumably, pilots can be taught to recognize these precursor conditions and to develop personal strategies to avoid CTM errors when these conditions are present. CTM instruction might most naturally fit into existing Crew Resource Management training programs.
This recommendation is based on the assumption that our experimental environment, involving a low fidelity simulator and (mostly) non-pilot subjects is, at a very high level of abstraction, similar enough to the real commercial transport aircraft environment to warrant extrapolation. This assumption should be tested, so our second recommendation is that further studies of CTM be conducted using full-mission scenarios in high fidelity training simulators with line pilots as subjects. The objectives should be to validate our earlier findings, to search for other factors affecting CTM performance, to identify patterns of both good and bad CTM, and to attempt to link CTM errors with human cognitive characteristics, such as short term (working) memory limitations.
Third, we recommend that research be conducted to develop and evaluate formal cockpit procedures to facilitate CTM performance, based on findings from the studies recommended above. Such procedures might, for example, involve memory aids and elaborated versions of the well-known pilots' prioritization maxim: aviate -- navigate -- communicate -- manage systems.
Finally, our fourth recommendation is that research be conducted to develop and evaluate computational aids to facilitate CTM performance. Such aids might, for example,
We have in fact conducted such research, which is described elsewhere in this website.
| Return to: Cockpit Task Management: Objectives |
Boeing. (1993). Statistical summary of commercial jet aircraft accidents, worldwide operations, 1959-1993. Seattle: Boeing Commercial Airplane Group.
Chou, C.D. (1991). Cockpit Task Management Errors: A Design Issue for Intelligent Pilot-Vehicle Interfaces. Unpublished doctoral dissertation, Oregon State University.
Chou, C. D., & Funk, K. (1990). Management of multiple tasks: cockpit task management errors. In Proceedings of the 1990 IEEE International Conference on Systems, Man, and Cybernetics (pp. 470-474). Piscataway, NJ: The Institute of Electrical and Electronics Engineers.
Chou, C.D., Madhavan, D. and Funk, K. (1996). "Studies of Cockpit Task Management Errors," International Journal of Aviation Psychology, 6(4), 307-320.
Federal Aviation Regulations, 49 Code of Federal Regulations ' 830.2 (1994).
Funk, K. (1991). "Cockpit Task Management: Preliminary Definitions, Normative Theory, Error Taxonomy, and Design Recommendations, The International Journal of Aviation Psychology, 1(4), 271-285.
Madhavan, D. (1993) Cockpit Task Management Errors: An ASRS Incident Report Study. Unpublished master's thesis, Oregon State University.
Navon, D., & Gopher D. (1979). On the economy of the human processing system. Psychological Review, 86, 3. 214-255.
Padulo, L., & Arbib, M. A. (1974). System Theory. Washington, DC: Hemisphere.
Raby, M., & Wickens, C.D. (1994). Strategic workload management and decision biases in aviation. The International Journal of Aviation Psychology, 4(3), 211-240.
Wickens, C. D. (1984). Processing resources in attention. In R. Parasuraman & D. A. Davies (Eds.), Varieties of Attention, pp. 63-102. Orlando: Academic Press.
Wickens, C. D., & Liu, Y. (1988). Codes and modalities in multiple resources: A success and a qualification. Human Factors, 30(5) 599-616.
Wilson, J.R. (1998). The Effect of Automation on the Frequency of Task Prioritization Errors on Commercial Aircraft Flight Decks: An ASRS Incident Report Study. Unpublished thesis, Oregon State University.
Wilson, J.R. and Funk, K. (1998). The effect of automation on the frequency of task prioritization errors on commercial aircraft flight decks: an ASRS incident report study," in Proceedings of the Second Workshop on Human Error, Safety, and System Development, Seattle, Washington, USA, 1-2 April 1998, pp. 6-16.
Listed below, most recent first, are changes made to this page since its creation.