Initial Identification of Factors That Affect Task Prioritization on the Flight Deck

Over the past decade, cockpit task management (CTM) has been isolated as a cognitive function that is intuitively well understood by pilots and almost always performed satisfactorily. However, there are documented accounts where tasks were not managed properly, resulting in an incident or accident (Chou et al., 1996). A very vivid example of improper CTM can be drawn from the 1972 Eastern Airlines accident in Miami, Florida, where the failure of a gear-down lamp ultimately engrossed the full attention of 3 pilots who failed to respond to an autopilot disengagement. The aircraft descended into the ground, killing 99 people on board (NTSB, 1973). Other CTM error examples can be found in Chou (1991) and Madhavan (1993).

Task prioritization, in the context of the flight deck, is defined as the proper allocation of attentional resources to tasks in order to achieve subgoals which support the overall mission goal. In other words, proper task prioritization ensures the pilot is "doing what he should be doing." A CTM prioritization error occurs when attentional resources are allocated to a task with a lower priority at the expense of another, higher priority task. Funk (1991) identified several categories of CTM errors. One of those error types, improper task prioritization, has been shown to be significantly present in both aircraft accidents and incidents (Chou, et al. 1996).

Of the existing work that has been done in the area of task management, none have addressed task prioritization exclusively. Stated very simply, this part of the study attempted to identify which factors pilots use to determine task priority, and ultimately, what task they will allocate their attention towards. For a comprehensive background of CTM and related topics, see Chapter 2.

In this experiment, pilots flew arrival procedures in a part-task simulator. Two knowledge elicitation techniques were used to probe the subjects for factors that influenced their attentional prioritization scheme while performing multiple, concurrent flight deck tasks.

There are two primary challenges with studying task prioritization or attention allocation strategies. First, it is very difficult to determine what pilots are attending to at any particular instant. It is accepted that the location of eye focus is often a good indication of where one’s attention is focused at a particular time. This is especially true in an environment such as the flight deck, where much of the task-related information is obtained visually. However, it is also very evident that this is not always the case. A pilot can be looking at a display, but truly be thinking about something totally unrelated to the display. This situation may occur quite often, and it is extremely difficult to determine from the external perspective of an experimenter.

Secondly, if we were able to satisfactorily determine to what the pilot was attending, the next challenge is to determine why. In other words, how does the task prioritization process work? Is it an internally-driven process, where the pilot uses all past experiences as a knowledge base to implicitly prioritize tasks, or is it a process driven by the environment, where a pilot merely reacts to events as they occur on the flight deck?

Task prioritization and attention allocation may very well be an internal, implicit process that is not directly accessible through any known measurement equipment or techniques (Adams, et al., 1991). The approach in this part of the study was to use knowledge elicitation techniques to verbally probe pilots as to what factors influence their task prioritization while flying.

The literature documents tens of knowledge elicitation techniques, each having advantages and disadvantages depending on the particular task environment (Salter, 1988; Cooke, 1994). Often, the recommendations are to use multiple techniques in order to emphasize each technique’s strengths and minimize its weaknesses. In general, each technique varies in its requirements. Some techniques are best implemented in a natural environment, while others are best used in a laboratory setting.

Another concern with knowledge elicitation techniques is the intrusiveness of the method: how much the technique disturbs normal task performance. The more intrusive a technique is, the more difficult it will be to use in an actual operational setting and the more it may disturb the very nature of the task being performed. This may result in the acquisition of knowledge that is not entirely representative of true task performance.

A fundamental characteristic of elicitation techniques is that of ecological validity: the measure of how much a task is like the actual task of interest. Actual task performance is clearly the most ecologically valid environment, and the data generated in that setting will be a very good sample of true task performance. However, thorough investigation of true task performance is not always possible, especially in an environment such as the piloting of an aircraft, where Federal Aviation Administration (FAA) regulations are but the first hurdle to data collection on the flight deck. Additionally, the complexity of the operational setting, the very aspect that gives task performance its ecological validity, often makes it difficult to isolate the relevant behaviors and knowledge. This tradeoff is inherent: the more ecologically valid the task setting, the more complex; the more complex, the more both data and noise are generated and collected; the more data and noise, the more difficult to separate them, and to focus on the questions of interest. On the other hand, in a simplified simulation environment, with a well-defined task and goal structure, performance data can be carefully and accurately collected. However, this same isolation may so transform the task that the knowledge used in this environment may significantly differ from that used in the true operational environment.

In the present study, two elicitation techniques were used: retrospective comment analysis (referred to here as the "retrospective" technique) and interruption analysis (the "intrusive" technique). These techniques were chosen for their strengths, their practicality of use and their compatibility with the equipment available for the present research.

In the retrospective technique, task performance is recorded as it occurs naturally, then reviewed for analysis with the subject. This is usually performed with the use of videotaping equipment, allowing for the subject to be placed back into the context of the task environment as much as possible.

One of the advantages of the retrospective technique is the ecologically valid data that can be collected, as tasks can be performed with little or no intrusion. However, the shortcomings of this technique include the possibly limited ability of the subject to remember and interpret behaviors after the situation has passed. The retrospective technique is best for explicit knowledge elicitation, however in certain circumstances, this technique can allow the subject to observe the application of his implicit knowledge and perhaps verbalize this implicit knowledge.

In this experimental study, the retrospective interview was employed with the aid of a videotaping system. The pilot performed an entire flight scenario while being videotaped. The videotape was a picture-in-a-picture configuration, where upon review, the pilot and experimenter could review both the instruments of the simulator and the body movements of the pilot. At predetermined situations in the scenario, the videotape was stopped, and the pilot probed for task prioritization information (see the cognitive interview section below for a description of the probing technique).

The intrusive technique involves observing the subject performing the actual task, and then interrupting the subject during actual task performance and probing about aspects of what has just occurred. While this method can be applied in a highly ecologically valid task environment, the task must lend itself to being interruptible. Thus in aviation research, for obvious reasons, such a method could never be applied in actual flight conditions, but is limited to flight simulators. Additionally, it is important to recognize that once task performance has been interrupted, the task environment has changed, so this method may not accurately obtain data of the behaviors of interest. Often, because of the intrusive characteristic of this method, it is used to focus on one aspect of task performance at a time.

In the intrusive technique used in this study, the pilot again performed a flight scenario on the simulator. However, at predetermined points in the flight scenario, the simulator was stopped, and the pilot was probed immediately. Following the interview, the flight scenario was again resumed until the next probe.

The justification of selecting these two techniques for this study is three-fold. First, because the current study was an initial investigation of task prioritization, it was in the hypothesis generation phase. Due to the nature of the free dialog during both the retrospective and intrusive interviews, the data tends to be extensive and at a relatively high level of detail suited for hypothesis generation. Second, if the intrusive technique was found to be too disruptive to the task environment, there would still be a significant amount of data collected during the retrospective interviews to be useful. Alternately, if the retrospective technique was found to elicit lower quality data, then the data collected using the intrusive technique could still be useful. Finally, one of the questions posed in the current study was a determination if there was any difference in the data collected using the two techniques, as it is anticipated that follow on studies will be performed using one or both of these techniques.

As described above, the interviewing opportunities in this experiment were determined by either the intrusive or retrospective elicitation techniques. However, once the probe was initiated, the actual form of the questioning was consistent between the two methods. The cognitive interviewing technique formed the basis of the probe questions during each of the interviews.

Many interviewing techniques are based on the structured interview, where the interviewing method and questioning are presented to each subject in a consistent manner. This is an attempt to minimize extraneous variables in data collection by presenting an environment where the questions and all participant-experimenter interactions are predetermined. Using this technique provides a very systematic approach, with little opportunity for experimenter bias. However, on the other hand, there is little opportunity for the subject to elaborate on internal thoughts and justifications of observed behavior.

Fischer and Geiselman (1992) developed the cognitive interviewing technique. Its initial application was in crime investigation, where investigators were tasked with interviewing eyewitnesses to crimes. They found that traditional interviewing methods had several shortcomings that often hampered criminal investigations. Through extensive development and testing, Fischer and Geiselman established that the cognitive interview is superior to traditional eyewitness interviewing techniques in gathering specific details about a crime. They identify 13 basic concepts that aid in the retrieval of detailed information that is stored in memory, but is not readily accessible. Not all of these concepts are appropriate for this application of the technique, but some were anticipated to be very useful in extracting the information about how the pilot prioritizes tasks in an operational environment. The 13 concepts of the cognitive interview are:

1. Encouraging active information generation on the part of the subject.
2. Active listening on the part of the interviewer.
3. Asking open-ended questions.
4. Pausing after the subject’s response before asking follow-up questioning.
5. Not interrupting the subject during information retrieval.
6. Explicitly requesting detailed descriptions of information.
7. Encouraging the subject to concentrate intensely.
8. Encouraging the subject to use imagery.
9. Recreating the original event context.
10. Adopting the subject’s perspective
11. Asking subject-compatible questions.
12. Following the sequence of the cognitive interview.
13. Establishing rapport with the subject.

At the core of the cognitive interview are several key concepts. First, the interviewer should strive to recreate the original context of the situation of interest. Second, the subject should be encouraged to form mental images of the situation, and the interviewer can facilitate that effort by carefully worded questions and ample time for the subject to form the images. Finally, the interviewer should help guide the subject through a systematic evaluation of those mental images. Through these concepts, at least in eyewitness interviews, a surpassingly rich and complete recall of information can be achieved.

In order to use the cognitive interview in this experiment, the experimenter extensively studied the cognitive interviewing technique and practiced applying it to subjects before data collection began.

One of the critical aspects of this study was the selection of the probing conditions. One of the hypotheses of prioritization strategies is that the prioritization of tasks is directly related to the task environment. The challenge for the present research is to select points that are not identical conditions, but representative of the task environment.

One approach was to randomly select data collection points. This was not chosen for the following reason. The task environment is very dynamic, and pilots move from being almost idle to quite busy, and vice versa, very quickly. With relatively few data collection points for each pilot, the high probability of probing a pilot during a nearly idle condition is quite high. This study was investigating the tactical or reactionary characteristics of behavior and was not focused on strategic or planning-ahead behaviors, when current task demands were few (see Chapter 2). Therefore, a systematic probing strategy was desired that queried pilots when they had multiple tasks active, yet not identical task conditions.

This study identified three types of events. These events are generally representative of the tactical flying environment of an arrival/approach phase of flight:

1. Procedure Events – these are events that occur as part of the mission tasks. For instance, as the pilot intercepts the instrument landing system (ILS) localizer (see Glossary in Appendix 1), the indication needle begins to move. This is a cue for the pilot to begin the turn onto the final approach. It is a standard procedure for pilot to follow: when the instrument commands turning the aircraft on to final approach, the pilot performs the task. These are anticipated events, and the pilot waits for the cue to begin the task, then follows the procedure to complete the task.
2. ATC Events – these are the incoming Air Traffic Control (ATC) calls that the pilots must interpret, adjust goal structures and then form and deliver a response. These are often expected events, but the exact time of occurrence and detailed content is not known.
3. Malfunction Events – these are unexpected events. They represent some equipment malfunction in the aircraft. These are of a cautionary nature, and can be immediately threatening to the airworthiness of the aircraft. The pilot knows the correct procedure to respond to and fix these malfunctions.

At each of these events, a probe opportunity exists just before and just after the event. Because of the nature of the intrusive interviewing technique, it is not practical to probe both before and after each of these events during a single scenario, as there was insufficient time to resume the simulator flight scenario. Therefore, only one of the opportunities (before OR after) was probed during the each of the scenarios. (For a complete description of the experimental design, see below.)

As mentioned above, the objective of this study was to identify the factors that pilot use to prioritize tasks. A review of the task management literature found no other studies that have specifically addressed this issue, so this appears to be the first study of its kind. While no data collection has been performed on this topic, the literature suggests factors that drive task prioritization (see Table 3.1).

It was anticipated that if this list was presented to pilots and instructions given to classify which of these factors affect their prioritization scheme, the reply would be a blank stare. Therefore, the approach used in this study was to let the pilots verbalize, in their own words, how and why they prioritized tasks in the simulator environment. The elicitation and interviewing techniques described above were used to merely provide an opportunity for the pilot to verbalize prioritization schemes. These interviews were then analyzed after the interviews, and the pilots’ responses were categorized into prioritization factors.

The present study was essentially performed with hypothesis generation as a primary objective. Virtually no data had been collected in an operational environment on how pilots prioritize the multiple, concurrent tasks that are inherent on the flight deck. It was anticipated that through this experiment, an initial indication of the factors that affect task prioritization would be gained.

The participants for this study were 8 airline pilots, all male, with an average of 7472.5 total flying hours. They had an average of 984.4 hours of single pilot time and 666.9 hours of electronic flight instrument system (EFIS) experience. Their age range was 25 to 44, with an average of 35.6 years. They were recruited on a volunteer basis and were not compensated in any way.

The part-task simulator was the NASA Stone-Soup Simulator version 4.1 obtained from NASA-Ames Research Center. The hardware consisted of 2 SGI Indigo2 workstations, running the IRIX 6.2 operating system. The workstations were networked together, with one serving as the experimenter’s station and the other displaying the simulator interface for the pilot. The simulator flight control was performed with a B&G Flybox and a mouse connected to the pilot’s workstation. Video equipment included a Panasonic 8mm camcorder, Sony PVM-1910 video monitor and Videonics MX-1 video mixer to obtain the picture-in-picture configuration. Video from the pilot’s workstation was collected using an SGI Galileo Video board and software for NSTC video output. Audio equipment included a Panasonic WM-F2040 stereo cassette recorder to record pilot interviews.

Data collection for the experiment consisted of two flight scenarios, designated as Bravo and Sierra (Figure 4.1 and Figure 4.2). These were similar scenarios, yet different enough that pilots could not anticipate ATC instructions for headings, altitudes and airspeeds or timing and type of equipment malfunction events. Subjects were balanced so that half of the subjects ran the Bravo scenario first and half ran the Sierra scenario first.

In each scenario, the elicitation method used was either the retrospective technique or the intrusive technique. If the subject performed the intrusive technique first, then the retrospective technique was used on the second scenario. Subjects were balanced so that half performed the intrusive technique first, while half performed the retrospective technique first.

In each scenario, six events were identified: two procedure events, two ATC events and two malfunction events (see Figures 4-1 and 4-2). Pilots were then probed just before or just after each event. The design was such that if a pilot was probed just before the first event on the first scenario, then he was probed after the first event on the second scenario, and vice versa, thus getting full coverage of the event over two scenarios. This was the case for all six events in each scenario, thus totaling twelve probes over both scenarios.

The experimental design was a full 2 x 2 x 2 factorial design with a full, single replication using 8 subjects. The treatments were: Scenario Order (Bravo/Sierra), Elicitation Technique (Retrospective/Intrusive) and Probe Timing (Before Event/After Event). This was given to the 8 subjects according to Table 4.1.

The tasks performed in the part-task simulator were designed to be consistent with actual flying tasks as much as possible. At the highest level, the mission goal was to fly the simulator the final 100 miles of an arrival into the San Francisco International airport (SFO). Pilots were given the published Big Sur-2 Standard Terminal Arrival Route (STAR) plate and a procedure plate for the San Francisco runway 28-right instrument landing system (SFO 28R ILS).

While the simulator’s displays and behavior was similar to a real aircraft, there were significant differences that required explanation to the pilots. For example, tasks such as dialing radio frequencies and altitude dial settings were accomplished using software buttons manipulated by the mouse instead of the physical knobs as in actual aircraft. The purpose of the explanation was to familiarize the pilots with the display layout and manipulation of the simulator’s various controls (See Figure 4.3).

Primary aircraft control was accomplished by manipulating the joystick for pitch and roll. There were no yaw control devices, such as rudder pedals. No use of automated flight control, such as autopilots, were allowed for this study. However, use of the flight director was required, and was very beneficial in aiding primary aircraft control. The flight director consists of command bars located on the attitude indicator, which direct pilot joystick inputs to accomplish desired heading and pitch for the aircraft. This functionality was used in an attempt to moderate pilot effort in primary control, since joystick input to the simulator was very sensitive.

Navigational equipment was limited to a single very-high-frequency omnirange (VOR) navigational instrument (see Glossary in Appendix 1 for aviation equipment definitions). While this is a very minimal configuration, it was ample for the scenarios and is an accurate partial representation of true navigational tasks. VOR displays and controls consisted of the navigational display, which included a VOR deviation indicator, DME distance from the ground-based VOR, course setting and VOR frequency. The VOR frequency control was located on the navigational radio display, and the course input selector (CRS) was located on the mode control panel.

Communications between the pilot and simulated ATC was performed by direct verbal exchange, as the pilot and experimenter were within approximately 5 feet of each other. Verbal exchanges were restricted to standard radio procedures and there was no free conversation between pilot and experimenter during data collection scenarios. As a small added communications task, the pilot was required to dial in the proper communications radio frequency before an exchange with ATC.

Other system management tasks were performed in the simulator through various controls and displays. For example, equipment malfunctions illuminated the caution indicator and display a message in the Engine Indication and Crew Altering System (EICAS) display area. The equipment malfunctions could then be acknowledged and reset by performing a series of mouse click inputs to the multifunction display/control panel of the simulator.

Pilots were never given clearance to fly the entire STAR, but used the information provided on the STAR to provide reference information regarding navigational waypoints and distances. ATC, which was simulated by the experimenter, provided clearances for the pilots to follow which led them inbound for interception of the ILS navigational aids for final approach.

A task analysis for this study was simplified and refined from the analyses performed by Alter and Regal (1992) and McGuire, et al. (1990). This resulted in the 21 tasks listed in Table 4.2. This is consistent with the standard ANCS taxonomy discussed in Chapter 2.

Pilots arrived for the 2.5-hour experiment and immediately completed an informed consent document (Appendix 2) and pre-trial questionnaire (Appendix 3) to record flight experience, age and to ensure no extenuating circumstances interfered with the trial, such as excessive caffeine or lack of sleep the night before.

Pilots were then given a brief overview of the experiment. They were notified that their flying performance was not being measured in this experiment and that their comments made to the experimenter would be separated from their names and would be kept confidential. The sequence of the experiment consisted of approximately 45 minutes of training on the simulator followed by 2 data-collection scenarios flown on the simulator. Each pilot flew a scenario using the intrusive elicitation technique and a scenario using the retrospective technique. The order of the application of techniques was determined by the experimental design outlined above.

The training consisted of 2 or more flights. The initial flight took a very informal form, with the experimenter introducing each of the displays and controls while directing the pilot to fly certain headings and altitudes. During this time, the pilot was free to ask questions about the simulator, as he became familiar with the location and format of information related to piloting the simulated aircraft. After approximately 15 minutes for the initial flight, the simulator was reset, and another flight was initiated. During the second flight, navigation information was given to the pilot in the form of ATC instructions. However, the pilot was still free to ask questions and, if required, the simulator was paused to explain more about operation of the simulator. During these flights, aircraft configuration checklists and equipment malfunction procedures were covered and practiced several times. After completion of the second training flight, the pilot was asked if he felt he required more training to become comfortable with the simulator. On occasion, a pilot requested an additional run through a particular portion of the training flight, and he was accommodated. Each of these training flights was loosely associated with the Big Sur-2 arrival so the pilot could become familiar with the navigational waypoints and DME distances, but the Bravo or Sierra data collection scenarios were never duplicated.

After training, pilots were given the opportunity for a short break. Next, a general description of the data collection scenario was presented, explaining how each of the interviewing technique was applied (retrospective using a videotape playback, and intrusive, immediately interrupting the scenario). The following instructions were given to the pilot:

The initial data collection scenario then began. Once again, the order of the Sierra/Bravo scenarios and Intrusive/Retrospective methods were predetermined according to the experimental design described above.

For the retrospective technique, the entire flight scenario was flown uninterrupted and videotaped. During the scenario, the experimenter recorded the precise simulator time that each of the events occurred so that the videotape could be paused upon review with the pilot. Upon completion of the scenario, the pilot was repositioned to view the video monitor and a replay of the just-completed scenario was started. As the videotape playback approached each of the event times, the pilot was alerted that a probe would be initiated very shortly. The 30 seconds of videotape preceding the event was reviewed three times and after the third time, the videotape was paused and a probe was performed.

The probe began in a structured manner, asking specific questions about what tasks the pilot was performing when the videotape was paused. The first part of the probe was to establish the task currently attended, the task that was to be attended to next, and a list of remaining tasks that were currently active in the pilot’s memory. Following this identification of the pilot’s current task list, a series of questions were posed that probed why the current task had a higher priority than the other tasks listed. It was during this portion of the probe that the techniques of the cognitive interview were employed in an attempt to retrieve as much insightful and detailed information from the pilot as possible as to what factors influenced the prioritization of tasks.

The initial dialog of the probe was presented to the pilots as follows:

As was discussed earlier, the cognitive interview allows for deeper probing on certain items reported by the subject. This concept was employed as often as possible in an attempt to get as much information from the pilot as possible regarding prioritization strategies.

For the scenarios that were performed using the intrusive technique, the probing technique was identical. The difference in the intrusive technique is that upon reaching the events in the scenario, the simulator was stopped and the probe initiated immediately, within several seconds of the pilot actually flying the simulator. All probes were recorded using a cassette recorder for further analysis and review at a later time.

After the pilot had completed the second scenario, he was immediately given a post-test questionnaire (Appendix 4), inquiring about how comfortable the pilot was with flying the scenario and if the training was adequate for testing purposes. Additionally, the pilot was encouraged to discuss his thoughts and feelings about anything related to the experiment. This completed the experiment.

The cognitive interviews of the pilots resulted in approximately 7 hours of audio tape. Although initial identification of tasks and prioritization factors was performed during the interviews, the experimenter performed a more thorough analysis by reviewing each probe response several times. This post analysis resulted in minor adjustments to the task classification, but significant changes to the classification of prioritization factors.

The process of analyzing the audio tapes for task prioritization factors was non-trivial. The probes were reviewed with the objective of determining all of the prioritization factors that pilots reported. After several iterations, the number of factors was set at 12, as this was the minimum number of factors that captured all factors reported by the pilots.

Next, each probe was reviewed in an attempt to determine how long the probe lasted, how detailed the pilot was during the probe and how specific the pilot was regarding prioritization factors. In subsequent replays, the pilots’ responses were then categorized into one of the twelve factors. It should be noted that every attempt was made to classify the pilots’ responses according to what was said during the probe, and not what was inferred by the experimenter. In other words, the analysis classified what the pilot actually verbalized and no attempt was made to infer beyond what the pilots said.

The analysis described above resulted in the identification of 12 factors that affect prioritization (See Table 4.3). The factors in the table include generalized descriptions of the factors, including common quotes from pilot responses. Recall that the task management literature suggested 13 possible factors. However, this study, in essence, began with a clean slate. The factors identified were solely a result of the analysis and not an attempt to fit the pilot’s responses into a pre-determined factor classification.

Table 4.4 presents the factors that affect task prioritization in the order of most reported to least reported. Two factors that clearly emerged were status with a total of 51 instances (30%) reported and procedure with 48 instances (28%) reported. In the middle range of frequently reported factors was verifying information, reported 13 times (8%) and importance, reported 12 times (7%). The remaining factors were reported less frequently (see Figure 4.4).

Due to the nature of the cognitive interviewing technique, traditional analysis of variance on the reported prioritization factors was not appropriate (Montgomery, 1997; Ostle, 1963). Therefore, an analysis was performed to determine if the frequency with which each factor was reported showed statistically significant differences from the other factors. A chi-square test for counted data was applied to the results presented above. In this analysis, the null hypothesis states that all factors are equally likely. A rejection of the null hypothesis accepts that the observed frequencies are not equally likely in each of the 12 factor categories (Devore, 1987).

The chi-square statistic from this analysis had a value of 223.02 with 11 degrees of freedom, resulting in rejection of the null hypothesis (p < .001). In other words, there is very strong evidence that the observed values do not come from a distribution where frequencies in each factor category are equally likely.

Since the status and procedure factors had such high frequencies compared to the other prioritization factors, these two factors were removed and a the chi-square test was performed on the remaining 10 factors. This test had a chi-square value of 18.00 with 9 degrees of freedom, resulting in rejection of the null hypothesis (p < .05). Thus, even with the high frequency factors removed, there is strong evidence that the remaining factors are not equally likely.

The prioritization factors broken down by the elicitation technique is presented in Table 4.5. A primary question of this study was to determine if the intrusive and retrospective techniques give significantly different results in data collection. In order to determine if there was a significant difference in the frequency of factors reported between the two techniques, a chi-square test was applied to the 2 x 9 matrix in Table 4.5.

In this application of the chi-square test, the null hypothesis is the assumption that the two rows come from the same distribution. If we fail to reject the null hypothesis, then the conclusion is made that the two rows are not significantly different.

The chi-square statistic from this analysis had a value of 4.36 (df = 11, p = 0.9583). Thus we fail to reject the null hypothesis and we conclude that the intrusive and retrospective elicitation techniques do not result in different results.

Another concern with the nature of this experiment is the use of the three event types (scenario, ATC and malfunction). Perhaps the prioritization factors depended upon which event was probed. Table 4.6 presents the summary of prioritization factors reported by event type.

The chi-square statistic from the individual breakdown of the 6 events was 62.35 (df = 55, p = .2312). Thus, we fail to reject the null hypothesis and conclude that there is no statistical difference in the reporting of prioritization factors between the 6 events.

Additionally, the chi-square statistic from the three event types was 26.13 (df = 22, p = .2461). Again, we fail to reject the null hypothesis, and conclude that the prioritization factors are independent of the event type.

To determine if the two different scenarios had an effect on prioritization factors reported by the pilots, the frequencies were summed over the Bravo and Sierra scenarios (see Table 4.7).

The chi-square statistic from this analysis had a value of 6.70 (df = 11, p = 0.8228). We fail to reject the null hypothesis and conclude that there is no statistical difference between the bravo and sierra scenarios.

Half of the probes were performed before a critical event and half after the event. To determine if the timing of the probe had an effect on the reporting of prioritization factors, the data was organized into Table 4.8, showing the timing of the probes before and after an event.

The chi-square statistic from this analysis had a value of 13.10 (df = 11, p = 0.2868). Again, we reject the null hypothesis and conclude that there is no statistical difference between the before and after event timing of the probe.

This study was an initial attempt to identify the factors that affect task prioritization. The task environment required pilots to fly realistic, published arrivals in a part-task simulator. The method of prioritization factor elicitation was the cognitive interview using either the retrospective or intrusive technique.

Status and procedure emerged as the two factors most reported by pilots. Additionally, verifying information, task importance and consequences of not performing task showed a substantial frequency of reporting. Statistical tests confirmed that significant differences in the frequency of factor reporting were present and that there were no significant differences in the frequency of reporting for the elicitation technique (intrusive/retrospective), scenario flown (Bravo/Sierra) or the timing of the probe questioning (before/after).

Data analysis for this study presented a very challenging task for the author. The nature of the data collected was rich with insights into how pilots prioritize tasks, perform multiple concurrent tasks and, in general, make decisions on the flight deck. However, with such rich and complex data, interpretation and classification of pilot responses is, to say the least, very difficult.

In the next several paragraphs, a more detailed description of the prioritization factors is presented. While none of these descriptions are direct quotes from the pilots, they represent the pilot’s justifications of how task prioritization was accomplished. They are presented in rank order of number of times reported.

Status – The perceived status of the current task was unsatisfactory, so it was currently being performed to bring its status to a satisfactory level. For instance, the pilot may have an assigned altitude of 15,000 feet. If the pilot were to find himself at an altitude of 100 feet or more below the clearance, it would result in the pilot consciously focusing on the task to reduce the deviation and get back to the assigned altitude. Similarly, if the status of high priority tasks, such as the aviate tasks, were satisfactory, then attention could be allocated to other, non-critical tasks.

Procedure – The reason the current task was being performed was because it was the proper task to execute in the current context. For instance, if an ATC instruction was given to descend to a particular altitude then the task was performed immediately. The relationship between ATC and pilots is one that ATC issues clearances and pilots follow those clearances. Similarly, when the filed flight plan required a turn at a navigational waypoint, then when that waypoint was reached, the turn was initiated. It was the procedure that needed to be followed in order to perform the duties of a pilot.

Verifying Information – Often, a pilot will perform a task, then immediately perform an additional task with the purpose of verifying that the initial task was accomplished. For instance, during the arrival, a pilot may initiate and complete a turn at a particular navigational waypoint. Upon completion of the turn, he will switch his attention from the Primary Fight Display (PFD), to the Navigational Display (ND) to verify that the information regarding his heading obtained from the PFD is verified by the information on the ND. It could be argued that this sequence of events is in fact all part of the same task (monitoring and controlling heading). However, the approach taken in this study assumed that a change from one display to another constituted a task switch. Ultimately, this level of specificity in the task analysis would address this concern. In the task analysis adopted for this study, the above example would represent a move from a primary aviate task to a task classified under the navigate category.

Importance – The importance of a task, relative to other competing tasks, determines where a pilot will focus his attention. There were many instances were the pilot reported that he was "flying the airplane first." In other words, he was attending to the primary aviate tasks of monitoring and controlling heading, altitude and speed which took precedence over other tasks, such as performing checklists, planning ahead or determining current position on the arrival. Pilots are very familiar with the ANCS ordering of tasks and strive to adhere to its hierarchy. It should be noted that a pilot’s response would not be categorized in this factor unless the pilot explicitly stated, "this task was more important than the others."

Consequences - Pilots reported the consequences of not performing the task and safety considerations associated with performing tasks. For instance, the consequences of not maintaining the assigned altitude could result in conflict with other aircraft or terrain.

Rate of change - When the status of a task is currently changing at a significant rate, the pilot tends to stay with that task until it is once again stabilized. For example, in a turn, where the heading of the aircraft is changing, pilots will closely monitor the progress of the task until they finish the turn and level the wings. They realize that if they were to divert their attention to another task, that upon returning to the task, it may have progressed to a status that is very unsatisfactory (a turn past their desired heading).

Time/Effort Required – Pilots reported that they evaluated the time and/or effort required to perform tasks and made decisions regarding which task to perform upon these evaluations. It is interesting to note that some pilots selected the quick/easy task to perform first, while others selected the task that would take considerable time/effort to perform first.

Salience of Stimulus – When there is a sudden, obvious change in a visual display, the pilots often switched their attention to the change, and at least, acknowledged its presence. For instance, this might occur when an equipment malfunction event was activated and the yellow caution light was illuminated along with a message on the EICAS display. It was interesting to note that the switch of attention was often very quick, to acknowledge the change, but then their attention was directed right back again to the task that was in progress when the visual change occurred. In the interviews, they often talked about their conscious evaluation of the meaning of the change, their realization that the priority of the associated task was not greater than the currently active task, so they returned to the original task.

Urgency – The urgency of a task is defined as the time it will take to complete a task in relation to the time until the task needs to be completed. Pilots are very aware of high workload situations, and continually try to "stay ahead of the airplane" by performing tasks as early as possible. However, in certain situations, tasks cannot be performed early and thus, take on urgency. Pilots reported currently performing a task because it needed to be completed in the very near future.

Needed information – Pilots reported the reason they were attending to a particular task was that its displays contained needed information. For example, when pilots were probed as to why they were monitoring the attitude indicator, their reply would be that it was the source information they needed in order to maintain heading and altitude.

Resist Forgetting – This factor was directly related to compliance with ATC clearances. Upon receiving an instruction from ATC, the pilots immediately began to give control inputs to the aircraft (usually the mode control panel), even before acknowledging the clearance with ATC. When prompted as to why they began immediate inputs, their reply was that they tended to forget clearances unless they immediately input the clearances into the mode control panel.

Expectancy – Pilots often tried to perform tasks well in advance of when they needed to be performed. Their justification for attending to such a task was that they were expecting a high workload in the near future, so they were attempting to get as much as possible done early so that they had more time to deal with the tasks during the high workload period. This is very consistent with strategic task management (Schutte and Trujillo, 1996), strategic workload management (Hart, 1989; Adams, et al, 1991; Raby and Wickens, 1994) and occurs in times of low workload, which was not frequency encountered in this experiment.

Recall from the introduction that although no studies have produced data on the prioritization of flight tasks, the literature suggests 13 possible factors that affect task prioritization. The current section evaluates the commonality between those predictions and what was empirically collected in this study.

Table 4.9 compares the predicted factors with the factors reported by the pilots in this study. It is reassuring to find that the most reported factor, status, was in fact predicted in the literature along with several other factors, such as importance, time/effort required and a few other less frequently reported factors.

However, a factor not predicted, procedure, was very apparent in the pilot’s reporting of prioritization factors. This is puzzling because for pilots, this was a very intuitive and obvious justification as to what task was currently being performed. During a probe, when the pilot was performing a task that was required to fly the scenario, such as turning the aircraft to the proper heading at a navigational waypoint, the pilot’s justification for performing that task was to the effect, "The current situation requires that I perform this task right now." When probed deeper, often the pilots could not verbalize any other reason for attending to the task other than the procedure of flying demanded that the task be performed at that point in time.

Another factor, importance, predicted in the literature, was explicitly reported only 7% in this study. However, the ANCS task hierarchy is well known by pilots, and practiced rigorously. In fact, during the probes, the primary aviate tasks were reported as the current task more than 80% of the time. It was surprising that the importance factor did not appear more frequently in the probes. Perhaps this factor is such a significant part of a pilot’s routine behaviors and decision making that it has taken on tacit knowledge characteristics, and it not consciously processed by pilots. Therefore, although it may affect how a pilot prioritizes tasks, he does not explicitly realize its influence and is therefore unable to report it verbally during a probe. Again, it should be noted that during data analysis, pilots would have to report that one task was "more important" than another task for the importance factor to be identified.

In the present section, the effort is directed at given meaning to the factors reported by the pilots. Upon reflection on the prioritization factors presented earlier, it is readily apparent that there exist relationships between the factors. Recall, however, that the objective of the data analysis was to classify what the pilots reported and not an attempt to infer what the pilots really meant.

The single most reported factor in this study was status. Pilots reported that the current unsatisfactory (or satisfactory) status of a task at least partially determined why the current task was the focus of attention. Rate of change of task status was reported as a separate factor, but this is closely related to status. It is known by the pilots that if they were to divert attention away from a task while its status is rapidly changing, it is possible that upon returning to the task, it will have an unsatisfactory status.

Verifying information is a crosscheck to reassure that what one source of status information reports is consistent with another information source. Therefore, verifying information is consistent with the status factor.

On the flight deck, many of the displays do not have excessively salient stimuli. For instance, the altitude display on the Primary Flight Display (PFD) is continually changing and appears very much the same if the altitude reading is 35,000 ft. or 100 ft. However, the status and warning indicators and EICAS message display areas are designed with the intent of capturing attention when information is available that the pilot needs to be informed about, such as exceeding aircraft limitations or equipment malfunctions. If the aircraft were to descent below 500 ft. without being configured for landing (gear down, flaps extended, etc.), the altitude display on the PFD merely displayed the altitude. However, the red warning light would flash and an EICAS message that alerts the pilot to an unsafe condition was displayed. In essence, the warning light and EICAS message are delivering status information about the aircraft or subsystems, so the factor salience of stimulus is related to the status factor.

Urgency also appears to be related to status. When a task is very urgent, its status will need to be complete or satisfactory in a relatively short time. As an extreme example, if vertical speed of the aircraft is –1500 ft./min. and the altitude of the aircraft is 1500 ft. above ground level, the status of the altitude task is satisfactory for the moment. However, after 30 seconds, the urgency of the altitude task is beginning to increase. If the pilot does not respond and give control inputs rather quickly, then the status of the altitude task will become severely unsatisfactory in a very short time.

Finally, needed information, although reported very infrequently also appears to be status related. When a pilot stated that that he was attending to a task because it was the source of needed information, he was after the information to determine a status parameter of the aircraft.

So the factors rate of change, verifying information, salience of stimulus, urgency and needed information are all related to the general prioritization factor status and is visually depicted in Figure 4.5.

During the interviews, pilots reported that the reason they were performing the current task was because it was the correct procedure to execute at the current time and in the current context. This appears to be a very intuitive and obvious justification of current task focus, however, upon closer examination, this may be the most complex factor identified in this study.

There are at least two dimensions to the procedure factor. First, there is an externally driven dimension, where the pilot waits for external cues from the environment to initiate a task. For example, if the pilot has an ATC instruction to turn the aircraft to a heading when it reaches a navigational waypoint, then when the aircraft reaches the waypoint, the pilot initiates the procedure and turns the aircraft. He follows the procedure when the environmental cue is encountered.

However, the representation of procedural knowledge in the mind is very much an internal dimension of the procedure factor. Exactly how humans represent, store and recall this type of knowledge is a research area with much activity, yet there is no consistent agreement on how this process functions (Anderson and Lebiere, 1999). Additionally, interactions between procedural knowledge and memory systems (long term and short term) do not have exact, widely accepted theory (NRC, 1998). To understand this internal dimension might equate to an overall theory of human cognitive processing.

Another factor reported by pilots was the expectancy of upcoming events. The pilots were searching for opportunities to perform anticipated procedures at an earlier time in the flight scenario. This suggests a link to the procedure factor.

The resist forgetting factor and the time/effort factor appear closely linked for the following reason. In the simulator, when the pilot was issued a clearance by ATC (simulated by the experimenter), he was handed a slip of paper with the clearance. The purpose of the slip was to allow the pilot to continue flying and not require him to write down clearances. If the pilot were to forget the clearance, he merely needed to glance down at the slip to again read the clearance. So the pilot’s justification of performing the task to avoid forgetting it is really an attempt at reducing the time and effort of performing the task. Similarly on the actual flight deck, the pilot has several resources available if he were to forget a clearance. He could ask his copilot for the information (small time/effort investment) or he could contact ATC again to obtain the information (a substantial time/effort investment).

Further, the time/effort factor is an evaluation of the procedure to be performed and an estimation of the requirements to perform the task. Therefore, it is suggested that the time/effort factor is related to the procedure factor.

Therefore, a second general factor labeled status is suggested, with the sub-factors expectancy, time/effort required and resists forgetting configured as in Figure 4.6.

The two factors importance and consequences of not performing a task are closely related. To a pilot, a task is important if failing to perform it jeopardizes the safety of the aircraft or passengers, violates FAA regulations or fails to comply with ATC clearances. When the pilots reported the consequences of not performing a task in the interviews, they were mostly concerned with safety and non-compliance implications.

The general prioritization factor proposed that encompass importance and consequences is labeled value, and presented in Figure 4.7. It can be conceptualized as the value or worth of performing the task towards reaching the goal or subgoals of the flight.

For example, if the overall mission goal of a flight is to deliver the passengers to the intended destination safely, quickly and comfortably, then subgoals of the mission might be decomposed into such goals as comply with ATC clearances and avoid exceeding aircraft performance limitations. Using the general prioritization factor then, pilots consider the value of performing a task as they determine their prioritization strategy while flying.

Similar to the general procedure factor, the value factor has internal representations that are not obvious or sharply defined. For example, when a pilot is challenged with the decision to comply with an ATC-instructed altitude change and a significant equipment malfunction simultaneously, which has a higher value? Failure to comply with ATC may result in an unsafe conflict with other aircraft or terrain, but failure to deal with a critical equipment malfunction may also escalate into a situation where the aircraft becomes no longer flyable. It is in these situations that a pilot’s knowledge, including training and experience, emerge to assign value to the tasks and prioritize in such a way as to meet the overall mission goals.

The discussion presented above identifies three general categories of prioritization factors: status, procedure and value. These three general categories incorporate all twelve of the specific factors identified in the interviews of the pilots.

While three distinct general factors emerge, there appears to be a logical relationship between them. The status factor represents the current state of the world. The value factor represents the desired state of the world or simply the goal state. The goal state is the pilot’s understanding and internal representation of what state the system needs to be in for the mission goals to be met. Finally, the general procedure factor is the link between the current state and the goal state. In other words, the pilot, operating the system in the current state, uses procedures to obtain the goal state. Figure 4.8 graphically presents this model of task prioritization.

In this model, the task prioritization process is driven by three general factors status (52%), procedure (35%) and value (13%). It should be noted that the weighting of the factors is preliminary and is based on the relatively limited data collected in this experiment.

The relationship between the value and procedure factors may be closely intertwined. In a normative theory, the development of the procedures is based on the value of performing the tasks. In other words, the procedures are a sequence of tasks that are performed in the order of the highest value. Therefore, the value of performing the task is inherent to the procedures that pilots perform on the flight deck. To a certain extent, the pilots realize this, and they are confident that if they perform the procedures the important tasks will be attended to and the consequences of not performing tasks will be minimized. Simply stated, the pilots trust the procedures.

A fundamental limitation of this study deals with the nature of task prioritization. In this study, we relied on the pilot’s verbalization of the prioritization process to obtain a sense of how task prioritization is accomplished. If, in fact, prioritization is an accessible, explicit behavior, than it is likely that this study was able to record and analyze at least part of those factors that affect task prioritization. However, if task prioritization is truly an implicit, unconscious process that pilots are not able to explicitly identify, then this study falls short of its expectations. Until researchers make further gains in an overall better understanding of human cognition and are able to identify which mental processes are explicitly accessible, this fundamental question will go unanswered. At this time, we continue with the assumption that task prioritization is, at least, partially an explicit process that pilots can identify and discuss.

Another limitation is the possibility of experimenter bias. The entire study was developed, performed and analyzed by the author. Every attempt was made to approach all data collection and analysis from a neutral perspective and not to succumb to a confirmation bias associated with expectancies of task prioritization factors. The author is confident in his neutral approach, however only additional research that generates supportive evidence of the findings in this experiment would truly eliminate this possible limitation.

As with all laboratory experiments, this study had limitations of ecological validity. It is true that the simulator used in this study was a single pilot, part-task simulator that differs greatly from the real world flight deck, which includes two pilots in an operational setting. However, the objective of the simulator was to put the pilots in the frame of mind of flying a published arrival and to challenge them with real-world thought processes and decision making. In the post-test questionnaire, each pilot expressed that the part-task simulator was successful in that respect, and they found themselves thinking like a pilot and performing the tasks present in the operational environment.

One opportunity for improvement associated with the simulator environment would be to present more challenging instances where task prioritization occurs. In this study, there were relatively few instances where the pilot was truly overloaded and had to prioritize more than 3 tasks. In other words, workload levels were always manageable and the consequences associated with the relatively minor equipment malfunctions were too insignificant. For future experiments, it is suggested that more non-aviate tasks be placed on the pilot in addition to more severe equipment malfunctions, where the consequences might be much more detrimental than in the current experiment.

The primary objective of this study was to identify factors that affect prioritization. Analysis of the pilot interviews resulted in 12 factors that were subsequently reduced to the three general prioritization factors of status, procedure and value. These factors formed the basis of a model of task prioritization (Figure 4.8).

With this model, we believe we have at least a preliminary understanding of task prioritization. However, because this study was essentially a hypothesis generation exercise, we are now tasked with developing further studies to support and refine the findings presented here.

The ultimate goal of the present research is twofold. First, with a better understanding of the prioritization process, perhaps training techniques can be developed that explain this process to pilots and explicitly emphasize how experts perform task prioritization, with the objective of modifying inexperienced pilot behavior to be more consistent with the behaviors observed in experienced pilots.

Secondly, the realization that the general status factor had such a major influence on the prioritization process is supportive of use of pilot aids that assist the pilot in enhancing situation awareness of the current state of the aircraft. Even if these aids are not destined for use in the operational flight deck, again, perhaps they can be employed in a training role to help pilots better prioritize tasks.

A secondary, methodological objective of this study was to evaluate elicitation techniques. In this study, we used both the intrusive and retrospective techniques and could determine no difference in the data that was collected with the techniques. Therefore, we conclude that in future studies, we can elect to utilize the technique that will best facilitate the study.

What's New

Listed below, most recent first, are changes made to this page since its creation.

23 Dec 99

• original page