NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

National Research Council (US) Committee on Opportunities in Neuroscience for Future Army Applications. Opportunities in Neuroscience for Future Army Applications. Washington (DC): National Academies Press (US); 2009.

Cover of Opportunities in Neuroscience for Future Army Applications

Opportunities in Neuroscience for Future Army Applications.

Show details

Appendix DResearch on Managing Information Overload in Soldiers Under Stress

This appendix provides background information on research in augmented cognition (AugCog) conducted by the Defense Advanced Research Projects Agency (DARPA) and the U.S. Army and discusses several of the neuroscience research and engineering challenges.

THE DARPA AUGCOG PROGRAM

DARPA began its research program in the area of augmented cognition in 2001 with a focus on challenges and opportunities presented by the real-time monitoring of cognitive state with physiological sensors (Augmented Cognition International Society, 2008a). The program, now known as the Improving Warfighter Information Intake Under Stress Program, was initially called the Augmented Cognition program, or AugCog. At first it had two stated purposes: (1) to gain battlefield information superiority and (2) neurology-related clinical applications. Linked to the idea of network-centric warfare, the stated rationale for AugCog was that “order of magnitude increases in available, net thinking power resulting from linked human-machine dyads will provide such clear information superiority that few rational individuals or organizations would challenge under the consequences of mortality” (Schmorrow and McBride, 2004). Another early AugCog objective was stated as the enhancement of a single operator’s capabilities such that he or she could carry out the functions of three or more individuals (McDowell, 2002).

It was proposed that information superiority could be achieved by enhancing an operator’s cognitive abilities and by measuring neurological and physiologic variables. One aspect of the enhancement would involve tracking markers of cognitive state, such as respiration, heart rate, or eye movement, using devices for electroencephalography (EEG) or functional optical imaging. The operator would wear a headset that contained such devices and might be connected to other devices as well—perhaps galvanic skin response sensors or pressure sensors in a seat—interacting with them by means of a traditional computer interface device such as a mouse or a joystick. Another aspect of enhancing cognitive ability would be dynamic control of the rate as well as the source of information by a “communications scheduler.” The former DARPA program manager envisioned that technologies developed by the program would be operational within 10 years. He predicted that within 20 years, the technology would be woven into the fabric of our daily lives (Augmented Cognition International Society, 2008b).

Proof-of-concept work for AugCog occurred in two phases. In Phase 1, researchers attempted to detect changes in cognitive activity in near real time in an operationally relevant setting. One relatively large demonstration, called the Technical Integration Experiment, was conducted with mixed results (St. John et al., 2003). Its objective was to determine which psychophysiological measurements could consistently detect changes in cognitive activity during a supervisory control task. Using 20 gauges of cognitive state (CSGs) such as EEG, functional near infrared (fNIR), and body posture, as well as input device measures (mouse pressure and mouse clicks), eight participants completed a series of four simplified aircraft monitoring and threat response tasks in 1 hour. Eleven of the CSGs, including fNIR and EEG measures, were reported to be significant and reliable for at least one independent variable, but no CSG was significant and reliable across all of three independent variables (St. John et al., 2004). Two CSGs, mouse clicks and mouse pressure, demonstrated statistically significant results across two of the independent variables, but these were probably already highly correlated.

There were significant problems with technical integration, several of which were acknowledged by St. John and his colleagues in Technical Integration Experiment (2004). These included questions about the study design constructs and their external validity, the statistical methods applied to the data, important data missing from the report of results, and the familiar problem in psychophysiological research of noisy data (substantial variability across test participants, test runs, etc.). Given the number and severity of confounding factors, the results of the study are preliminary at best and in no way constitute unequivocal scientific evidence that the CSGs identified as statistically significant can effectively detect change in cognitive activity in a complex human supervisory control task.

A second set of four experiments was conducted in Phase 2 of AugCog. The stated objective of Phase 2 was to manipulate an operator’s cognitive state as a result of near-real-time psychophysiological measurements (Dorneich et al., 2005). The experiments used a video game environment to simulate military operations in urban terrain (MOUT) in either a desktop setting or a motion-capture laboratory. In addition to the primary task, navigating through the MOUT, participants had to distinguish friends from foes while monitoring and responding to communications. A communications scheduler, part of the Honeywell Joint Human–Automation Augmented Cognition System,1 determined operator workload via a cognitive state profile (CSP) and prioritized incoming messages accordingly. The CSP was an amalgam of signals from cardiac interbeat interval, heart rate, pupil diameter, EEG P300, cardiac quasi-random signal (QRS), and EEG power at the frontal (FCZ) and central midline (CPZ) sites (Dorneich et al., 2005).

As in the Phase 1 experiment, there were only a few participants (16 or fewer) in each of the four Phase 2 experiments. Construct validity and statistical models were questionable, with significant experimental confounds. There is no open account of how the neurological and physiological variables were combined to form the CSP, making independent peer-researched replication of these experiments difficult. In light of these concerns, claims such as a 100 percent improvement in message comprehension, a 125 percent improvement in situation awareness, a 150 percent increase in working memory, and a more than 350 percent improvement in survivability should be considered tentative. In addition, the authors claim anecdotal evidence that their CSGs can indicate operator inability to comprehend a message (Dorneich et al., 2005).

The focus in these experiments appears to have been on generating measurable outcomes on a very tight time schedule. Most of the technical data on the performance of the actual sensors and of the signal processing and combination algorithms were not published. This information would have been useful for further scientific evaluation and confirmation of the reported results.

The problem with AugCog as a development lies less in the intrinsic concept of managing cognitive workload through neural and physiological feedback to a smart information system than in the assumptions that were made about the maturity of the technologies required to implement such a system and thus about the time frame for an initial operational capability. Unfortunately, no follow-on studies have reported how the successful CSGs could or would be combined in an operational system. The engineering obstacles to combining EEG, fNIR, and eye-tracking devices are substantial. Unless dramatic leaps are made soon in the miniaturization of these technologies and in improved signal-processing algorithms, the realization of a single headset that can combine all—or even a subset—of these technologies is at least a decade away.

Other engineering problems, such as how to measure EEG signals in a dynamic, noisy environment, have not been addressed, at least in the open literature. Basic sensor system engineering problems like these will be critical to any operational deployment of these technologies. A similar engineering problem underlay the use of the eye-tracking devices assumed for AugCog applications. These devices currently require a sophisticated head-tracking device in addition to the eye-tracking device, and encapsulating this technology into an unobtrusive device that can be worn in the field appears also to be at least 10 years in the future.

In addition to hardware limitations on the use of neural and physiological technologies in an operational field setting, the software/hardware suite required to interpret cognitive state reliably in real time is beyond current capabilities, particularly in the highly dynamic, stochastic settings typical of command-and-control environments. The experiments for the AugCog program were conducted under controlled laboratory conditions. While this is to be expected for preliminary, proof-of-concept studies, such a limitation constrains the extrapolation of the reported results. For example, the communications scheduler in the Phase 2 experiments made changes in information presentation based on gross differences in perceived cognitive state. In actual battlefield conditions, the amount of task-relevant information and the degrees of freedom in cognitive state will require more precision and reliability in ascertaining an operator’s condition and making situation-appropriate adjustments rather than limiting access, perhaps inappropriately, to information that may be critical for a real-time decision. Not only must the sensors and signal-processing algorithms improve substantially; significant advances are also needed in decision-theoretic modeling. In particular, these models will have to accommodate a significant range of individual variability.

Overall, the AugCog goal of enhancing operator performance through psychophysiological sensing and automation-based reasoning is desirable but faces major challenges as an active information filter. Suppose a system is implemented that can change information streams and decrease the volume by filtering incoming information presented to a user. How is the system to know that its filtering in a specific situation is both helpful to this user and passes along the correct information for the current situation? The system software must correctly determine an optimal cognitive load for an individual in a dynamic, highly uncertain context and decide which information to emphasize and which information to minimize or even filter out altogether. The problem is that, in command-and-control settings, there are no general principles for what information is truly optimal. Before deploying such an active-filter system, which controls inputs to a military operator, rigorous testing and validation would have to demonstrate that at the least, it does no worse than an unaided soldier.

ARMY FOLLOW-ON WORK

When the DARPA AugCog program formally ended, the Army continued working with portions of the concept at the U.S. Army Natick Soldier Research, Development and Engineering Center. The original goal of the Army effort was to incorporate AugCog technology into the Army Future Force Warrior Program by 2007 (U.S. Army, 2005). The primary focus of the effort has since changed from operational to training applications; moreover, the technology focus has narrowed to the use of EEG and electrocardiography sensors instead of the array of neurological and physiological sensors envisioned for the DARPA AugCog scheme (Boland, 2008).

An experiment was conducted to extend the previous DARPA effort, directed by the same Honeywell team that performed the set of experiments described above. This Army-sponsored experiment focused on developing an experimental test bed for mobile cognitive-state classification and testing it in a dismounted-soldier field setting using the previously discussed communications scheduler (Dorneich et al., 2007). The authors developed an EEG headset connected to a laptop computer worn in a backpack by the test subject. This laptop supported the signal processing algorithms, the communications scheduler, and other experimental testing elements. For the experiment, eight subjects with no military experience completed a 1-hour navigation and communication task with a handheld radio, a personal digital assistant, and a 35-lb backpack. The authors reported that, with the communications scheduler prioritizing messages based on whether the subjects were in a low-task-load or high-task-load condition, mission performance metrics improved from 68 percent to 96 percent with cognitive-state mitigation.

As with the earlier experiments, significant confounds limited the validity of the results. Problems were reported with movement-induced signal noise, as well as significant loss of data that reduced the subject pool to just four individuals for a portion of the experiment. In addition, the approach to classifying cognitive state was extremely limited in state estimation (i.e., costs of actions were not considered), and it depended on relatively short temporal gaps between training and testing (Dorneich et al., 2007). This latter constraint in an operational setting means that soldiers would require extended training to “condition” the system before each mission. Since actual combat never follows a carefully planned script, an issue yet to be addressed is how a priori classification training can ever be based on events that are real enough to give reliable results. The ultimate problem is not that the information filter might fail to accurately gauge the cognitive state of the user, but that it might act in a way that results in a bad decision.

REFERENCES

  • Augmented Cognition International Society. 2008. a. History: Emergence of augmented cognition. Available at http://www​.augmentedcognition​.com/history.htm. Last accessed August 17, 2008.
  • Augmented Cognition International Society. 2008. b. Frequently Asked Questions. Available at http://www​.augmentedcognition​.org/faq.htm#q2. Last accessed August 17, 2008.
  • Boland, R. 2008. Army uses advanced systems to understand what soldiers know. Signal. Fairfax, Va.: Armed Forces Communications and Electronics Association.
  • Dorneich, M.C., P.M. Ververs, M. Santosh, and S.D. Whitlow. 2005. A joint human–automation cognitive system to support rapid decision-making in hostile environments. Pp. 2390- 2395 in IEEE International Conference on Systems, Man and Cybernetics, Vol. 3. Los Alamitos, Calif.: IEEE Publications Office.
  • Dorneich, M.C., S.D. Whitlow, S. Mathan, P.M. Ververs, D. Erodogmus, A. Adami, M. Pavel, and T. Lan. 2007. Supporting real-time cognitive state classification on a mobile system. Journal of Cognitive Engineering and Decision Making 1(3): 240-270.
  • McDowell, P. 2002. The MOVES Institute’s Context Machine Project. Available online at http://www​.movesinstitute​.org/~mcdowell/augCog/. Last accessed May 12, 2008.
  • Schmorrow, D., and D. McBride. 2004. Introduction: Special issue on augmented cognition. International Journal of Human-Computer Interaction 17(2): 127-130.
  • St. John, M., D.A. Kobus, and J.G. Morrison. 2003. DARPA Augmented Cognition Technical Integration Experiment (TIED). Technical Report 1905, December. San Diego, Calif.: U.S. Navy Space and Naval Warfare Systems Center.
  • St. John, M., D.A. Kobus, J.G. Morrison, and D. Schmorrow. 2004. Overview of the DARPA Augmented Cognition technical integration experiment. International Journal of Human–Computer Interaction 17(2), 131-149.
  • U.S. Army. 2005. Mission: Augmented cognition. The Warrior. Natick, Mass.: Public Affairs Office, U.S. Army Natick Soldier Research, Development and Engineering Center.

Footnotes

1

Honeywell was the prime contractor for the DARPA AugCog program and remains the prime contractor for the follow-on Army program in augmented cognition.

Copyright 2009 by the National Academy of Sciences. All rights reserved.
Bookshelf ID: NBK207983

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (3.0M)

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...