The material was assigned a clearance of CLEARED on 18 Nov 2022.
Beavercreek, Ohio – Surveillance and reconnaissance analysts seek to improve the process for accurately and efficiently monitoring multiple screens or multiple sources of information simultaneously. As a result, Parallax Advanced Research scientists are researching holistic methods for teaming artificial intelligence (AI) with individual analysts to improve dynamic real-time performance assessments and need-based interventions. The goal of this will be to develop effective methodologies for managing when a human needs to monitor and control a task, versus when that task can be effectively executed by an autonomous agent.
“We were interested in figuring out how to create a system that determines the quality and efficiency of both the human component and the AI system in real-time, and, depending on which one is underperforming, have one or the other take over,” said Dr. Mary Frame, director of Cognitive Research at Parallax. “In other words, at what point do you want to tag the AI partner? And at what point do you want to have the human take control of the task?”
Parallax’s research team determined the need to concretize the implications of having analysts conduct multiple tasks simultaneously, whether this means monitoring multiple screens at once or monitoring multi-modal sources of intelligence, which may include both audio and visual information.
“Does that just mean you're viewing more screens, or does it mean you're listening to audio while you're looking at a screen?” said Dr. Frame. “For the visual monitoring, you can only look at one screen at a time whereas, for the latter, you are using the auditory processing system along with the visual system. An analyst can divide attention but can't truly scope three screens in the same way.”
To offset the potential problems with surveillance multitasking, Parallax scientists are testing a system they refer to as an “Autonomous Manager” that can independently facilitate a seamless switch between the human component and the AI system as needed. The Autonomous Manager is software that was initially developed by Dr. Frame and was later further developed by Cubic research scientist, Dr. Christopher Fisher. Although initially developed using Matlab, it has now been programmed in Java for experimental deployment. It is a system that can plug into existing multi-task experimental designs or workflows. The Autonomous Manager is not yet in the operational environment and is being tested under various experimental conditions.
While the concept of AI taking over when a human operator becomes less efficient sounds desirable, there can be trust issues experienced by the users. A user might have doubts about the competency and reliability of an autonomous partner. Parallax is addressing this issue by conducting testing of the Autonomous Manager in diverse experimental environments and user scenarios, including under various realistic edge cases. Currently, it is configurable for novel multitasking task structures, since each multitasking environment has different tasks that may be independent or interdependent. Further, interdependent tasks do not always impact each other identically across different intelligence scenarios. Scientists are refining the system to make it more complete, flexible, automatically adaptive, and fully integrated into testing systems.
“We want to test it under a variety of circumstances and with a variety of tasks to make sure that the algorithm is trustworthy when it switches tasks. The idea is to ensure that we are giving users the best potential algorithm to accomplish this,” said Dr. Frame.
To date, the Autonomous Manager has been tested in small-scale laboratory studies involving multi-tasking scenarios with independent visual surveillance tasks and in a multi-tasking scenario involving interdependent tasks that emulate the workflow of operational intelligence. Currently, Parallax and Cubic scientists are in the process of having the Autonomous Manager refined to include physiological measures of workload rather than relying on subjective workload measurements. In addition, it is being tested in an environment where the tasks are interdependent. Research on Autonomous Manager was presented at the second International Conference on Intelligent Human Systems Integration in 2019, at the Annual Meeting of the Psychonomic Society in 2019, and at the International Conference of Cognitive Modeling in 2021. It has been published in associated conference proceedings, and a journal publication is in press with the Journal of Defense Modeling and Simulation: Applications, Methodology, Technology.
Dr. Mary Frame is the Director of Cognitive Research at Parallax Advanced Research working as part of the Joint All-Domain Integrated Intelligence, Surveillance, and Reconnaissance (JADII) group at the Air Force Research Laboratory (AFRL). Her research expertise focuses on human factors psychology and human-machine teaming, as well as mathematical modeling of higher-order cognitive processes and decision-making. She utilizes a variety of process tracing methods, including eye and mouse movement tracking, to observe human cognitive processes more directly in action. Dr. Frame obtained her M.A. and PhD in Psychology from Miami University, MBA from Wright State University, and her B.S. in Psychology and Criminal Justice from Baldwin Wallace University.
###
About Parallax Advanced Research
Parallax is a 501(c)(3) nonprofit that tackles global challenges by accelerating innovation and developing technology and solutions through strategic partnerships with government, industry and academia across Ohio and the Nation. Together with academia, Parallax accelerates innovation that leads to new breakthroughs. Together with government, Parallax tackles critical global challenges and delivers new solutions. Together with industry, Parallax develops groundbreaking ideas and speeds them to market.