Cognitive Load Research

Research experiment

Related activities

Cognitive Load Research
Machine Learning
Decision Support for Incident Management
Infrastructure, Transport and Logistics
Intelligent Pipes
Infrastructure, Transport and Logistics
Security and Environment
About the Team

Cognitive Load Research
Machine Learning

"Change not only the way humans use computers - through more intuitive and natural interactive modalities, but also the way computers assist humans - through context and cognition-aware decision support"

Research in educational psychology has shown that due to the limited capacity of humans' working memory, the impact of task complexity directly reflects on task performance. Theoretical models have been devised, in particular the Cognitive Load Theory separates the intrinsic (due to inherent task difficulty) and extraneous (due to presentation complexity) cognitive loads, and establishes the existence of various sub-systems (e.g. visuospatial sketchpad, phonological loop) to process distinct input modalities.

Real-time, unobtrusive cognitive load measurement (CLM)

So far, our research has identified specific changes in a number of modalities: speech, namely acoustic, prosodic and linguistic level changes; interactive gesture; digital pen input, both interactive and freeform; and most recently eye-gaze and pupil dilation. We have successfully identified significant correlations between changes in input behaviour and increasing levels of cognitive load. The data suggests that it is feasible to use features extracted from behavioural changes in multiple modal inputs as indices of cognitive load:

  • The speech based indicators of load, based on data collected from user studies in a variety of domains, have shown a lot of promise. Scenarios include single-user and team-based tasks, think-aloud and interactive speech, single-word, reading and conversational speech, among others. This speech measurement technology is already used as a recruitment screening tool in the BrainGauge® commercial effort;
  • Pen based cognitive load indices have also been tested with some success, specifically with pen-gesture, handwriting and free-form pen input, including diagramming;
  • This research forms the technology basis for the Decision Support for Incident Management (DSIM) project, in collaboration with the NSW Roads and Traffic Authority (RTA), investigating the operations in their Transport Management Centre.

Research Goals

  • Further validate the other input modalities for multimodal cognitive load measurement, including digital-ink, eye-gaze, freehand gesture and handwriting in a variety of scenarios;
  • Explore the viability of using other features of behavioural data streams for cognitive load measurement, including posture, head position, etc;
  • Use psychophysiological signals such as pupil size and electrical conductivity of skin, as well as EEG to validate our findings in a variety of domains;
  • Employ the measurement outcomes to shed light on working memory models and human cognition.

Past research activities and outcomes

Our main research activities, over the past few years at NICTA include:

Robust Cognitive Load Indicators (CLIND)
- Objective: This continuing activity aims to identify robust and practical indicators for cognitive state and levels, from speech, response time, to physiological measurements.
- Outcomes: We have contributed to the extension of Cognitive Load Theory, by introducing real-time cognitive load measurement based on behaviour and physiological measurement analysis. We have developed the world's first real-time cognitive load measurement kit. One of our papers on cognitive load measurement based on speech analysis won best paper prize at the OZCHI 2007 conference. A patent on measuring cognitive load has been filed internationally.
- Selected publication: pdfCHISIG)
Multimodal User Interaction (MMUI)
- Objective: This continuing activity explores new interactive modalities (speech, pen, gestures, etc.) that are more intuitive, natural, can reduce user cognitive load, and can improve task performance.
- Outcomes: We have developed a family of multimodal and GUI interfaces for traffic incident management applications. Some of these interfaces have contributed to the new UI designs at the RTA. A patent on using multimodality for computer and web navigation has been filed.
- Selected publication: pdfIEEE)
Multimodal Input Fusion (MMIF)
- Objective: This activity seeks to develop novel ways to interpret complementary multimodal commands, e.g. when a user issues a command using a combined speech and hand gesture together.
- Outcomes: We have developed a novel semantic input fusion technique that can be used to interpret joint multimodal commands, based on multimodal grammars. The technique also includes how to make use of the fusion results. A practical multimodal system has been built with this technique.
- Selected publication: pdfCHISIG)
Affective and Cognitive State Indication with Galvanic Skin Response (GSR)
- Objective: This activity researches into feasibility of using GSR to evaluate user's affective and cognitive state.
- Outcomes: We have found that GSR is an easy-to-use, affordable and objective technique for real-time stress analysis. Results from our extensive user experiments have shown that a user's GSR increases systematically and consistently when cognitive load experienced by the user increased.
- Selected publication: pdfCHISIG)
Multimodal Integration Patterns (MIP)
- Objective: This activity explores whether, when and how users would prefer to use speech, gestures and pens instead of keyboard and mouse to interact with computers.
- Outcomes: Our research has found that, operating on a speech and gesture based interface, user's multimodal behavioural features are sensitive to cognitive load variations. We found significant decreases in multimodal redundancy and trends of increased multimodal complementarity, as cognitive load increases.
- Selected publication: pdfCHISIG)
Automatic Speech Recognition (ASR)
- Objective: This activity investigates various robust feature extraction methods that can be used for noisy speech recognition.
- Outcomes: Several noise-robust feature extraction algorithms for ASR have been developed and published at top-tier international signal processing and pattern recognition conferences. The algorithms include variable frame rate (VFR) processing, Mel-output compensation (MOC), cumulative distribution mapping (CDM) and noise masking via Hough transform (HT). Evaluation of the algorithms has been done by using the ETSI Aurora digits database. Comparable speech recognition accuracy with that of the ETSI advanced front-end has been achieved, at less than half of the computation load. A patent on the front-end speech processing has been filed.
- Selected publication: pdfISCA)
FPGA-based Smart Device for Cognitive State Analysis
- Objective: This activity aims to develop embedded multimedia signal processing systems for real-time cognitive state analysis that can be easily integrated into a CADS or a MMUI.
- Outcomes: We have built an FPGA-based smart camera development platform. Several prototypes have been developed using this platform, including GestureCam, a smart camera that can recognise simple head and hand gestures and can be used to allow people to use these gestures to navigate web pages.
- Selected publication: pdfSpringer-Verlag)