Decoding Mental State in Real Time with Multimodal Foundation Models

We present an interactive demonstration of the brain.space multimodal sensing and foundation model platform for modeling human state. The system combines our proprietary and patented 117 electrode dry EEG headset with ECG, EDA, eye tracking, IMU, and video. All signals, task events, and behavioral responses are synchronized on a shared timeline.

During the demo, a participant will wear the sensing system and complete a short cognitive task. Attendees will be able to follow the session from live signal acquisition through processing, foundation model encoding, and downstream inference. The interface will display processed EEG activity, eye movements and behavioral measures, physiological signals, task performance, and model outputs in a synchronized view.

The live cognitive performance inference will use learned EEG and eye tracking representations. ECG, EDA, IMU, and video will be recorded and synchronized as part of the broader sensing platform, allowing attendees to see both the complete multimodal acquisition system and the specific signals used by the demonstrated inference model.

Our foundation models are pretrained using self supervised learning on approximately 10,000 hours of synchronized EEG, ECG, EDA, eye tracking, IMU, video, behavioral data, and experimental context. Rather than learning a new representation for every individual experiment or label, the models learn reusable representations that can support multiple downstream tasks through lightweight task specific heads.

The demonstration makes this reuse visible in a live setting. During the cognitive task, model estimates will be shown alongside the participant's measured behavior and task performance. Attendees will also be able to inspect intermediate stages of the pipeline, including processed and source localized EEG activity, synchronized sensor streams, learned representations, and downstream outputs. Brain age prediction will be shown as an additional example of information that can be extracted from the same pretrained EEG representation.

The goal of the demo is to show how foundation models for neural and physiological signals can move beyond offline benchmark evaluation and operate as part of a live sensing and inference system. Because brain.space develops both the sensing hardware and the modeling stack, the platform is designed around precise synchronization, consistent acquisition across modalities, and large scale collection of aligned physiological data.