Invited Talk: Feeling in Real Time: Affective Intelligence for Conversational Agents

Real-time conversational agents are rapidly becoming able to see, hear, speak, and act. Yet the step from fluent interaction to genuinely natural interaction requires more than low latency and increasingly powerful multimodal models: it requires sensitivity to the continuously evolving affective and social state of the human interlocutor. In this talk, I will argue that affective intelligence should be understood as an inherently real-time problem. Human emotion is rarely communicated through a single word, facial expression, or acoustic event. It unfolds dynamically across speech, language, facial behaviour, gaze, physiology, and interaction itself. Likewise, an intelligent conversational agent cannot wait for a complete utterance—or an entire interaction—before deciding what a person feels and how to respond. It must continuously listen, observe, interpret uncertainty, anticipate what may happen next, and adapt its own behaviour while the conversation is still unfolding. I will discuss progress from affect recognition towards multimodal, context-aware and generative affective computing, and the opportunities created by foundation models for understanding and synthesising human behaviour. At the same time, I will highlight uncomfortable open problems: the distinction between appearing empathic and actually understanding the user, and the risks of increasingly persuasive machines that can sense and influence human emotion. Ultimately, the challenge is not simply to build conversational agents that respond faster. It is to build machines that know what to sense, when to listen, when to speak, when to interrupt—and perhaps most importantly, when not to.