Emotion and AI

"A model that labels a face "angry" has not detected anger. It has detected a configuration of pixels that co-occurred with the word "angry" in its training set — which, in a person who is merely concentrating, is exactly the same configuration."- Claude 2026

Emotion and AI

Emotion looks like the opposite of computation — the thing that interferes with clear thinking. The neuroscience says otherwise: strip the feeling out of a brain and the reasoning collapses too. That finding launched an entire engineering field, and also set the trap it keeps falling into.

Credit: source

Learning objectives

By the end of this page you should be able to:

  1. Explain the role of emotion in human cognition.
  2. Describe affective computing models in artificial intelligence.
  3. Analyze emotion recognition systems built with cognitive computing techniques.
1

The Role of Emotion in Human Cognition

An emotion is a coordinated response to something that matters — a bodily change, an urge to act, and a felt experience, all triggered by an evaluation of the situation. That evaluation is the key: emotion is the nervous system's fast verdict on whether an event helps or harms you.

Where feeling happens

No single structure produces emotion, but three regions carry most of the load, and each contributes something a computational system would also need.

Amygdala

Flags biologically significant input — threat above all — within roughly a tenth of a second, before the visual cortex has finished identifying what the object even is. It is a salience detector that interrupts whatever else was being processed.

Insula

Maps the state of the body itself — heart rate, gut, breath, temperature. This interoception is what gives an emotion its felt texture, and what makes "gut feeling" more literal than the metaphor suggests.

Ventromedial prefrontal cortex

Binds bodily states to specific situations and outcomes, so that a remembered feeling can be re-evoked when a similar choice comes round again. Damage here leaves intelligence intact and decision-making in ruins.

The somatic marker hypothesis

Antonio Damasio's explanation, the somatic marker hypothesis, is that past outcomes leave bodily "markers" attached to options, and those markers prune the space of choices before deliberate reasoning begins. Without them, every alternative looks equally worth considering and the decision never converges. This is the finding that gives emotion a computational job description: it is a learned value signal that makes intractable choices tractable — which is precisely the role a reward function plays in reinforcement learning.

Three theories, three engineering choices

Psychology has not settled on one account of what emotions are, and this matters practically: each theory implies a different data structure for a machine to predict.

Theory The claim What a machine would output
Basic emotions A small set of evolved, discrete categories (anger, disgust, fear, happiness, sadness, surprise), each with its own expression and physiology A class label from a fixed list
Dimensional Emotions are regions in a continuous space, principally valence (pleasant–unpleasant) and arousal (activated–calm) Two or three real numbers
Appraisal Emotion follows from how an event is evaluated — is it novel, does it block my goal, who caused it, can I cope? A structured evaluation of the situation, from which the emotion is derived

A fourth position, the theory of constructed emotion, goes further: there are no fixed emotional essences to detect at all. The brain begins with raw valence and arousal, and constructs an instance of "anger" using learned concepts and context, the same way it constructs an object from ambiguous visual input. If that is right, then a system trained to detect universal emotional signatures is looking for something that does not exist.

Feeling and computing

Convergence: value shapes search

In both brains and agents, an evaluative signal narrows an unmanageable option space to a few candidates worth examining. Somatic markers and learned value estimates do the same structural work — they make choice possible by making most of it unnecessary.

Divergence: a body to feel with

Human emotion is grounded in a body that can be harmed, exhausted, or soothed, and interoception feeds that state back into every evaluation. A model has a scalar where a person has a heartbeat, and nothing at stake in the outcome.

2

Affective Computing Models in AI

Rosalind Picard named the field in 1995: affective computing is computing that relates to, arises from, or deliberately influences emotion. Her argument was not that machines should be moody, but that a system blind to the emotional state of its user is missing most of the signal in the interaction. The field splits into three problems, and they are not equally hard.

Recognizing affect

Infer a user's state from observable signals. The most commercially developed branch, and the one whose validity is most contested — the whole of the next section.

Expressing affect

Generate emotionally appropriate behaviour — expressive speech, facial animation on an agent, empathic phrasing in dialogue. Effective on users whether or not the system feels anything, which is itself an ethical problem.

Having affect

Give the machine internal states that function like emotions — modulating attention, urgency, and exploration. The least developed branch, and the only one that would make emotion part of the architecture rather than the interface.

Choosing a representation

Russell's circumplex model of affect: emotion words arranged in a circle within a two-dimensional space, with valence from unpleasant to pleasant on the horizontal axis and arousal from low activation to high activation on the vertical axis.
The circumplex: every affective state as a point in valence–arousal space. Credit: source

James Russell's circumplex model is the workhorse representation of affective computing, and the reason is practical rather than theoretical. Categories force a system into a discrete guess it may have no grounds for; a point in a continuous plane can express partial confidence, blend, and drift over time. It also makes the annotation problem tractable — asking someone to rate pleasantness and energy on two sliders yields far more consistent labels than asking them to choose between "irritated" and "frustrated".

Category labels remain useful for interfaces and for comparison against older work, so systems routinely convert between the two. The mapping is crude but explicit.

import numpy as np
circumplex = {
    'excited': (0.6, 0.8), 'happy': (0.9, 0.4), 'content': (0.7, -0.4),
    'calm': (0.5, -0.8), 'bored': (-0.5, -0.7), 'sad': (-0.8, -0.4),
    'angry': (-0.7, 0.7), 'afraid': (-0.6, 0.9), 'neutral': (0.0, 0.0),
}

def to_affect(probabilities):
    points = np.array([circumplex[k] for k in probabilities])
    weights = np.array(list(probabilities.values()))
    valence, arousal = weights @ points / weights.sum()
    return valence, arousal

print(to_affect({'angry': 0.55, 'afraid': 0.30, 'neutral': 0.15}))

Appraisal architectures and emotion as a control signal

Rule-based appraisal

Systems in the OCC tradition derive emotion from symbolic evaluation: an event that advances a goal produces joy, one caused by another agent's blameworthy act produces anger. Because the reasoning is explicit, the output is inspectable and explainable — the reason appraisal models persist in virtual agents and tutoring systems despite the dominance of learned approaches everywhere else.

Affect as internal control

The closest functional analogue of emotion inside a machine is not a label but a modulator. Intrinsic motivation rewards an agent for encountering surprise rather than for reaching a goal, driving exploration where external reward is sparse — a computational cousin of curiosity. Signals of this kind change what the system attends to and how boldly it acts, which is what emotion does in a brain.

Modelling affect, two ways

Convergence: appraisal is a computation

Appraisal theory was written as psychology and reads as a specification: check novelty, check goal relevance, assign agency, estimate coping ability. That a program can execute these steps and produce human-plausible emotional responses is genuine evidence for the theory.

Divergence: expression without state

A system can produce every outward sign of concern while having no internal state the signs refer to. In humans the expression is evidence of the feeling; in a machine the link is severed by design, and users reliably fail to notice.

3

Analyzing Emotion Recognition Systems

A recognition system reads one or more channels, extracts features, and predicts an affective state. Every channel is informative and every channel is ambiguous, which is the argument for combining them — and the source of the field's central difficulty.

Face

The Facial Action Coding System decomposes any expression into action units — individual muscle movements such as "inner brow raiser" — which a computer vision model can detect far more reliably than it can name the emotion behind them.

Voice

Prosody — pitch height and variability, energy, speaking rate, pauses — carries arousal strongly and valence weakly. Modern systems skip hand-built features and train directly on the spectrogram, the same input a speech recognizer sees.

Text

Word choice and context yield affective judgements without any sensor at all. See Sentiment Analysis for the methods, and Deep Learning in NLP for the representations underneath them.

Physiology

Electrodermal activity, heart-rate variability, and EEG index autonomic arousal and are hard to fake deliberately — but they say almost nothing about valence on their own, and require contact sensors.

Structure of a multimodal emotion recognition system: separate feature extraction for speech, visual, and text inputs, followed by multimodal information fusion and a final emotion classification stage.
The standard pipeline: extract per modality, fuse, classify. Credit: source

Where to fuse

  • Early (feature-level) — concatenate features before classifying. Captures interactions between channels, but breaks when the modalities run at different rates, and a single missing sensor takes the whole system down.
  • Late (decision-level) — classify each channel separately, then vote. Robust and modular, at the cost of discarding every cross-channel interaction.
  • Cross-attention — let each modality learn which parts of the others to weight, at each moment. Now the default, and handles asynchrony far better than concatenation.

The validity problem

Reported accuracies in this literature are frequently above ninety percent. They should be read with care, because three things systematically inflate them.

Acted data

Most benchmarks use performers instructed to portray an emotion. Posed expressions are exaggerated, prototypical, and cleanly separable — nothing like the muted, ambiguous faces of ordinary life. Accuracy drops sharply on spontaneous data.

Circular labels

The ground truth is usually what human annotators guessed from the same recording. A model scoring well has matched human inference about an expression, not the emotion the person actually had.

Narrow samples

Training sets skew heavily by culture, age, and skin tone, and error rates differ measurably across demographic groups — a fairness problem before it is an accuracy one.

Deployment, and where the law has drawn a line

These systems are already in use — driver-monitoring for drowsiness, call-centre quality scoring, market research, screening in hiring, engagement tracking in classrooms, and clinical work on autism and depression. The risk is not that they fail visibly; it is that a confident, unreliable inference about someone's inner state gets used to make a decision about them.

Regulators have responded. Since February 2025 the EU AI Act has prohibited the use of AI systems to infer emotions from biometric data in workplaces and educational institutions, with narrow exceptions for medical and safety purposes. The stated rationale is the technology's limited reliability and the power imbalance in exactly those settings. Outside them, emotion recognition is classified as high-risk rather than banned.

Credit: source

Reading emotion, human and machine

Convergence: inference from many weak cues

People do not read emotion from faces alone either — they integrate voice, posture, words, and situation, weighting whichever channel is most reliable at that moment. Cross-attention fusion is a recognisable version of the same strategy.

Divergence: context and stakes

A human observer knows the person, the history, and what just happened, and holds the reading loosely enough to revise it. A deployed system sees a cropped face, outputs a label with a confidence score, and that number goes into a record someone acts on.

Tools & Tutorials

  • Py-Feat — an open-source Python toolbox that detects facial landmarks, action units, and emotion categories from images or video in a few lines, with notebook tutorials for preprocessing, analysis, and visualization.
  • MediaPipe Face Landmarker — runs in the browser or in Python and outputs 478 3D landmarks plus 52 blendshape scores per frame; the fastest way to see what an expression looks like as numbers.
  • Circumplex Models (Psychology of Human Emotion, open textbook) — a clear walkthrough of how the valence–arousal space was derived and what its bipolar structure does and does not claim.
  • DEAP dataset — EEG, peripheral physiological signals, and face video from 32 participants watching music videos, each rated on valence, arousal, dominance, and liking; the standard starting point for physiological affect work.

Further reading

→ This page was created with help from Claude AI.