Perception and Neural Networks

Perception is not passive reception — the brain actively constructs a representation of the world, combining incoming signals with prior knowledge. Two opposing flows drive this process:

Beyond these two flows, perception in the brain is organized as a hierarchy of increasingly complex feature detectors: simple elements detected early are progressively combined into richer, more abstract representations at each successive stage. The two sections below trace that hierarchy — first the landmark experiment that revealed it, then the chain of cortical areas it runs through. This same hierarchical organization is what convolutional neural networks later imitate.

Feature detection: Hubel and Wiesel (1959–1962)

The visual hierarchy

Recording deeper into the visual system revealed that V1 is only the first stage. Signals pass through a chain of cortical areas, each responding to more complex and more position-independent features than the last — culminating in neurons that fire for whole objects regardless of where they appear.

Damage to the inferotemporal cortex (IT) causes visual agnosia: the ability to see, but not to recognize. This local-to-global progression is the template for deep CNN architecture.

Gestalt principles and multisensory integration

Convolutional Neural Networks (CNNs) translate the Hubel-Wiesel visual hierarchy into learned computation — each layer type has a direct biological counterpart.

Convolutional layers — edge detectors

A small filter (typically 3×3 or 5×5 pixels) slides across the input image, computing a dot product at every position. Each filter learns to detect one local pattern — a vertical edge, a diagonal, a color transition. Early layers learn low-level features exactly as V1 neurons do; deeper layers combine these into complex shapes and object parts, as IT does.

Pooling layers — position tolerance

A max-pooling operation keeps the strongest activation in a small region, discarding exact position. This builds position tolerance into the representation — directly analogous to Hubel and Wiesel's complex cells, which fired for an edge anywhere in their receptive field.

Fully connected layers — classification

After several conv/pool blocks, a flat vector of high-level features feeds into dense layers that map to class probabilities — analogous to the IT cortex's role in object identity, the final stage of the ventral visual stream.

Speech perception in the brain extracts acoustic features — frequency, timing, phoneme boundaries — and routes them toward language areas. Computational speech recognition mirrors this pipeline in three stages.

Spectrogram input

Raw audio is converted into a spectrogram — a 2D representation of frequency content over time. This is the acoustic analog of an image: just as a CNN receives pixel values, a speech model receives spectrogram values. The conversion mirrors the cochlea's role in decomposing sound into frequency components.

Sequence modeling

Unlike images, speech unfolds over time — each phoneme depends on what preceded it. Recurrent networks (RNNs) handle this with hidden state carried forward through time. Transformers handle it with self-attention: every time step can directly attend to any other, capturing long-range dependencies RNNs struggle to maintain.

End-to-end learning

Early systems hand-engineered acoustic features and phoneme boundaries. Modern audio transformer models like Whisper are trained end-to-end directly from spectrograms to text, learning their own internal representations — just as the visual cortex learns its own feature hierarchy from experience.

CNNs and the visual cortex show striking similarities — and equally striking failures that reveal how the analogy breaks down.

Adversarial examples

An adversarial example is an image modified by imperceptible noise that causes a neural network to misclassify it — while looking completely unchanged to any human.

• Canonical example: a panda classified correctly at 57.7% confidence becomes a gibbon at 99.3% confidence — after adding noise no human can see.
• Why it happens: CNNs learn statistical shortcuts through training data, not the structured, invariant representations biological vision builds.
• Humans are immune: the same perturbation has no effect on human perception — the gap made visible.
• Safety implications: object detectors, medical imaging classifiers, and autonomous vehicle cameras are all potentially vulnerable.

Tools & Tutorials

• CNN Explainer — an interactive, in-browser visualization of a small CNN processing a real image; hover over any neuron to see its receptive field, activations, and the filters it learned. Excellent for building intuition for conv and pooling layers.
• TensorFlow Playground — adjust network depth, activation functions, and learning rate and watch feature representations form in real time; complements CNN Explainer for understanding what layers learn.
• TensorFlow — DeepDream tutorial — a runnable tutorial that uses gradient ascent to visualize what each layer of a CNN has learned to detect, making the feature hierarchy concrete and visible.

Further reading

• Hubel, D. H. & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of Physiology, 160(1), 106–154. — the foundational paper establishing the visual feature hierarchy; directly inspired CNN architecture.
• Goodfellow, I., Shlens, J., & Szegedy, C. (2015). Explaining and Harnessing Adversarial Examples. ICLR 2015. — introduces the fast gradient sign method and the panda/gibbon example; essential reading on the limits of CNN perception.
• Yamins, D. L. K. & DiCarlo, J. J. (2016). Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19(3), 356–365. — reviews the evidence that CNN layer representations correspond to visual cortex areas V1 through IT, and what this convergence implies for both neuroscience and AI.