Skip to main content

In Brief

We trained a multimodal digital twin to predict V1 spiking in freely moving mice from retinocentric video together with locomotion, pupil, and eye and head movements, then froze it and probed it in silico under controlled conditions.

Abstract

Decades of controlled experiments in passive, head-fixed animals have characterized the response properties of mouse primary visual cortex (V1) and their strong modulation by behavioral state. However, in natural vision the animal is active, continuously shaping its own visual input as it moves and coupling vision to behavior. To bridge this gap between classical controlled experiments and natural vision, we trained a multimodal digital twin to predict V1 spiking in freely moving mice from retinocentric video together with locomotion, pupil, and eye and head movements, then froze it and probed it in silico under controlled conditions. The representation it learned generalized across stimulus classes, animals, and recording regimes. Although never trained on parametric stimuli or given cell-type labels, the model recovered V1’s canonical tuning for orientation, spatial frequency and contrast, including the systematic differences between putative excitatory and inhibitory cells. A single model transferred to new mice from a few minutes of recording. Without retraining, it matched the representational geometry of an independent head-fixed dataset of natural movies as closely as two recorded mice match each other. Behavior modulated most V1 neurons, and clamping each variable in turn showed that locomotion and pupil dilation share a gain increase, while only locomotion shifts sensitivity toward high spatial frequencies. By learning this representation from the freely behaving animal and probing it under controlled conditions, freely moving digital twins bridge classical visual physiology and active vision.

Related Research

We study vision during natural behavior: how eye, head, and body movements shape the visual input, how the brain responds, and how biological and artificial systems solve similar sensorimotor problems.

Yuchen Hou Marius Schneider