Neural populations as motor controllers

World, body and brain on one clock

One route to a world model is to re-derive it from passive video, as in LeCun's programme. We are testing a different one: start from the world model that a high-performing species already carries. That requires recording the world, the body and the brain on a common clock during free behaviour, rather than collecting three datasets and aligning them after the fact. The infrastructure for it is running.

What the platform records, on one clock

• WorldProgrammable enclosures whose lighting, sound, olfaction, texture, reward and obstacle state are recorded alongside the participant. Environmental state is in the dataset.
• Body3-D skeletal motion from synchronised multi-camera IR capture, exported as MuJoCo-compatible bodies that replay into simulation without a 2-D-to-3-D lift step.
• BrainWireless multi-channel headstage in freely behaving animals; non-invasive fNIRS / EEG / EMG / gaze in humans. Lab Streaming Layer (LSL) puts every stream on the same per-session clock, written into one XDF file.

Behavioural reconstruction pipeline

On the animal side the reconstruction pipeline is: synchronised infrared multi-camera capture → LightningPose3-D for per-camera 2-D keypoints → multi-view triangulation → EKS (Ensemble Kalman Smoother) temporal smoothing → MuJoCo-compatible skeletal motion. The output is true 3-D skeletal motion under naturalistic behaviour, in a representation that replays directly into simulation without a separate 2-D-to-3-D lift step.

On the human side the equivalent path uses full-body motion capture in the stationary clinic and a markerless pipeline (FreeMoCap multi-cam, or a single-camera MediaPipe Pose route running at roughly 25 fps over WebSocket) in mobile sessions. Both paths produce the same kind of MuJoCo-compatible body model, fitted per participant from body-surface and joint-angle data.

Wireless headstage

The headstage is a custom multi-channel neural-logging device built in-house. Removing the tether is what enables continuous longitudinal recording, unrestricted free behaviour, and scalable cohort recording across multiple cages running in parallel. The device is small enough that the animal's behavioural repertoire is not constrained by its mass — see the wireless-headstage image below for the hand-scale reference.

Modular cage instrumentation

The cages run continuously across curated environments whose physical and sensory properties can be modified programmatically in real time. Stimulus systems, lighting, peripheral sensors, wireless receivers and intervention hardware mount onto shared rails and can be reconfigured between experiments without rebuilding the environment. A PsychoPy-based stimulus controller pushes paradigm events into the LSL clock as a marker stream, so behaviour, neural state and stimulus history are written into the same XDF file with a common timeline.

Data infrastructure

The platform currently manages on the order of hundreds of terabytes of synchronised multi-modal experimental data with automated ingestion, longitudinal archival, behavioural tokenisation, neural aggregation and sequence assembly for transformer training. Behavioural trajectories, neural activity, environmental state and trial context are converted into unified sequential representations suitable for large-scale autoregressive modelling. The point of the infrastructure is not the storage scale on its own — it is that signal quality and temporal continuity are the binding constraints on current foundation models in neuroscience, and the recording stack was designed against those constraints from the beginning.

Behaviour as multi-scale dynamics, recorded as such

Biological behaviour is the visible output of coupled processes running at very different spatial scales — synaptic chemistry, single-neuron firing, local circuit dynamics, body mechanics, and large-scale systems organisation — and the coupling between scales is part of what we are trying to model. A recording stack that captures only one scale loses that coupling. The platform records across scales on the same clock: neural activity at unit and population scale, body kinematics and muscle activation at the actuator scale, and environmental state at the world scale, all timestamped together so the cross-scale relationships are preserved in the dataset rather than reconstructed afterwards.

The comparison this is set up for

Narrow form of the embodied-world-model claim: a world-model architecture trained on world + body + brain trajectories from a behaving organism reaches a given prediction quality on held-out environments from fewer trajectories than one trained on video. Either result is informative. The infrastructure below is what makes that comparison runnable. The next two tabs are the two other empirical questions that ride on top of the same recording stack.

Live examples from both sides

Headstage · deviceWireless headstage — the device used for free-behaviour neural recording

Paired neural and behavioural trajectories as a learning target

Most public sensorimotor datasets are behaviour-only: motion capture, video, joint angles. The neural state of the organism producing the behaviour is absent. Our claim is narrow: when both channels are recorded on a common clock, the pair supplies a sensorimotor-control supervision signal that neither channel supplies on its own. Whether that supervision signal reduces the sample cost of competent control is the empirical question in §03.

What "paired" means here

Stationary clinic and mobile wearable stack use the same synchronisation layer (LSL), the same XDF storage format and the same downstream tokenisation. Neural and behavioural channels arrive in a single per-session file on a common clock, so alignment is built in rather than reconstructed.

LSL gives each stream its own producer clock plus a host-wide reference, and LabRecorder writes those into XDF as time-series chunks tagged with both. Per-stream offsets sit within the protocol's reported tolerances; the binding constraint at high modality counts is host-clock drift across multiple acquisition machines, mitigated by using a single time-server on the recording subnet.

Why the paired channel matters — timescales of biological adaptation

Most ML systems implement learning at one timescale — typically a single training run measured in hours or days, plus periodic fine-tuning. Biological nervous systems implement adaptive processes that run continuously from milliseconds to years: fast neuromodulatory state changes, online sensorimotor adaptation, episodic replay during quiet wake, sleep-stage consolidation, and slow systems-level reorganisation. A behaviour-only recording captures the trajectory but not the adaptation; the paired neural + behavioural channel is the substrate for studying the cross-timescale processes ML largely does not yet implement.

• Neural · slowKernel Flow 2 fNIRS · 48 modules · 5 Hz · haemodynamic correlates of engagement and motor planning.
• Neural · fastHigh-density / dry-electrode EEG · cortical band power and event-related responses (e.g. DSI-24, 300 Hz).
• ActuationDelsys Trigno surface EMG · 16ch · 2 kHz · forearm and intrinsic-hand placements. The actuation signal we model muscle activations against.
• AttentionPupil Neon · 200 Hz binocular gaze · 110 Hz IMU. Egocentric attentional channel and head-frame kinematics during free movement.
• BodyFull-body motion capture (stationary) and markerless wearable IMU + pose (mobile). 3-D, MuJoCo-compatible.
• HandStretchSense capacitive-stretch glove — joint-angle hand pose as label channel.
• SyncLSL + LabRecorder · one XDF file per session contains every modality on a common clock.

Robot bodies the data targets

Reconstructed pose, EMG-derived muscle activations and gaze can be retargeted onto standard robot platforms. The three below are the platforms we run locally for motion-imitation comparison (MimicKit baselines, Peng et al.). Retargeting passes through SMPL or pi_plus as the intermediate skeletal target before mapping to the specific robot's URDF. The motion-imitation methods are trained inside Isaac Gym / Isaac Lab / Newton with 4096 parallel environments by default for the on-policy PPO variants. Whether conditioning policies on paired biological trajectories changes the number of environment interactions needed to reach a fixed competence threshold is the sample-efficiency test in §03.

humanoid · 28-DoF

Unitree G1

Unitree Go2

Fig. 04 · MimicKit baseline rollouts on the three platforms used locally. Frames courtesy MimicKit (Peng et al.).

Motion-imitation methods we exercise on these platforms

The seven tiles below are the published reference results from each motion-imitation algorithm in MimicKit, taken from the original paper teasers (Peng et al.). We run all seven locally on the humanoid / G1 / Go2 platforms above and use them as the baseline policies the paired-trajectory conditioning is compared against.

DeepMimic · Peng 2018

AMP · Peng 2021

ASE · Peng 2022

AWR

LCP

ADD

SMP

local run · in-scene

Fig. 05 · MimicKit method teasers (Peng et al.) plus a local in-scene rollout. DeepMimic / AMP / ASE / AWR / LCP / ADD / SMP are the algorithms we use as baselines.

What each baseline gives us

• DeepMimic (PPO)Phase-conditioned per-link tracking reward against a reference motion clip. Strongest per-clip tracking but weakest generalisation across clips. The vanilla baseline.
• AMPAdversarial Motion Priors — a GAN-style discriminator learns a style reward from a motion dataset, so the policy is rewarded for "looking biological" rather than for tracking a single clip. Decouples skill objective from style.
• ASEAdversarial Skill Embeddings — extends AMP with a learned latent skill code, so a single policy can be conditioned to produce a span of biological behaviours from one continuous latent.
• AWRAdvantage-Weighted Regression — off-policy actor update by weighted regression onto past actions. Cheaper than PPO at the cost of slower convergence; useful for offline-data ablations.
• LCPLipschitz-Constrained Policies — explicitly bounds action-space sensitivity, which matters when the policy is conditioned on noisy biological signals.
• ADDAdversarial Differential Discriminator — discriminator on action deltas rather than state-action pairs; tends to produce smoother control under the same reward.
• SMPScore-Matching Motion Priors — diffusion-style score-matching on motion priors, used as a sample-efficient alternative to AMP-style discriminators.

The conditioning ladder we run

For the sample-efficiency comparison the simulator, robot body, task definition, reward, and base method are held fixed. The single varied factor is the conditioning channel passed to the policy:

• A · motion onlyReference trajectory is the recorded kinematic motion; loss is the standard per-link tracking term as published. This is the published baseline reproduction.
• B · motion + EMGAdds simulated muscle activations from the MyoSuite forward model fitted to the participant's EMG, as an auxiliary reference channel.
• C · motion + EMG + gazeAdds the egocentric attentional channel from the head-mounted eye tracker as an observation prior, addressing tasks where head-frame attention matters.
• D · full paired channelAdds slow EEG / fNIRS state. High cost. Expected to be informative only on tasks where cognitive state varies appreciably (instructed vs. self-paced, fatigued vs. rested).

Dependent variable: environment interactions required to reach a fixed task-success threshold (e.g. 80% success on held-out initial conditions). Confidence intervals come from seed variation across five seeds per condition. The same comparison is repeated on humanoid, G1 and Go2 so that a positive result has to generalise across morphology rather than fit one body.

Transfer ladder

A decoder fitted on neural state predicts motor output. The ladder runs that same decoder against actuators of increasing scale. Each rung is a different body; the readout layer does not change between rungs.

Rung 01

Headstage + instrumented cage

Wireless multi-channel headstage on a freely behaving mouse. Cage and headstage share a clock — neural and behavioural state recoverable from one record.

headstage · device

cage · mouse in environment

Rung 02

Free behaviour

Continuous recording across locomotion, rearing, grooming, exploratory turning. Paired neural + behavioural trajectories during unconstrained behaviour.

curated environment

pose tracking

Rung 03

Mouse-driven vehicle

Trained mouse drives a small vehicle via a directional neural readout (cf. Lambert et al. 2019; IBL operant tasks). First rung where the decoder addresses a non-biological actuator.

stand-in · mouse physics replay

robot mouse · cage playback

Rung 04

Quadruped ground platform

The same decoder layer mapped onto a larger-scale ground platform. The claim being tested at this rung is portability of the readout across actuators with calibration rather than retraining.

quad ped · in operation

Unitree Go2 · reference

Rung 05

Humanoid joint from neural intent

20-neuron population from real brain recordings drives a G1 joint — first version is waist-yaw on a sinusoidal stim; second version is an arm waypoint from live camera input. See §03 for the controller and the goalkeeper demo.

G1 · neural waist-yaw

platform · reference

Niche-specific biology, niche-specific priors

Animals are niche learners. Each species has been shaped by evolution for a specific ecological setting, and the inductive biases that come out of that shaping are why a biological system can learn a behaviour from a handful of trials rather than from billions. For artificial sensorimotor control, that argues for using niche-specific biological structure as a starting prior. The version of the argument that goes furthest is a robot controller whose parameters came directly from a recorded brain, with no training done on the robot side.

Different stimuli for different senses

The cage is experimental infrastructure. Sensory channels and reward schedules are set programmatically per task, so a single cohort can run through a battery of paradigms without the environment being rebuilt between them.

• VisualSudden light change · obstacle appearance · stimulus on the left or right of a screen.
• AuditorySound stimulus for orientation response · startle.
• Tactile / somatosensoryFloor texture change for gait adjustment · air-puff startle.
• OlfactoryProgrammable VOC delivery; olfactory state modulates exploration in mammals.
• Reward / motivationFood / liquid delivery, schedule-controlled.
• FutureTargeted neural stimulation for direct neural → behaviour causality.

top · electronics

enclosure

PCB rail · close

Modular cage — reconfigurable rails for stimulus, sensing and intervention hardware. Stimulus hardware can be swapped between paradigms without rebuilding the environment.

Olfaction-conditioned control

Olfaction is one of the programmable stimulus channels above, and it is also a biological sensing prior on the robot side. This clip conditions a quadruped policy on live readings from the Scentience 14-channel VOC sensor — an olfaction-vision-language-action (OVLA) demonstration of a non-visual channel driving embodied behaviour, on the same gas-sensing hardware that sits in the acquisition stack in §02. Sensor and demo footage courtesy Scentience (Kordel France).

Cross-species latent structure

Each species is specialised for its own niche. The useful question is what behavioural structure carries across the species boundary. The platform's body models for humans and mice are linked through a shared latent-alignment framework: a perceptual decision task is adapted in parallel for an embodied human and an embodied mouse with the same stimulus geometry, and the matched behavioural representations feed into a shared embodiment model.

Alignment · human ↔ mouseHuman–mouse modelling alignment — parallel embodied tasks, shared behavioural and neural blueprints

Robot mouse — playback of recorded cage behaviour

A simulated robot mouse re-enacting a real animal's recorded trajectory in the instrumented cage. The replay closes the loop from free-behaviour recording back into an embodied model, giving a controllable, physics-compatible proxy for the niche-specific behaviour the priors above are fitted to.

Twenty brain-fitted neurons, zero controller training

The clearest demonstration we have on the robot side is a controller for the Unitree G1 in which the policy is not a learned network. It is a 20-neuron population whose firing-rate parameters came directly from a neural-encoding fit on a rodent visual-decision dataset. The population vote drives a robot joint. Zero training was done on the robot side.

The encoding fit (Phase-1, "step1a cat-sanity")

Each unit is fit as a separable temporal-kernel encoder: predict the unit's spike rate from a single task variable (the "world variable") convolved with a Gaussian-of-lag temporal kernel. The fit returns three parameters per (neuron, variable) pair — α (signed gain), τ (peak lag in ms), σ (temporal width in ms) — plus a fit score and an R². Selection criteria: world_variable ∈ {stim_side, choice}, score ≥ 0.10, model_r² > 0, then top-10 per variable by score.

• Sessions processed459
• Neurons processed86,033
• Non-zero α80,054
• Positive model R²53,269
• "Strong" units (score ≥ 0.10, R² > 0)17,663
• World variables modelled13 (task / behaviour)
• Corpus median session R²≈ 0.002
• Corpus median session correlation≈ 0.09
• Strongest selected unitchoice unit in MRN, R² ≈ 0.17, corr ≈ 0.68

The 20 selected units

The slice below is the exact 20-unit set frozen in neural_controller/config/selected_neurons.json, each carrying its session ID, probe and cluster ID in the source file. The signed α distribution is part of the design: 15 of the 20 are negative-α, so the population produces signed votes without any extra logic in the controller.

ROS 2 controller package

The controller ships as a ROS 2 Humble package (neural_controller) that runs inside the neural-encoding-humanoid workspace. Per-tick it ingests a scalar stimulus, evaluates the 20-unit population, applies the ±0.01 dead-band, integrates the vote into a yaw offset with a wall-dt-aware integrator, and publishes the full 29-DOF command in unitree_hg/LowCmd on /lowcmd. Joints other than the driven one are held at their measured start positions.

• /neural/stim_signalstd_msgs/Float32 — stimulus used this tick (sine demo by default; external override via /stim_override).
• /neural/population_votestd_msgs/Float32 — summed α · firing_rate across the 20 selected units.
• /neural/commandstd_msgs/String — "left", "right" or "stand" after dead-band.
• /lowcmdunitree_hg/LowCmd — full 29-DOF command; only the driven joint moves under the neural decoder.

Launch parameters (defaults)

G1 hardware interface — what the controller is publishing into

The G1 is a 29-DOF humanoid with mode_machine = 5 by default. The Jetson on the robot sits at 192.168.123.164/24 on an internal eth0; CycloneDDS is bound to that interface with config at /home/unitree/cyclonedds_ws/cyclonedds.xml. The front RealSense D435i is exclusively held by the boot-time videohub_pc4 service on /dev/video4. Three publishers share the bus: a high-level locomotion controller on rt/lowcmd at roughly 84 Hz (always on, holds the arms in a fixed pose); the neural controller above on rt/lowcmd; and an IK arm overlay on rt/arm_sdk (Pinocchio + CasADi, 250 Hz, velocity-limited to 2.0 rad/s, with a startup-race fix and a try/finally ramp that releases the engagement weight back to 0 on exit so the joystick is never left locked out).

Pinned-pelvis MuJoCo substrate

The demo runs against a sibling of unitree_mujoco.py with three modifications baked in, no URDF / XML edits:

• Leg pin12 leg joints plus waist_roll and waist_pitch are nailed to zero by clobbering their qpos and qvel after every mj_step. Only waist_yaw (idx 12) and the 14 arm joints (idx 15..28) move under /lowcmd.
• Pelvis springPelvis held by a virtual 6-DOF spring through xfrc_applied (KP_POS = 1500 N/m, KP_ROT = 80 Nm/rad, both heavily damped). Replaces the GLFW-keyed elastic band that fails to receive keypresses over Wayland / XWayland.
• PD recomputePD torque is recomputed every mj_step from current sensordata via bridge.ApplyLatestLowCmd(), rather than only when a new /lowcmd arrives. Without this fix the kd · dq damping term goes stale at high kp (especially waist_yaw at kp = 120), the joint winds up to hundreds of degrees, and the actuators saturate at ±88 Nm trying to recover.

Goalkeeper: the same 20 neurons on real perception

The waist-yaw demo above ran on a synthetic sinusoidal stimulus. The goalkeeper closes the loop on live perception. The pipeline: MJPEG from the G1's front RealSense (served by the in-house camera/mjpeg_server wrapping VideoClient::GetImageSample because videohub_pc4 exclusively owns the camera device) feeds YOLOv8s detecting COCO class 32 (sports ball) at confidence 0.10, the largest bounding box is taken, its centre cx is mapped to the stim signal s = (cx − w/2) / (w/2) ∈ [-1, +1], the same 20-neuron population votes, the vote crosses the ±0.01 dead-band into left / right / stand, and the resulting command selects one of three Cartesian arm waypoints (READY / LIFT_LEFT / LIFT_RIGHT). Dual-arm IK (G1_29_ArmIK, Pinocchio + CasADi) solves them at 50 Hz with 2 s interpolation between waypoints, velocity-limited to 2.0 rad/s, and publishes 29-DOF commands to rt/lowcmd with motion_mode = False. The vision and neural stages run in a background thread at 5 Hz with hysteresis (3 consecutive agreeing frames to commit a command; 5 empty frames to fall back to stand). The IK / publish loop runs in the foreground at 50 Hz. The decoder is still the recorded population. No retraining on the robot side.

Getting the arms to actually move — the engineering parts

Most of the engineering effort was not in the neural model. It was in discovering how to command the arms at all. The natural path — publishing an rt/arm_sdk overlay that blends with the locomotion controller — was published correctly (weight = 1.0, kp = 80, valid CRC) but ignored by the robot's active ai controller; a direct bypass ramp moved no joint. The working path, mirrored from ik_demo_direct.py, is to call MotionSwitcher.Enter_Debug_Mode() to release the high-level controllers and publish directly to rt/lowcmd with motion_mode = False. The cost: the legs go limp the instant debug mode is entered, so the robot is suspended on a crane for the entire run, and a try/finally always hands control back to the ai controller on exit. This is the difference between the demo running (the pipeline computing correct commands, which it did for days) and the demo working (the arms physically moving — the 29 May result).

Asymmetry from population composition

Because 15 of the 20 selected units are negative-α, the population's response is asymmetric by construction: the response magnitude to a left-side stimulus is roughly 8.6× the response to a right-side stimulus (|vote| ≈ 2.59 vs. ≈ 0.30). This is a fingerprint of the recorded population's composition, not a controller-side choice. In the earlier MuJoCo waist-yaw demo this asymmetry had to be normalised per side to avoid the robot slamming to one side; in the arm demo it is harmless because the readout is discrete (left / right / stand) rather than continuous.

Scope of the claim: the single-neuron encoders are weak (corpus median session R² ≈ 0.002; the strongest selected units reach R² ≈ 0.17). The fit captures sign and latency structure (which side a unit prefers, and when it responds), and the controller uses only that. The discretisation, the ±0.01 dead-band and the fixed 300 ms kernel lag are engineering choices motivated by the biology rather than validated as the brain's own readout.

p. 1 · abstract

p. 2 · architecture

Fig. 09 · Neural_Humanoid_Goalkeeping.pdf — abstract and system architecture. Download full PDF (4 pp.)

The sample-efficiency comparison this sets up

The goalkeeper is the existence proof: a brain-fitted population is sufficient to close a real sensorimotor loop with no downstream training. The next comparison is the sample-efficiency test itself. Concrete protocol:

• TaskA fixed closed-loop sensorimotor task on the humanoid platform — e.g. reaching, stable single-leg stance, or a contact-rich manipulation primitive. Reward and termination held constant across conditions.
• MethodOne MimicKit baseline held fixed across conditions (DeepMimic for the per-clip case, AMP for the priors-from-dataset case).
• Conditioning ladderA · motion only. B · motion + EMG-derived muscle activations. C · adds gaze. D · adds slow EEG / fNIRS. The conditioning is the only thing that varies between runs.
• Dependent variableEnvironment interactions required to hit a fixed task-success rate (e.g. 80% on held-out initial conditions).
• VarianceFive seeds per condition for first-pass CIs; bump to ten for any condition that lands close to a baseline.
• GeneralisationRepeat on humanoid, G1 and Go2. A positive result has to survive morphology change.
• Null hypothesisPaired-channel conditioning makes no difference relative to motion-only. A null is informative — most public discourse assumes the positive result without measuring it.

What the robotics contribution actually is

The interesting contribution of neurobehavioural foundation models to robotics is unlikely to be "animal imitation" in the literal sense — robots moving like the animals in the recordings. Modern ML is already strong at trajectory imitation and representation learning. The contribution is access to the organisational principles underlying adaptive embodied intelligence: the things biological nervous systems do that current ML architectures still implement poorly.

Three areas where biology continues to substantially outperform current autonomous systems: multi-scale, multi-timescale adaptation; state-dependent policy switching via neuromodulation; and continual operation under severe energetic constraint. Each one motivates a different sort of prior the platform could supply.

Plasticity is multi-scale and continuous, not a single global update

Most ML training is one form of plasticity (weight updates) under one optimisation regime, applied globally. Biological nervous systems run several forms of plasticity in parallel, at multiple spatial scales — synaptic-level long-term potentiation and depression, neuron-level intrinsic excitability changes, local circuit rewiring, body-mechanics adaptation, and large-scale systems reorganisation — and these processes interact. Behaviour is the output of those interactions, not of a static policy network. For robots operating in naturalistic environments where the agent has to maintain stability while adapting online to uncertain inputs and unexpected perturbations, the multi-scale aspect is probably what matters.

Neuromodulation reconfigures policy in response to state

Dopamine, serotonin, acetylcholine and norepinephrine do not encode reward and arousal in isolation. They jointly regulate saliency assignment, attentional allocation, uncertainty estimation, exploration vs. exploitation, and the plasticity thresholds that gate when learning happens at all. The same external environment can therefore produce different behavioural policies depending on internal state — stress, metabolic load, novelty, social context. Most robotic systems run with comparatively fixed exploration dynamics and reward structures; biological systems reconfigure behavioural strategy in response to changing internal conditions. The relevant question is which of these reconfiguration mechanisms can be lifted from the recorded dynamics into a controller prior.

Energy and continual operation

The human brain runs at roughly 20 W while supporting perception, memory, motor control, planning, social reasoning and continual adaptation. Evolution has spent hundreds of millions of years optimising embodied intelligence under severe energetic constraint. Modern ML systems are computationally expensive, data-hungry, energy-intensive at training time, and often brittle outside the training distribution. For robotics, this matters in any deployment where edge processing, low-power operation or continuous on-device adaptation is required and continuous cloud-based inference is impractical. Neuromorphic substrates (sparse asynchronous signalling, event-driven computation, distributed processing, local learning rules, low-power operation) are one route; identifying which biological organisational principles are the load-bearing ones for adaptive embodied behaviour is what neurobehavioural foundation models can contribute toward.

Seven biologically-grounded priors for robotics

The narrow list of properties biological organisms still substantially outperform current autonomous systems on, and that the platform's recordings are positioned to supply priors for:

• Exploration under uncertaintyWhere do animals look, sample and probe when the environment is novel or ambiguous, and how does that change with internal state? Maps to robot exploration policies that are not fixed schedules.
• Continual adaptationOnline behavioural change without catastrophic forgetting. The substrate biology uses is multi-timescale plasticity plus replay; current ML alternatives are mostly absent or rudimentary.
• Saliency allocationWhich stimulus features get processed at all, in which order, and at what gain. Closely tied to attention and to the neuromodulator wheel above.
• Behavioural flexibilitySwitching between qualitatively different behavioural modes (forage, escape, rest, social) without retraining the underlying controller.
• Replay and consolidationOffline structured reactivation of recent experience, plus slower systems-level reorganisation. The biological alternative to continuous gradient descent on incoming data.
• Context-dependent policy reconfigurationThe same external environment producing different policies depending on internal state. Currently approximated in ML by goal/skill embeddings; biology does it natively.
• Multi-agent coordinationConspecific interaction, communication, joint attention. Biological organisms have priors for these that current robot multi-agent systems mostly do not.

Principle extraction over direct emulation

Bottom-up replication of biological nervous systems remains very difficult and is probably not the right objective. The relevant goal is to identify the organisational principles underlying the seven items above, and to extract them at a level abstract enough to drop into a robot controller. The IBL-fitted population in §03 / the goalkeeper demo is the existence proof for one such principle (sign-and-latency-structured sensory readout). The same approach generalises to the rest of the list: pick a property, find the dataset that resolves it, fit an encoder, run the encoder forward on the robot. The platform sections above are what makes that pipeline runnable for properties beyond the one already demonstrated.

Related published work the comparison is calibrated against

• MimicKit / DeepMimicPeng et al., 2018+ — motion-imitation policies on humanoid and quadruped platforms; the baseline policy family we report against.
• AMP · ASEPeng et al., 2021 / 2022 — adversarial motion priors and skill embeddings, also baselines in MimicKit.
• MyoSuiteCaggiano et al., 2022 — musculoskeletal MuJoCo environments. Used here as the muscle-activation action space the EMG channel maps to.
• HumanPlus / H2OStanford and CMU, 2024 — humans-as-demonstrators for humanoid policy learning. The paired-channel comparison is set against the same humanoid platforms.
• International Brain LaboratoryIBL standard operant tasks — source of the visual-decision dataset the goalkeeper's neural population was fitted on.
• Lambert et al., 2019Rodent-driving paradigm — the paradigm cited for Rung 03 of the transfer ladder.