Reinforcement Learning Quadruped

Overview

A custom-designed 8-DOF quadruped (thigh and calf per leg, no hip motors) driven by 8 Unitree GO-M8010-6 motors and controlled from an onboard Jetson. The goal: train a walking policy entirely in simulation with reinforcement learning (PPO, MuJoCo + Gymnasium), then deploy it on the real robot and close the loop between sim and hardware until it walks forward reliably in both.

Workflow

1. 1.Model the robot in Onshape CAD, exporting both a real mesh/mass model and a real joint-kinematics model.
2. 2.Generate a MuJoCo model combining the two into one physics model, plus the canonical motor-to-joint mapping every other stage shares.
3. 3.Train a PPO policy in a Gymnasium environment that sees exactly what the real robot senses (motor encoder angles/velocities and IMU data, nothing privileged like absolute position), first to stand, then to walk.
4. 4.Export the trained policy to TorchScript so it runs with just PyTorch at inference time.
5. 5.Run it on the real robot: read real motor + IMU data, build the same observation the policy trained on, command real motor targets, every control tick. Then feed what real hardware reveals (like sensor noise and true physical limits) back into the sim and reward design.

CAD → sim training → export → real deployment, with real-hardware findings feeding back into reward and sim tuning

Training Architecture

A standard PPO setup, deliberately kept simple: the actor and critic see the exact same observation, nothing privileged or asymmetric between them. The only place privileged sim-only state (true torso height, tilt, velocity) gets used is reward computation, which trains both networks through the normal PPO loss. Difficulty (domain randomization, target walk height, learning-rate/entropy decay) is controlled by manual flags set at launch, not an automated curriculum.

Shared observation into a symmetric actor-critic; reward alone draws on privileged sim state

Training Convergence

Mean episode reward over ~14.5M PPO training steps for one of the walk fine-tuning runs (`back_clearance_v1`), a fairly clean, steady climb, typical of a run once the reward shaping for that stage had settled.

rollout/ep_rew_mean over training, PPO

Belt/Pulley Calf Decoupling

Most small quadrupeds direct-drive the calf from a motor mounted on the swinging thigh link. This robot instead mounts both the thigh and calf motors on the torso, and drives the calf joint through a timing belt routed through the thigh, keeping both motors' mass close to the body instead of out on the leg. The tradeoff: the calf's real-world angle is no longer just "thigh angle plus a hinge" the way a direct-drive design would be. Rotating the thigh alone doesn't move the calf at all; the belt cancels it out. The simulation has to account for that conversion explicitly so the trained policy and the real robot always agree on what a given calf command means.

For Contrast: A Direct-Drive (Coupled) Calf

A simulated demo of the traditional design this robot avoids, built purely to illustrate the difference described above.

Forward Lean While Walking

A gait that's rewarded for keeping the torso level ends up drifting backward as it walks, eventually tipping over onto its back. The fix: reward a deliberately forward-pitched torso instead of a level one, closer to how the robot naturally carries itself while moving, and it gives the policy a stable direction to lean into instead of hunting around level.

Why position control.

Because of the belt decoupling mechanism, when the hip (thigh) rotates, the knee and feet counter-rotate. As a result, the physical range of motion the knee joint is allowed to move in shifts dynamically based on the thigh's rotation. If we were to use torque control, this belt decoupling mechanism cannot be represented in a torque mode. We need to explicitly account for the belt decoupling and shifting joint limits, and we cannot calculate that via raw torque. Position control natively handles this by allowing us to calculate and command the precise, coupled target angles required.

Hardware

• Motors: 8x Unitree GO-M8010-6 actuators (one thigh and one calf per leg), all mounted on the torso to minimize leg inertia.
• Transmission: Calf motors drive their joint through a timing belt/pulley routed through the thigh, rather than a direct-drive gearbox.
• Compute: An onboard Jetson Orin Nano runs the sim-to-real policy bridge and reads real motor states over RS485.
• IMU: An LSM6DSO32 6-DoF Accelerometer/Gyroscope is used for orientation and angular-velocity sensing, calibrated directly on the real robot.
• Control Scheme: Position-mode (PD) motor control end-to-end; no torque-mode path exists on the hardware today.

Not Every Run Converges Cleanly

Fine-tuning a walk policy from a strong existing checkpoint climbed steadily for the first ~3M steps, then reward crashed to less than half its peak before slowly, partially recovering, while the policy's own action noise (its exploration std) never stopped climbing for the entire run. Consistent with this project's own separate finding that fine-tuning an already-good checkpoint at too high a learning rate can push it toward instability instead of refining it.

A real walk fine-tuning run: reward (top) destabilizes mid-run and partially recovers, while action noise (bottom) never settles

Sim-to-Real Challenges

• Motor sign conventions: which direction is "positive" for a given motor isn't uniform across the four legs; motors are mirror-mounted left-vs-right. An early assumption that it was uniform caused a long chain of hard-to-diagnose direction bugs, only fully resolved by directly measuring each joint's real motion rather than trusting any documented convention.
• An unmodeled passive spring: the physics model had an unreviewed default that added a small centering spring to every leg joint. Invisible during normal walking, but it fought the actuators hard at large joint angles (a real robot never has this, since without active control a motor just goes limp), found via a dedicated full-range sweep test and removed.
• A gait that looked right in sim didn't fully match on hardware: side-by-side sim/real data comparison showed the back legs settling into a steadier-state stance the policy never encountered during training, a genuine unmodeled difference between how sim and the real robot load the front vs. back legs, and the main driver behind the ongoing gait-refinement work below.
• Sensor noise: a policy that looked fully converged in sim still visibly chattered on real hardware, traced to the simulation's observations being perfectly clean; real encoders and the real IMU aren't. Fixed by training with injected sensor noise.

Build

Videos

Walking

Standing

Trotting

Galloping

Robot Walking on Different Terrains

Conclusions & Future Work

• Pushing peak running speed: The robot currently reaches up to 1.3 m/s in simulation. We are actively refining reward structures and discovering faster gait patterns to push this boundary further.
• Parameterized locomotion styles: While the policy currently discovers natural gaits (such as trotting, galloping, or walking) organically, it converges to a single style randomly. Future work focuses on training the robot to deterministically select specific styles (trot, gallop, canter, pace, amble, or walk) on demand via command parameters.
• Adding jump capabilities: Expanding the action space and training curriculum to enable dynamic jumping behaviors.