Reinforcement Learning Quadruped

Overview

A custom-designed 8-DOF quadruped (thigh and calf per leg, no hip motors) driven by 8 Unitree GO-M8010-6 motors and controlled from an onboard Jetson. The goal: train a walking policy entirely in simulation with reinforcement learning (PPO, MuJoCo + Gymnasium), then deploy it on the real robot and close the loop between sim and hardware until it walks forward reliably in both.

• Credit: Benji B for creating the base robot.

Workflow

1. 1.Model the robot in Onshape CAD, exporting both a real mesh/mass model and a real joint-kinematics model.
2. 2.Generate a MuJoCo model combining the two into one physics model, plus the canonical motor-to-joint mapping every other stage shares.
3. 3.Train a PPO policy in a Gymnasium environment that sees exactly what the real robot senses (motor encoder angles/velocities and IMU data, nothing privileged like absolute position), first to stand, then to walk.
4. 4.Export the trained policy to TorchScript so it runs with just PyTorch at inference time.
5. 5.Run it on the real robot: read real motor + IMU data, build the same observation the policy trained on, command real motor targets, every control tick. Then feed what real hardware reveals (like sensor noise and true physical limits) back into the sim and reward design.

CAD → sim training → export → real deployment, with real-hardware findings feeding back into reward and sim tuning

Standing

Training Architecture

A standard PPO setup, deliberately kept simple: the actor and critic see the exact same observation, nothing privileged or asymmetric between them. The only place privileged sim-only state (true torso height, tilt, velocity) gets used is reward computation, which trains both networks through the normal PPO loss. Difficulty (domain randomization, target walk height, learning-rate/entropy decay) is controlled by manual flags set at launch, not an automated curriculum.

Shared observation into a symmetric actor-critic; reward alone draws on privileged sim state

Walking

Forward Lean While Walking

A gait that's rewarded for keeping the torso level ends up drifting backward as it walks, eventually tipping over onto its back. The fix: reward a deliberately forward-pitched torso instead of a level one, closer to how the robot naturally carries itself while moving, and it gives the policy a stable direction to lean into instead of hunting around level.

Training Convergence

Mean episode reward over ~14.5M PPO training steps for one of the walk fine-tuning runs (`back_clearance_v1`), a fairly clean, steady climb, typical of a run once the reward shaping for that stage had settled.

rollout/ep_rew_mean over training, PPO

Belt/Pulley Calf Decoupling

Most small quadrupeds direct-drive the calf from a motor mounted on the swinging thigh link. This robot instead mounts both the thigh and calf motors on the torso, and drives the calf joint through a timing belt routed through the thigh, keeping both motors' mass close to the body instead of out on the leg. The tradeoff: the calf's real-world angle is no longer just "thigh angle plus a hinge" the way a direct-drive design would be. Rotating the thigh alone doesn't move the calf at all; the belt cancels it out. The simulation has to account for that conversion explicitly so the trained policy and the real robot always agree on what a given calf command means.

For Contrast: A Direct-Drive (Coupled) Calf

A simulated demo of the traditional design this robot avoids, built purely to illustrate the difference described above.

Why position control.

Because of the belt decoupling mechanism, when the hip (thigh) rotates, the knee and feet counter-rotate. As a result, the physical range of motion the knee joint is allowed to move in shifts dynamically based on the thigh's rotation. Raw torque control can't represent that shifting coupling directly, so any torque-based scheme still has to separately track and correct for it. An impedance controller (a torque/position hybrid where a commanded target angle relies on virtual stiffness and damping to produce torque) is a real, viable way to handle this too, and was on the table. But it adds a second layer of tuning on top of the belt-coupling bookkeeping: choosing and validating stiffness and damping gains around a moving, coupled target, not just computing the target itself. Position control needed only the latter. Calculating the precise, coupled target angles and commanding them directly provided a simpler mental model and a much shorter path to a working controller. Working under a hard project deadline, that faster, more tractable path won out over the theoretically more capable but slower-to-get-right impedance approach.

Controlling Speed: Max Degrees per Second

Position control alone doesn't set how fast the robot moves; that comes from a separate per-motor cap on how quickly a commanded target angle is allowed to change, in degrees per second. Early in training this cap is left deliberately loose (1000 deg/s) so PPO has enough freedom to discover a working gait at all; a tighter cap that early had previously stopped a gait from being found in the first place. Once a gait is established, a curriculum tightens the cap over training, ramping it down to 250 deg/s, the same limit already enforced on the real robot's deployment node. Training against that same real-world limit closes a sim-to-real gap: the policy learns to work within the actual speed the hardware allows, rather than relying on motor speeds sim would permit but the real motors can't deliver.

Trotting

Hardware

• Motors: 8x Unitree GO-M8010-6 actuators (one thigh and one calf per leg), all mounted on the torso to minimize leg inertia.
• Transmission: Calf motors drive their joint through a timing belt/pulley routed through the thigh, rather than a direct-drive gearbox.
• Compute: An onboard Jetson Orin Nano runs the sim-to-real policy bridge and reads real motor states over RS485.
• IMU: An LSM6DSO32 6-DoF Accelerometer/Gyroscope is used for orientation and angular-velocity sensing, calibrated directly on the real robot.
• Control Scheme: Position-mode (PD) motor control end-to-end; no torque-mode path exists on the hardware today.

Galloping

Not Every Run Converges Cleanly

Fine-tuning a walk policy from a strong existing checkpoint climbed steadily for the first ~3M steps, then reward crashed to less than half its peak before slowly, partially recovering, while the policy's own action noise (its exploration std) never stopped climbing for the entire run. Consistent with this project's own separate finding that fine-tuning an already-good checkpoint at too high a learning rate can push it toward instability instead of refining it.

A real walk fine-tuning run: reward (top) destabilizes mid-run and partially recovers, while action noise (bottom) never settles

Cantering

Sim-to-Real Challenges

• Motor sign conventions: which direction is "positive" for a given motor isn't uniform across the four legs; motors are mirror-mounted left-vs-right. An early assumption that it was uniform caused a long chain of hard-to-diagnose direction bugs, only fully resolved by directly measuring each joint's real motion rather than trusting any documented convention.
• An unmodeled passive spring: the physics model had an unreviewed default that added a small centering spring to every leg joint. Invisible during normal walking, but it fought the actuators hard at large joint angles (a real robot never has this, since without active control a motor just goes limp), found via a dedicated full-range sweep test and removed.
• A gait that looked right in sim didn't fully match on hardware: side-by-side sim/real data comparison showed the back legs settling into a steadier-state stance the policy never encountered during training, a genuine unmodeled difference between how sim and the real robot load the front vs. back legs, and the main driver behind the ongoing gait-refinement work below.
• Sensor noise: a policy that looked fully converged in sim still visibly chattered on real hardware, traced to the simulation's observations being perfectly clean; real encoders and the real IMU aren't. Fixed by training with injected sensor noise.

Reward Shaping

Each gait shown on this page came from its own reward-shaping pass: the same PPO setup, but a different sequence of small, targeted reward and termination changes to get from a generic locomotion objective to that specific gait's behavior. These are the real iterations, pulled from the training history, not a stylized summary.

Build

Videos

Robot Walking on Different Terrains

Different Gaits While Suspended

Conclusions & Future Work

• Pushing peak running speed: The robot currently reaches up to 1.3 m/s in simulation. We are actively refining reward structures and discovering faster gait patterns to push this boundary further.
• Parameterized locomotion styles: Each gait shown above (walk, trot, gallop, canter) is currently the result of its own deliberately trained policy, targeted at that specific style through reward shaping. Future work focuses on unifying these into a single policy that can deterministically switch between styles (trot, gallop, canter, pace, amble, or walk) on demand via command parameters.
• Adding jump capabilities: Expanding the action space and training curriculum to enable dynamic jumping behaviors.