ARC's Summer of Simulation

Introduction

Welcome to the first issue of the ARC newsletter.

This summer ran on one idea: before you build a robot in the real world, you build it a thousand times in simulation. We called the program The Sims, a five-week climb that takes a complete beginner from spawning a robot in a physics engine to training one to walk with reinforcement learning. This issue walks through what we set out to learn, the tools we used, and what happened week by week, and it ends with a sneak peek at the task waiting for this year’s incoming juniors.

Learning Objectives

By the end of the program, a member should be able to:

Spawn and control a robot in MuJoCo, and read the .mjcf model format.
Derive forward and inverse kinematics, then generate trajectories and gaits from them.
Explain the RL fundamentals: value functions, policy gradients, actor-critic, and PPO.
Train a quadruped locomotion policy on the mjlab + RSL_RL stack.
Build a custom RL environment from scratch: its observations, its action space, and its reward terms.

Tech Stack

Five libraries, each with exactly one job, plus the tools we used to track and run everything.

Layer	Tool	Its one job
Physics	MuJoCo	Simulate bodies, contacts, gravity, and torques
GPU physics	MuJoCo Warp	Run thousands of those simulations in parallel
RL environment	mjlab	Define observations, actions, rewards, terminations
Tensor math	PyTorch	Batched GPU math and automatic gradients
Learning algorithm	RSL_RL	Run PPO and train the actor-critic networks
Experiment tracking	Weights & Biases	Log reward curves and checkpoints
Robot middleware	ROS	Sensor and command topics (micromouse task)

Timeline

Week 1: Spawn and Control

The first job was getting our two robots to simply exist in MuJoCo. We loaded the SO-100 arm and the Stanford Pupper, got comfortable with the .mjcf format (a cousin of URDF/xacro, not the same thing), checked mass, joint and actuator limits and sensor configs, sent raw torque commands, logged proprioceptive sensor data, and wired up sliders to drive joints by hand.

The SO-100 robotic arm spawned in the MuJoCo viewer

The SO-100 arm in the raw MuJoCo viewer.

The Stanford Pupper quadruped loaded and standing in MuJoCo simulation

The Stanford Pupper, loaded and settling under gravity. This is the quadruped we spent the rest of the summer teaching to walk.

Week 2: Kinematics and Gaits

Next came the math that makes limbs move on purpose. We derived forward and inverse kinematics geometrically, drove the SO-100’s end effector through a set of points using cubic-spline trajectories, and gave the Pupper a sine-wave gait: vary each foot’s X and Z position sinusoidally, offset the phase per leg, and the dog walks.

Weeks 3 and 4: RL Foundations

Two weeks inside the HuggingFace Deep RL Course, building the vocabulary that the training code later depends on: states, actions, rewards, value functions, policy gradients, actor-critic, and PPO.

A slide from the HuggingFace Deep RL Course showing the action-value function on a mouse-and-cheese gridworld

The action-value function, illustrated with the classic mouse-and-cheese gridworld: from any square, taking any step, how good is the future?

Week 5: Locomotion with Reinforcement Learning

This is where the stack comes together. Before training anything, we confirmed the plumbing worked by loading one of mjlab’s shipped demos and hitting play: a pretrained humanoid dancing across the viewer, no training involved, purely a check that the GPU and install were happy.

With the stack confirmed, our first real milestone was the Unitree Go1, which ships with a ready-made mjlab environment. Training it end to end proved the learning side of the stack before we built anything of our own:

# Train the shipped Go1 velocity task across 512 parallel robots
uv run train Mjlab-Velocity-Flat-Unitree-Go1 --env.scene.num-envs 512

Then you watch the reward curves climb on Weights & Biases as a flailing pile of joints turns into a walk:

The real engineering is not the command above; it is building the environment itself. That means deciding what the robot senses (its 12 joint angles and velocities, its torso orientation, its foot position vectors, its last action) and what earns it reward. A useful detail: the policy never outputs torque. It outputs a small offset to a default joint position, which a built-in PD controller converts into torque.

policy output  →  scaled offset  →  target joint position  →  PD controller  →  torque  →  physics

The reward is a weighted sum of hand-designed terms: stay upright, hold a target torso height, track the commanded velocity, avoid wasting energy, avoid dragging feet. Tuning those terms for a specific robot is the work that carries us from the shipped Go1 to our own quadruped environments.

Sneak Peek: The CNI Micromouse Task

Alongside the training, a chunk of the summer went into the tasks that will welcome this year’s incoming juniors. The one we are really excited about is a micromouse maze solver.

We made a micromouse simulation, driven entirely through ROS topics. It does not get a map. It has to subscribe to its wall sensors, reason about which way is open, and publish velocity commands to move, feeling its way to the center of the maze one cell at a time.

   sensor topics  ──▶  [ your maze-solving node ]  ──▶  velocity command topic
   (what walls?)          (flood fill? wall follow?)        (drive!)

Flood fill, wall follower, or something smarter is left for the juniors to figure out. Here is a first look at the task in action:

Member of the Month

Every issue, we crown a Member of the Month: the person who contributed the most and clearly stood out.

For this Summer of Simulation, it is Pratyush Priyadarshi. From designing the CNI micromouse tasks, to simulating the Pupper, to improving and maintaining the very website you are reading this on, he showed up wherever the hard work was and quietly turned it into something that worked.

ARC is deeply, sincerely proud to have a genius like Pratyush in our ranks, and we can only hope this newsletter does not tip off Crux or SWD, because we would really rather not lose him. (If you are reading this, Pratyush: we said nice things. Please stay.)

Follow Along

The entire curriculum, week by week, is public. Clone it, fork it, and build your own walking robot:

👉 github.com/arc-bphc/the-sims

Sources and External References

The Sims curriculum: ARC’s five-week simulation track.
MuJoCo documentation: the physics engine.
MuJoCo Menagerie: reference .mjcf models for many robots.
urdf2mjcf: URDF to MuJoCo converter.
HuggingFace Deep RL Course: the RL foundations for Weeks 3 and 4.
mjlab (docs): the RL environment framework.
RSL_RL: ETH Zürich’s PPO / actor-critic implementation.
ANYmal velocity task: a complete custom-robot integration blueprint.
mjlab_playground: additional task examples.
pip-loco: a worked, end-to-end RL locomotion codebase.
Weights & Biases: experiment tracking.

Acknowledgements and Credits

Written by: Yash Bhatt and Rayan Das Gupta

Photos and videos: Yash Bhatt and Rayan Das Gupta

Special thanks: Karan Nair for guiding us throughout the summer And obviously Thanks to Mr. President Aviral Maindola and Prateek Sharma for Architecting the sims repository and preparing such great learning opportunities for the people of this club

See you in the next one.

The Team

← Back to newsletter

Table of contents