Tabular RL on a Robot I Built

Sutton & Barto, but on a custom robot in PyBullet instead of a grid world

Main repo Monte Carlo Temporal Difference n-step Bandits

TL;DR I designed a small pick-up robot in Onshape, simulated it in PyBullet and solved it with ten classic RL algorithms from Sutton & Barto, from Monte Carlo up to n-step Q(σ).

Why

Working through Reinforcement Learning: An Introduction, I got tired of grid worlds. I wanted the algorithms to control something physical, so I built a robot for them. Two robot designs failed before this one worked, and I ran a lot of PyBullet experiments before the environment behaved.

The robot

CAD render of the pick-up robot: a rotating stand, a vertical slider arm and a two-part gripper. — **The pick-up bot**, designed in Onshape: a stand that rotates a full 360°, a slider that moves the arm ±210 mm, and two gripper jaws.

The state is a discretised tuple (stand rotation, slider height, gripper 1, gripper 2), which gives 24 reachable configurations. The goal is to find the most efficient sequence of moves to a target state.

Algorithms

Monte Carlo: on-policy control without exploring starts, and off-policy control with importance sampling
Temporal difference: SARSA, Q-learning, Expected SARSA and Double Q-learning
n-step bootstrapping: n-step SARSA, n-step off-policy learning, n-step tree backup and n-step Q(σ)
Plus k-armed bandits as the warm-up

One thing that tripped me up

PyBullet position targets are relative to where each joint started, not where it is now. So "move down 0.21" means something different depending on where the arm already is. Until I shifted every command by the joint's original pose, the robot kept overshooting, and the whole environment design depended on getting that right.