Pendulum DQN

Swing a pendulum up and hold it upright, using a torque of at most 2 in either direction. The classic Gym task is continuous control, and DQN only chooses between a fixed set of actions, so the first question is whether a discretised DQN can balance it reliably at all. The second is what happens when gravity changes: normal, zero, reversed and stronger.

Reinforcement learning results are easy to fool yourself with. One lucky training seed, a checkpoint picked because it scored well on the same states it is reported on, or a start position that happens to be easy, and a policy looks better than it is. Most of my part was building the test so that couldn’t happen.

The state is the pendulum’s angle as cosine and sine, plus its angular velocity. Three, five and seven evenly spaced torques were compared; seven gave far smoother control, switching action on 46% of steps instead of 74% to 78%, and was kept. Vanilla, Double and Dueling DQN were then compared under matched conditions, along with exploration, replay, target-network and optimisation settings.

• Network

  Value Dueling DQN, two hidden layers of 64, 4,936 parameters

• Actions

  Value 7 torques from −2 to 2

• Exploration

  Value ε from 1.00 to 0.05 over 20,000 steps

• Replay and target

  Value 10,000 transitions, batch 32; target copied every 250 updates

• Budget

  Value 150 episodes, 30,000 steps, evaluated every 25 episodes

Dueling beat Vanilla by 3.88 in mean return and won 87.5% of paired start states, for 65 extra parameters. That lead is inside the noise margin described below, so it is reported as a practical choice, not a significant one. Three fresh seeds then confirmed the frozen recipe at −139.08 ± 0.98.

• Development states24 fixed starting positions covering near-upright, sideways and hanging, spinning both ways. All comparisons, tuning and checkpoint choices used only these.
• Held-out statesA second set of 24, checked for no overlap, sealed through every training, selection and gravity decision, and opened once, after all four policies were frozen.
• A noise marginThree seeds of the same baseline differed by up to 11.34 in mean return. Any difference smaller than that counts as practically close and needs more seeds on both sides before it can decide anything.
• Matched seedsCandidate and reference always trained on the same seed and scored on the same states; all four gravity runs used seed 4242.
• Same-gravity referencesEach policy is compared with a random and an untrained policy at its own gravity. Returns across gravities aren’t ranked against each other, because the physics differs.

The frozen recipe trained four separate fresh policies, not one policy that adapts. Each had a stated hypothesis first: reversed gravity pulls toward upright, so holding it should get easier; stronger gravity should make the swing-up and the hold harder with the same torque limit.

• Default, g = 10

  Checkpoint chosen Episode 50Upright fraction, dev 0.864Held-out mean return −135.38

• Zero, g = 0

  Checkpoint chosen Episode 50Upright fraction, dev 0.911Held-out mean return −53.77

• Anti-gravity, g = −10

  Checkpoint chosen Episode 125Upright fraction, dev 0.903Held-out mean return −53.37

• Supergravity, g = 15

  Checkpoint chosen Episode 125Upright fraction, dev 0.827Held-out mean return −203.31

Returns are negative costs, so closer to zero is better. For scale, a random policy at default gravity averaged −1,220 on the development states.

• Seal the held-out states until everything is frozen

  If held-out states inform any choice, they stop being held out. They were defined early for reproducibility, but only their count and an overlap check were shown until the architecture, recipe and all four policies were final.

• Measure the noise before comparing anything

  Without knowing how much one recipe varies between seeds, every small win looks real. Three baseline seeds set an 11.34 margin, locked before the comparisons it would judge.

• Choose checkpoints on development states only

  The supergravity policy peaked at episode 125 with −204.89 and fell to −325.84 by episode 150. Picking the best checkpoint is legitimate; picking it on the states you then report on is not. The choice was fixed before the held-out suite opened.

• Compare each gravity with itself

  A pendulum is easier to hold when gravity points up. Ranking the anti-gravity policy above the default one would reward the physics, not the learning, so each is judged against random and untrained policies in its own world.

• Prove the saved weights are the ones reported

  Each file was rebuilt into a fresh Dueling network and re-scored on the held-out states, reproducing the stored results exactly. One file was left out because its hash matched another byte for byte.

• DQN needs the continuous torque discretised; an actor-critic method could use it directly.
• Training stops at 150 episodes, and the supergravity collapse shows how much a checkpoint choice matters at this budget.
• The recipe was tuned at default gravity; the other three weren’t tuned independently and may not be at their best.
• The saved files are inference weights only, without replay memory or optimiser state, so training can’t be resumed exactly.
• Smoothness is measured by action switching, a proxy for actuator wear, not wear itself.