Q-learning — one step at a time

  • 1 select action a (ε-greedy)
  • 2 take a → observe reward r, next state s′
  • 3 look up best next value maxa′ Q(s′,a′)
  • 4 form target = r + γ·max
  • 5 TD error = target − Q(s,a)
  • 6 update Q(s,a) ← Q(s,a) + α·error
🐱 …
Q(s,a) ← · + α·( r + γ·max′ − Q ) = ·
episode 1
steps this ep 0
goals hit while exploring 0
Wedges = the 4 action-values (blue ≥ 0, red < 0).   Gray arrows = current best action per cell.   The thick route = where 🐱 goes from the start by following those arrows: green reaches 🏠, red = stuck in a loop (never reaches it).   Yellow arrow = the move being taken now.