Each click rolls out one trajectory from every state, following the policy and summing discounted rewards G = Σ γ^kR. This MDP has no end state, so a trajectory would run forever — but the discounted tail γ^H shrinks to ~0, so we sum a finite horizon of 80 steps and stop (the horizon grows as γ → 1). Each state's running average (colored line) homes in on its exact value (dashed). The left panel animates the trajectory from Junk.