Can a network learn XOR?
— 2 inputs → 2 ReLU hidden units → 1 output, full-batch gradient descent on MSE
XOR: output 1 when exactly one input is 1. The four points below are the whole dataset — no single line separates the blues from the reds.
predicted class 1
predicted class 0
ring:
green = correct
·
yellow dashed = wrong
Linear only (no hidden layer)
2 ReLU hidden units
Train (gradient descent)
Randomize weights
loss
–
· solves XOR:
–
Hand-shape the hidden units · h₁ = ReLU(w₁₁x₁ + w₁₂x₂ + b₁)
w₁₁
w₁₂
b₁
h₂ = ReLU(w₂₁x₁ + w₂₂x₂ + b₂)
w₂₁
w₂₂
b₂
The output layer (v₁, v₂, c) auto-fits by least squares as you drag — try w₁₁=1, w₁₂=1, b₁=−0.5 and w₂₁=1, w₂₂=1, b₂=−1.5 (OR minus AND).