Tiny Neural Network: XOR
One step = one gradient-descent update that uses all four examples: every weight and bias moves by −α times its partial derivative of L.
Each challenge checks itself. Work on this keeps it in view above the controls.
Train a small XOR network (2 inputs, a sigmoid hidden layer of 2, a linear output, mean squared error) with backpropagation; step through one forward and backward pass with every number shown, watch the loss plateau and the decision boundary form.
How to use
- This is a small XOR network: 2 inputs, a hidden layer of 2 sigmoid neurons, and a linear output ŷ = W[2]a + b[2], trained on the mean squared error L. It should output 1 at (0, 1) and (1, 0) and 0 at (0, 0) and (1, 1): the XOR function, which no single straight line can separate.
- It opens at the standard starting weights (W[1] = I, b[1] = 0, W[2] = [1 1], b[2] = 0) with α = 0.1. Press a training button: each step is one gradient-descent update on L. Watch the loss curve and the decision map. Pause stops a long run.
- In the decision map, blue areas with diagonal hatching are where ŷ > ½ (the network answers 1); red areas are where ŷ < ½ (it answers 0). The thick black curve is the boundary ŷ = ½. Targets y = 1 are filled squares, targets y = 0 are open circles, and the number beside each point is its prediction ŷ.
- The dashed and dotted lines are where hidden neuron 1 and hidden neuron 2 switch (z1 = 0 and z2 = 0, where ai = ½). The hidden layer bends the plane so that one linear output can separate what a single line cannot.
- “One step, all numbers” shows every intermediate value for one example and the full update, so you can check your hand computations.
- Try “All weights equal”: the two hidden neurons then stay identical forever. That is why training starts from random weights.
- Keyboard: every control is a standard form field. The status box is read aloud by screen readers. Training is shown step by step unless your system asks for reduced motion.
Accessibility: every control is a standard form control you can reach with Tab and change with the arrow keys; the status box is announced by screen readers; the picture has a text description (press Describe) that updates as you work; animations are skipped when your system asks for reduced motion.
Decision map
Network output ŷ(x1, x2) over the square −0.5 ≤ x1, x2 ≤ 1.5.
Loss
One step, all numbers
Forward pass: all four examples
| k | x(k) | y(k) | a(k) = σ(z(k)) | ŷ(k) | e(k) = ŷ(k) − y(k) | Reads as | Correct? |
|---|
“Reads as” reads ŷ > ½ as 1 and ŷ < ½ as 0.
One example: forward and backward
The forward pass and backpropagation for one example with the current weights: z(k) = W[1]x(k) + b[1], a(k) = σ(z(k)), ŷ(k) = W[2]a(k) + b[2], with the error e(k) = ŷ(k) − y(k) and the hidden-layer error δ(k), with entries δ(k)j = w[2]j e(k) σ′(z(k)j). W[1] is 2×2 and W[2] is 1×2 (a row).
| Quantity | Formula | Value |
|---|
The full update (all four examples)
By backpropagation (the chain rule) each gradient is 2⁄N = ½ times the sum of the four terms k = 1, …, 4 in the table above. The output layer is listed first.
| Parameter | Now | ∂L/∂(parameter) = ½ Σk (term k) | After one step (α times the gradient subtracted) |
|---|