Computer Science › Section L · Artificial intelligence › Topic L.5

Artificial neural networks

Instead of following instructions written step by step, an artificial neural network learns its behaviour from examples by adjusting thousands, or billions, of numerical weights. This topic builds one neuron at a time, shows what a single neuron cannot do, and explains how networks are trained.

Inspired by the brain

Tasks such as recognising a face resist step-by-step programs, so researchers borrowed an idea from biology. A biological neuron receives signals through branching dendrites and sends its own along an axon. Synapses, the junctions between neurons, make an incoming signal more or less likely to excite the receiving cell, and changes in their strength are believed to underlie learning. The human brain has roughly 86 billion neurons, by a widely cited estimate, each linked to thousands of others. Artificial networks copy only the outline of this picture.

An artificial neuron

An artificial neuron is a small computing unit. It receives inputs \(x_1, x_2, \ldots\), each multiplied by a weight \(w_1, w_2, \ldots\), and adds the products to get its weighted sum. In the simplest kind, a threshold unit, the output is 1 if that sum exceeds a fixed threshold t, and 0 otherwise. A positive weight makes its input excite the neuron, pushing the sum up; a negative weight makes it inhibit, pulling the sum down. The size of a weight sets how strongly that input counts. Changing the weights and threshold changes what the neuron computes, with no change to its program.

Worked example: computing a neuron's outputs

A neuron has weights 2, −1 and 1 and threshold 1.5. For which inputs, each 0 or 1, is its output 1? 1. Weighted sum: \(2x_1 - x_2 + x_3\). 2. Inputs 1, 0, 0 give 2, which exceeds 1.5, so the output is 1. 3. Inputs 1, 0, 1 give 3: output 1. 4. Inputs 1, 1, 1 give 2: output 1. 5. Inputs 1, 1, 0 give only 1, below 1.5: output 0. 6. With \(x_1 = 0\) the sum is at most 1, so every such input gives 0. 7. Conclusion: the neuron fires exactly when \(x_1 = 1\) and it is not true that \(x_2 = 1\) while \(x_3 = 0\). The second input acts as an inhibitor that the third can cancel.

Neurons as logic gates

Single neurons can act as the gates of 'Bits, gates, and flip-flops'. With weights 1 and 1, a threshold of 1.5 gives AND, since only 1 + 1 exceeds it, and a threshold of 0.5 gives OR. A single input with weight −1 and threshold −0.5 gives NOT. XOR is different: no single threshold neuron can compute it. A neuron's output changes along one straight line through the space of inputs, and no straight line separates (0, 1) and (1, 0), which should give 1, from (0, 0) and (1, 1), which should give 0. Marvin Minsky and Seymour Papert's 1969 book Perceptrons made this limitation famous.

Layers of neurons

Combining neurons removes the limitation. A typical network is arranged in layers: an input layer that receives the data, one or more hidden layers, and an output layer that gives the answer. In a fully connected, feed-forward network, every neuron in one layer sends its output to every neuron in the next, and signals flow only forwards. A network with 4 inputs, 3 hidden neurons and 2 outputs has 4 × 3 + 3 × 2 = 18 weights, plus a threshold for each of its 5 non-input neurons; practical networks have millions or billions.

Worked example: a network for XOR

Hidden neuron H1 is OR: weights 1, 1, threshold 0.5. Hidden neuron H2 is NAND: weights −1, −1, threshold −1.5, so it outputs 1 unless both inputs are 1. The output neuron is AND of H1 and H2: weights 1, 1, threshold 1.5. 1. Inputs 0, 0: H1 = 0; H2's sum 0 exceeds −1.5, so H2 = 1; the output sum 1 gives 0. 2. Inputs 0, 1: H1 = 1, and H2's sum −1 exceeds −1.5, so H2 = 1; the output sum 2 gives 1. 3. Inputs 1, 0: by symmetry, 1. 4. Inputs 1, 1: H1 = 1, but H2's sum −2 is below −1.5, so H2 = 0; the output sum 1 gives 0. 5. Check: 0, 1, 1, 0 is XOR, 'OR and not both'.

Training instead of programming

A network's weights are not worked out by hand: it is trained by supervised learning (see 'Additional areas: knowledge and learning'), comparing its output on each example with the correct answer and nudging the weights to shrink the error. Frank Rosenblatt's perceptron of 1958 used a simple rule for one threshold neuron with learning rate η:

w_i \leftarrow w_i + \eta\,(\text{target} - \text{output})\,x_i

with the threshold moved the opposite way, \(t \leftarrow t - \eta\,(\text{target} - \text{output})\). A correct output changes nothing; a wrong one shifts the neuron towards the right answer.

Worked example: training a neuron to compute OR

Start with \(w_1 = w_2 = 0\), t = 0 and η = 0.5, presenting inputs 00, 01, 10, 11 in each pass. 1. Pass 1: input 01 gives sum 0, not above 0, so output 0, target 1. The error is +1, so \(w_2\) becomes 0.5 and t becomes −0.5. 2. Pass 2: input 00 now gives 0 > −0.5, output 1, target 0; t rises back to 0. Input 10 gives 0, not above 0, output 0, target 1; \(w_1\) becomes 0.5 and t −0.5. 3. Pass 3: input 00 is wrong again, so t returns to 0. 4. Pass 4: every output is right, so training stops with \(w_1 = w_2 = 0.5\) and t = 0. 5. Check: 00 gives 0; 01, 10 and 11 give 0.5, 0.5 and 1, all above 0: OR.

Smooth neurons and backpropagation

Multilayer networks are trained with smooth neurons whose outputs vary gradually, such as the sigmoid \(\sigma(z) = \frac{1}{1 + e^{-z}}\), which turns a sum of 0 into 0.5 and a sum of 2 into about 0.88, or the simpler ReLU, which outputs the larger of 0 and the sum. A loss function measures the total error. Gradient descent then changes every weight a small step in the direction that reduces the loss, and backpropagation, popularised in 1986 by David Rumelhart, Geoffrey Hinton and Ronald Williams, computes those directions efficiently, working backwards from the output layer. Repeating this over many examples gradually shapes the network.

Deep learning: power and limits

Networks with many layers, called deep learning, became practical in the 2010s, helped by huge labelled data sets and fast GPUs (see 'Other architectures: pipelining and multiprocessors'). Convolutional networks transformed image tasks, and the transformer, an architecture introduced in 2017, underlies most large language models. Networks have limits too. They need much data and energy to train, their weights are hard to interpret, so decisions are hard to explain, they absorb bias in their training data, and small changes to an input, invisible to people, can sometimes fool them.

Common mistakes to avoid

First, forgetting the strict inequality: a sum equal to the threshold gives 0. Second, ignoring signs: a negative weight inhibits. Third, believing one neuron can compute any logic function, when XOR needs a hidden layer. Fourth, adding layer sizes to count weights instead of multiplying neighbouring layers. Fifth, updating weights after a correct output, when the perceptron rule changes nothing. Sixth, thinking networks are programmed weight by weight; they are trained, and are only as good as their data.

Key takeaways

A neuron forms a weighted sum of its inputs; a threshold unit outputs 1 if the sum exceeds its threshold. Positive weights excite, negative weights inhibit. Single neurons compute AND, OR and NOT but not XOR, which needs a hidden layer. Between two full layers, the number of weights is the product of their sizes. Networks are trained, not programmed: the perceptron rule nudges weights after errors, and multilayer networks use smooth neurons, gradient descent and backpropagation. Deep networks power modern vision and language but are data-hungry, opaque and can inherit bias.

Flashcards

Flashcard

Synapse

Junction between neurons that strengthens or weakens a signal

Changes here are believed to underlie learning.

Test your understanding

Question 1 of 5

A threshold neuron has weights 3, −2 and 1 and threshold 1.5. Which input gives an output of 1?

Question 2 of 5

A threshold neuron has weights 1 and 1 and threshold 2. What does it output when both inputs are 1?

Question 3 of 5

A fully connected feed-forward network has 3 inputs, 4 hidden neurons and 2 outputs. How many weights does it have?

Question 4 of 5

Why can no single threshold neuron compute XOR?

Question 5 of 5

A neuron has weights 0.5 and 0, threshold 0.5 and learning rate 0.5. Input 1, 1 has target 1. What does one perceptron update give?

← Previous topicNext topic →