Computer Science › Section L · Artificial intelligence › Topic L.2
Perception: vision and language
Before an agent can act sensibly it has to make sense of what it sees and hears. This topic follows how machines recognise symbols and scenes in images and how they untangle the meaning of sentences, from early hand-built methods to the learned systems that now dominate.
Perceiving is more than recording
To act sensibly, an agent must make sense of what its sensors report. A thermometer's reading is already a number, but images and speech are not. A camera captures millions of pixel values in an instant, yet storing an image is not understanding it. Perception means extracting the information an agent needs, such as which digit is written in a box or what a spoken sentence asks for. People do this without effort, but it proved among the hardest problems in AI.
Worked example: matching a digit to templates
Digits are read on a grid 3 pixels wide and 5 high, 1 for ink and 0 for blank. Stored templates, row by row: 1 is 010, 110, 010, 010, 111; 7 is 111, 001, 010, 010, 010; 4 is 101, 101, 111, 001, 001. The input is 111, 001, 011, 010, 010. 1. Compare the input with each template pixel by pixel, counting agreements out of 15. 2. Template 1 agrees in 7 places, template 4 in 8 and template 7 in 14. 3. The best score wins: the digit is 7, despite a stray pixel in row 3. 4. The weakness: the same 7 drawn larger, smaller or one column to the right would no longer line up with its template.
Matching features instead
A more flexible method describes each symbol by its features rather than its exact pixels: a 0 is one closed loop, an 8 is two stacked loops, a 1 is a single upright stroke. Recognition takes two steps: extract the features from the image, then compare them with the feature lists of known symbols. Size and position matter less, because a large 8 and a small 8 both have two loops. But a small flaw can change the features completely: a gap in the outline of an O makes its loop an open curve, so it may be read as a C or a U.
From pixels to meaning
Understanding a scene is usually split into two stages. Image processing finds structure in the pixels. Smoothing removes speckles of noise, though too much blurs away detail. Edge detection, or edge enhancement, marks places where brightness changes sharply, turning a photograph into something like a line drawing. Region finding, or segmentation, groups neighbouring pixels of similar colour, brightness or texture, which probably belong to one object. Image analysis then decides what those edges and regions are. That is the hard part: objects may be partly hidden, seen from odd angles or lit differently, and a mug from above looks nothing like a mug from the side.
Worked example: finding edges in a row of pixels
One row of a greyscale image has brightness values 20, 22, 21, 90, 92, 91, 30, 28, where 0 is black and 255 white. Find the edges, using the rule that an edge lies where neighbouring pixels differ by more than 30. 1. Subtract each value from the next: 2, −1, 69, 2, −1, −61, −2. 2. Only 69 and −61 exceed 30 in size. 3. So there are edges between pixels 3 and 4 (dark to bright) and between pixels 6 and 7 (bright to dark): a bright bar three pixels wide. 4. Compare smoothing: averaging each value with its two neighbours turns the speck in 50, 50, 200, 50, 50 into 100 at each of the middle three pixels, weakening noise but also blurring.
Why natural language is hard
Encouraged by how well compilers translate programming languages (see 'Language implementation: translation and compilers'), many early researchers expected machines to understand English within a few years. A programming language is designed so that every statement has one structure and one meaning. Natural language is full of ambiguity: 'I saw the man with the telescope' does not say who held the telescope. The intended meaning may also differ from the literal one: 'Can you pass the salt?' is a request, not a question about ability. People resolve all this with knowledge of the world and of the situation.
Three levels of analysis
Understanding a sentence involves several interacting levels. Syntactic analysis, mainly parsing, finds the grammatical structure: in 'The parcel was delivered by Kemi' the grammatical subject is 'the parcel'. Semantic analysis finds each word's role in the meaning: the action is delivering, the agent is Kemi and the object is the parcel, exactly as in 'Kemi delivered the parcel', so the two sentences mean the same despite different subjects. Contextual analysis brings in the situation: 'It's freezing in here', said to someone sitting by an open window, is really a request to close it.
Worked example: 'I saw her duck'
Analyse 'I saw her duck' at each level. 1. Syntax: two parses exist. Either 'duck' is a noun and 'her' says whose it is (I saw the duck belonging to her), or 'duck' is a verb and 'her' is the person doing it (I saw her drop her head quickly). 2. Semantics: in the first, the only action is my seeing and its object is a bird; in the second, she is the agent of a second action, ducking. 3. Context decides: after 'We walked past the farm pond' the bird is likely; after 'The ball flew straight at her head' the movement is. 4. Conclusion: the levels are not independent, because the right parse depends on context.
Working with whole documents
Information retrieval means finding the documents relevant to a need, as a search engine does. Matching keywords alone gives false leads and misses pages that use a synonym, such as 'physician' for 'doctor', so practical systems also try to capture what words mean. Information extraction means pulling specific facts out of text into a structured form. One classic tool is a frame, a template of slots: a frame for job adverts might have slots for job title, employer, location and salary, filled in from each advert. Another, the semantic net, appears in 'Additional areas: knowledge and learning'.
Learning to perceive
Most current vision and language systems are not hand-written rules but neural networks (see 'Artificial neural networks') that learn from huge numbers of examples. Trained on labelled images, a network's early layers tend to detect edges and textures and its later layers larger parts and objects; after a deep network won a major image-recognition contest by a wide margin in 2012, the approach spread quickly. Large language models are networks trained on vast amounts of text to predict the next word or word fragment; with further tuning they translate, summarise and answer questions fluently. Both have limits: vision systems can fail in conditions unlike their training data, and language models can state falsehoods confidently. Whether they understand is debated.
Common mistakes to avoid
First, expecting template matching to cope with changes in size, position or handwriting style. Second, over-smoothing, which removes the very edges recognition needs. Third, treating parsing as understanding: two sentences with different subjects can mean the same thing, and one sentence can have two parses. Fourth, taking every sentence literally, when 'Can you pass the salt?' is answered by passing it. Fifth, assuming fluent output proves accuracy: a language system can sound sure and still be wrong.
Key takeaways
Perception means extracting meaning from sensor data, not just storing it. Template matching compares pixels with stored patterns and breaks when size or position changes; feature matching compares loops and strokes. Image processing (smoothing, edge detection, region finding) prepares an image; image analysis decides what its parts are. Ambiguous natural language needs syntactic, semantic and contextual analysis, which interact. Information retrieval finds relevant documents; information extraction fills structures such as frames. Modern systems learn perception from large data sets but can still fail or mislead.
Flashcards
Flashcard
Extracting the information an agent needs from sensor data
Test your understanding
Question 1 of 5
Using the 3-by-5 templates from the worked example, in how many of the 15 pixels does the input 111, 001, 010, 110, 010 agree with the template for 1?
Question 2 of 5
A row of pixels reads 15, 18, 80, 84, 82, 20, 19. With an edge wherever neighbours differ by more than 30, where are the edges?
Question 3 of 5
Recognising that 'The vase was broken by Tom' and 'Tom broke the vase' describe the same event is the job of which level of analysis?
Question 4 of 5
Which step belongs to image analysis rather than image processing?
Question 5 of 5
A system reads thousands of job adverts and records the job title, employer and salary of each in a table. What task is this?
