Computer Science › Section H · Software engineering › Topic H.6

Quality assurance and testing

No large program is free of errors, but disciplined teams find most of them before users do. Quality assurance covers everything that keeps software trustworthy, from standards and reviews to carefully chosen tests that find the most faults with the fewest runs.

More than debugging

In the early years of computing, software quality mostly meant removing programming errors once code was written. Today quality assurance is much broader. It includes improving the development process itself, training and certifying staff, following published standards, reviewing every document a project produces, and testing at many levels. The reason is cost and risk: a misunderstanding in the requirements, caught in a review, can be fixed by editing a sentence, while the same misunderstanding found after release may mean rewriting modules, retesting and reissuing the software. Quality must therefore be built in at every stage of the life cycle, not inspected in at the end.

Standards and quality groups

Standards bodies publish requirements for quality systems. The ISO 9000 family sets out quality management practices for organisations in any industry, and ISO and the IEC publish further standards for assessing software processes. The IEEE publishes software standards too, including IEEE 1028 on software reviews. Many large clients will only hire developers who meet such standards, so software companies set up software quality assurance (SQA) groups to oversee the quality system. In a plan-driven project the SQA group might have to approve the requirements specification before design begins, and approve the design before coding starts. Throughout, decisions and their reasons are recorded, because records that are not kept up to date mislead everyone who relies on them later.

Reviews

A review is a meeting in which people examine a product of the project, such as a requirements document, a design or a piece of code, to find problems. Reviews happen throughout development: a prototype demonstration for users, a structured walkthrough in which designers step through a design together, a formal inspection with checklists and assigned roles, or a code review in which a colleague reads each change before it is merged. Some reviews are milestones, such as the meeting at which stakeholders approve the final requirements specification. Reviews find errors early, spread knowledge across the team, and should themselves be documented.

Why testing cannot prove correctness

As "Efficiency and correctness" showed, passing tests proves only that a program works on the cases tried. Testing every possible path is out of reach for real programs. A function containing 30 independent if statements, one after another, has \(2^{30}\), over a billion, routes through it, and a loop multiplies the routes further. Testers therefore aim to choose a small set of tests with a good chance of revealing errors. A useful attitude follows from this: a test that finds a fault is a successful test, because it has given the team information they did not have. Tests are run at several levels: unit tests check single functions or classes, integration tests check modules working together, system tests check the whole product, and acceptance tests check it against the client's requirements.

Glass-box testing

Glass-box, or white-box, testing designs tests using knowledge of the code. Statement coverage is the percentage of statements executed at least once by the tests. Branch coverage asks whether every decision has been taken both ways, true and false, which is stronger. Basis path testing goes further, using graph theory to pick a set of independent paths; for a single function their number equals its number of simple decisions, such as ifs and loop tests, plus one, a measure called cyclomatic complexity. Experience also shows that faults cluster: a few modules usually contain most of them. This is often summarised as a Pareto principle, after the economist Vilfredo Pareto, and it suggests testing the most fault-prone modules hardest.

Worked example: measuring coverage

Python function def ticket_price(age, student): with six statements in its body: (s1) price = 12, (s2) if age < 13:, (s3) price = 6, (s4) elif student:, (s5) price = 9, (s6) return price. 1. Test ticket_price(30, False) runs s1, s2, s4 and s6, giving 4 of 6, about 67% statement coverage, and returns 12. 2. Add ticket_price(8, False): it runs s1, s2, s3 and s6, so together 5 of 6 statements, about 83%, and returns 6. 3. Add ticket_price(20, True): it runs s1, s2, s4, s5 and s6, completing 100% statement coverage, and returns 9. 4. Branches: age < 13 went both ways, and student went both ways, so branch coverage is also complete. 5. Cyclomatic complexity: 2 decisions plus 1 is 3, matching the three tests, one per path.

Black-box testing

Black-box testing ignores the code and tests behaviour from the user's point of view: does the program give correct, timely results? A key method is equivalence partitioning: divide the possible inputs into classes that the program should treat alike, and test a few values from each, because one value usually stands for its whole class. Boundary value analysis adds tests at the edges of each class, where off-by-one errors hide. Alpha testing is done by the developer's own staff, in house. Beta testing gives a near-final version to real users outside the company, which reveals problems in real conditions and gathers feedback before release.

Worked example: boundary values

A youth club form accepts ages from 13 to 19 inclusive. 1. Equivalence classes: below 13, invalid; 13 to 19, valid; above 19, invalid. 2. Boundary tests: 12 and 13 at the lower edge, 19 and 20 at the upper edge, plus a typical value such as 16. 3. Suppose the code says if 13 < age <= 19. Run the tests: 12 is rejected, correctly; 13 is rejected, wrongly; 19 and 20 behave correctly. 4. The test at 13 exposes the off-by-one fault, which should read 13 <= age. 5. Note that a test at 16 alone would have passed, which is why the edges are tested.

Worked example: estimating remaining faults

Before final testing, a team deliberately plants 40 artificial faults, a technique called fault seeding. Testing then finds 30 of the planted faults and 90 genuine ones. 1. Testing caught \(\frac{30}{40} = 75\%\) of the planted faults. 2. Assume genuine faults are just as easy to find, so testing also caught about 75% of them. 3. Estimated genuine faults: \(\frac{90}{0.75} = 120\). 4. Remaining genuine faults: about \(120 - 90 = 30\), plus the 10 planted ones still to be removed. 5. Caution: the estimate is only as good as the assumption in step 2, since planted faults may be easier or harder to find than real ones.

Common mistakes to avoid

First, treating quality as testing at the end, when reviews of requirements and designs catch cheaper errors. Second, thinking 100% statement coverage proves a program correct; it means every statement ran, not that every case is right. Third, testing only typical values and skipping the boundaries. Fourth, confusing glass-box tests, designed from the code, with black-box tests, designed from the specification. Fifth, calling a test that finds no faults the successful one.

Key takeaways

Quality assurance spans the whole life cycle: standards such as the ISO 9000 family and IEEE 1028, SQA groups, accurate records and reviews. Testing cannot prove correctness, because paths multiply, so tests must be chosen well, at unit, integration, system and acceptance level. Glass-box testing measures statement and branch coverage, uses basis paths, and focuses on fault-prone modules. Black-box testing uses equivalence classes and boundary values, then alpha and beta tests. Fault seeding estimates how many faults remain.

Flashcards

Flashcard

Quality assurance

All activities that keep software quality high across the life cycle

Far more than debugging.

Test your understanding

Question 1 of 5

A program accepts exam marks from 0 to 100 inclusive. Which set of tests applies boundary value analysis?

Question 2 of 5

A function has 10 statements. A test suite executes 8 of them at least once. What is its statement coverage, and what does that show?

Question 3 of 5

A team seeds 50 faults. Testing finds 40 of them and 120 genuine faults. About how many genuine faults remain?

Question 4 of 5

A tester designs tests by reading the code so that every if statement is taken both true and false. What kind of testing is this?

Question 5 of 5

Why can testing alone not prove that a large program is correct?

← Previous topicNext topic →