Testing

The big picture: what is a test for?

A test has three parts. First you prepare a situation. Then you do the thing you want to check. Finally you compare the result with what you expected. If the result is different, the test fails and a human is told.

Words you will meet

• Defect (bug): a mistake in the code that makes the program behave wrongly.
• Test suite: a group of tests that run together, like a whole exam made of many questions.
• Pass and fail: a test passes when the result is what we expected, and fails when it is not. Many tools show passing as green and failing as red.
• Verification: checking that the software does what its written description (the specification) says. The question is: did we build the thing right?
• Validation: checking that the software is what the customer really needs. The question is: did we build the right thing? A program can pass the first check and fail the second, for example when the written description itself was wrong.
• Functional test: checks what the system does, for example 'for this input, this output'.
• Non-functional test: checks how the system behaves while doing it, for example how fast it is, how many users it can serve, how safe or easy to use it is.

Test levels and the test pyramid

Think about checking a toy car. You can look at just one wheel. You can look at the wheel and the axle together. Or you can put the whole car on the floor and push it. Each check looks at a bigger piece. Tests are the same, and we call each size a test level.

The test pyramid is advice about how many tests of each level to write: many unit tests at the wide base, fewer integration tests in the middle, and very few end-to-end tests at the top. The reason is cost. The higher the level, the slower, the more fragile and the more expensive each test is.

      the pyramid                 the ice-cream cone
         /\                      ___________________
        /e2e\   few              \   many e2e + manual /
       /------\                   \   few integration/
      / integ. \  some             \   very few unit /
     /----------\                   \______________/
    /   unit     \  many
   /--------------\         slow, shaky, hard to debug

The upside-down version is a known mistake, called the ice-cream cone or inverted pyramid. Almost all checking is done by slow end-to-end and manual tests. The suite takes a long time, fails for reasons that have nothing to do with the code, and does not say where the bug is. The cure is to push each check down to the lowest level that can see the behaviour.

The arithmetic of the pyramid

Suppose a suite has 400 unit tests of 5 ms, 50 integration tests of 200 ms and 10 end-to-end tests of 6 seconds, run one after the other. The time of each level is the number of tests times the duration of one test.

The whole suite takes 72 seconds, and the 10 slow tests (a small share of all tests) use 60 of them, about 83% of the time. This is why we do not want a wide top.

Which level sees which bug?

A test only finds bugs that are on the path it runs. A mistake in a price rule written as a plain function is seen by a unit test. A wrong column name between the code and the database is seen only when code and database run together. A button whose click is connected to the wrong element of the page is seen only by a test that opens the page, because the server code is perfectly fine.

The three stages of Sommerville

• Development testing: made by the people who build the system, to look for defects (unit, component and system tests).
• Release testing: a separate team tests a complete version to show that it meets the requirements and is good enough to be used.
• User testing: real users or customers try the system. In alpha testing they test together with the team. In beta testing an early version goes to volunteers who use it in their real environment. In acceptance testing the customer decides whether to accept the system.

Unit tests and isolation

A unit test checks one small piece of the program, such as one function, alone. 'Alone' is the key word. If you want to know whether a single light bulb works, you test it by itself, not inside the whole house.

A good unit test is fast (milliseconds), isolated (it does not depend on other tests), repeatable (same result every time) and self-checking (the test itself says pass or fail with assertions, so no person has to read the output and compare). A self-checking suite can run thousands of tests, and you only look at the ones that failed.

Arrange, Act, Assert

test("a 10% coupon lowers 200 to 180", () => {
	// Arrange: prepare the situation
	const cart = new Cart([{ price: 200 }]);
	// Act: do the one thing being tested
	const total = cart.totalWith("TEN");
	// Assert: compare with what we expect
	expect(total).toBe(180);
});

Keep one Act per test. Then, when the test fails, you know exactly which behaviour is wrong.

The fixture and the xUnit family

The fixture is the stage where the test plays. In the xUnit family of tools (JUnit, pytest, Bun test and many more), setUp builds the fixture before each test and tearDown cleans it after each test. Because every test starts from a fresh stage and leaves nothing behind, the tests can run in any order, or one alone.

A test runner must also treat a failed test as a normal result. It catches the error of each test, writes it down, and goes on with the next test, so a single run shows all the failures at once. A runner that stops at the first error would hide the others.

Shared state breaks isolation

const cart: string[] = [];          // shared by every test

test("adds an item", () => {
	cart.push("book");
	expect(cart.length).toBe(1);
});
test("adds a second item", () => {
	cart.push("pen");
	expect(cart.length).toBe(2);   // passes only if the first test ran before
});

Run the whole file and it is green. Run only the second test in a new process and it fails, because the list starts empty, so after one push it has 1 item and not 2. The classic sign of shared state is a test that passes in the full suite but fails alone or in another order.

Things that make a unit easy to test

• Pure function: for the same arguments it always returns the same value and changes nothing outside itself. It needs no clock, no random numbers, no files and no network, so you only give arguments and check the returned value. Put the logic in pure functions and keep the side effects in a thin layer around them.
• [[Dependency injection|giving a piece of code the things it needs from the outside, instead of letting it create them]]: if registerUser does new SmtpMailer() inside, a test cannot stop it from sending real e-mail. If it receives a mailer as a parameter or in its constructor, the test hands it a pretend mailer.
• Test through the public interface: check what the class promises (returned values, visible state). A test that reads a private field breaks when the code is rewritten inside, even though nothing changed for the users.
• Objects with state: the effect of a method depends on the methods called before it. Testing a Connection class with one test per method is not enough: also try the sequences, including the invalid ones, such as send after close. All lines can be covered and the dangerous sequence still never run.

Test doubles: pretend helpers

A film uses a stunt double for the dangerous scenes. In software, a test double is an object that takes the place of a real helper (a database, an e-mail server, a clock) during a test. We do this because the real helper may be slow, may cost money, or may be impossible to control.

The words below come from Gerard Meszaros and are used by Martin Fowler. There are five kinds.

State verification and behaviour verification

• State verification: the test uses the unit and then looks at the result, either the returned value or the final state of the objects.
• Behaviour verification: the test looks at the calls made to the helpers. It is needed when the effect is the call itself, such as sending an e-mail.
• The second style ties the test to how the code is written. A test that only counts calls can pass when the answer is wrong, and fail after a harmless refactoring. When there is a returned value or a final state, the main assertion should be about it.

A fake often gives shorter and stronger tests than a mock when the helper has memory. If a sign-up service saves a user and later finds and updates it, a fake repository lets the test just look at the stored user at the end. A mock would need one expectation for each call, which describes the conversation, not the result.

Uses worth knowing

• Hard-to-cause situations: a full disk, a service that answers 503, a timeout. A double set to fail causes the error whenever you ask, so you can check how the code reacts. The double replaces the helper (the HTTP client), never the unit being tested.
• Time as a dependency: code that reads the system clock has a hidden input. Inject a FakeClock that starts at a fixed moment and moves only when the test calls advance. Then 'the token expires after 15 minutes' is tested in milliseconds, exactly at the edge, and never fails by luck.
• Self Shunt: Kent Beck's name for the case where the test class itself plays the role of the double.

Test-driven development: red, green, refactor

Most people write the code first and the test afterwards. Test-driven development (TDD), created by Kent Beck, turns this around: you write the test first. It sounds odd, but it is like drawing the finish line before running the race.

    +--------+   make it pass, quickly   +--------+
    |  RED   | ------------------------> | GREEN  |
    | a new  |                           | all    |
    | test   |                           | tests  |
    | fails  | <------------------------ | pass   |
    +--------+      next small test      +---+----+
         ^                                   |
         |      clean up, same behaviour     v
         +------------------------------ REFACTOR

1. Red: write one small test that fails.
2. Green: make it pass in the quickest way you can.
3. Refactor: improve the structure and remove duplication without changing behaviour, with every test still passing.

TDD has two rules: write new code only when an automated test has failed, and remove duplication. If you have an idea in the middle of the work (for example 'this method should reject negative numbers'), it first becomes a test, or a line on your to-do list. The to-do list is the short-term memory of TDD: you write the cases you know, add the ones that appear on the way, and cross them out when they pass.

Why watch the test fail first?

The red step tests the test itself. If a new test already passes before the code exists, or fails for a silly reason such as a typo or a wrong import, it is not checking what you meant. Only after you see it fail for the right reason does the green bar count as proof.

Three ways to get to green

The steps can be big or small. When things go well, use large steps with obvious implementation. If an unexpected red appears, take smaller steps (fake it, triangulation), because a small step limits the place of the error to a few lines.

Reading the history of a session

Each step leaves a signature on the suite. Red: a new test is added and the suite goes from green to red. Green: production code changes and the suite goes from red to green. Refactor: production code changes, the suite is green before and after, and no new test appears. Never mix a refactoring with a change of behaviour in the same step: the unchanged tests are the proof that behaviour was kept, and if you also edit the expected values, you lose that proof.

The money example

In Beck's book, Dollar and Franc are merged into one Money class. Money is a value object: it is defined by its data, not by its identity, so two objects with the same values are equal. When the classes are merged, the currency becomes a field, and equals must compare it too.

equals(other: Money): boolean {
	return this.amount === other.amount;   // forgets the currency
}
// 5 USD equals 5 CHF? A test asking this fails, and forces the fix:
// return this.amount === other.amount && this.currency === other.currency;

Benefits and limits

Benefits claimed for TDD: coverage as a by-product, a regression suite that grows with the program, easier debugging (the bug is in the few lines just written), and tests that work as executable documentation. The limit: TDD gives mostly unit tests written by the developer, who checks the code against what the developer understood. Mismatched formats between services, slowness under load, and the question 'does the user need this?' need other kinds of testing.

Integration tests: pieces working together

Two puzzle pieces can each be perfect and still not fit together. An integration test runs two or more modules together, through their real connection. Its target is the bugs that live only at the border: swapped parameters, different formats, wrong beliefs about the other side, wrong settings. Unit tests replace the neighbours with doubles, so they never cross that border.

Interface errors

Sommerville sorts the errors at an interface into three classes.

How to join the pieces

• Big-bang: put all the modules together at once and test the whole at the end. A failure can come from any interface, so it is very hard to find.
• Incremental: add one module at a time and re-run the tests, so a new failure points at what just came in.

While some pieces do not exist yet, we write support code. A driver calls the module under test in the place of whatever would be above it. A stub answers in the place of whatever would be below it. Bottom-up integration starts with the lower layers real and uses drivers. Top-down starts with the upper layers real and uses stubs. Imagine three layers, UI on top of Service on top of Repository. If the team builds from the bottom and the UI does not exist yet, it writes a driver that calls the Service.

  [ driver ]  -->  [ Service ]  -->  [ Repository ]
   test code          real              real
   (stands in for the UI that is not written yet)

Real database or fake?

An in-memory fake is fast, but it only knows what we programmed into it. A database has its own rules: unique keys, foreign keys, types, transactions, the real SQL. A repository that passes with a Map can break in production with PostgreSQL when two users have the same e-mail. So a healthy suite has some integration tests against the real database, or at least the same engine.

A real database is shared, so each test must start from the same point. A common trick: open a transaction before each test and roll it back afterwards. Everything the test wrote disappears, without the cost of rebuilding the database, and tests do not depend on the order.

beforeEach(() => db.run("BEGIN"));
afterEach(() => db.run("ROLLBACK"));  // forgets what the test wrote

// test 1 inserts an order and sees 1 row
// test 2 runs later and sees 0 rows again

An integration test may also send a request to the HTTP handler inside the same process and then read the row in a real SQLite file. It checks the whole seam: the body is read, converted, passed on, turned into SQL and accepted by the table. Each layer can pass its unit tests and the whole can still fail because of one different column name.

Contract tests

Suppose the orders service is tested against a stub of the payments service that answers { "total": 30 }, and the payments team renames the field to amount. Both teams' tests pass and production breaks. A contract test fixes this: what the consumer expects becomes a file, and that file is run against the real provider in the provider's own pipeline. You get the interface check without starting every service together.

End-to-end tests with Playwright

An end-to-end test (e2e) uses the system the way a person does: through the screen, with every real layer connected. It gives the most confidence that everything works together. In return it is the slowest, the most expensive to keep and the least exact about where the bug is. So it sits at the top of the pyramid, in small numbers.

Playwright is a tool that controls a real browser from a test: it opens pages, clicks buttons, types text and reads what is on the screen.

Auto-waiting

Web pages are not instant. A button may be disabled while data loads. Playwright waits by itself until an element is ready before it acts, so the test needs no fixed pauses such as waitForTimeout(2000) (a pause is either too long and wastes time, or too short on a slow computer). If the element never gets ready, the action fails with a timeout error.

Locators: find things as the user sees them

Web-first assertions and the missing await

A web-first assertion such as await expect(locator).toBeVisible() receives the locator and repeats the check until it passes or the time runs out. Compare it with expect(await locator.isVisible()).toBe(true): it gets a finished true or false, a photo of one instant. If the screen has not updated yet, the photo says false and the test fails by luck.

await page.getByRole("button", { name: "Add to cart" }).click();
// WRONG: no await, the promise is dangling and the test ends before the check
expect(page.getByTestId("cart-count")).toHaveText("1");
// RIGHT:
await expect(page.getByTestId("cart-count")).toHaveText("1");

Playwright assertions are asynchronous and return a promise. Without await, the test runs to the end without waiting for the result, so it can be green even when the counter shows 0. A test that cannot fail is worse than no test. A lint rule for floating promises catches this.

Isolation by browser context

Each test gets its own browser context, which works like a new, empty profile: its own cookies, localStorage and cache. Creating a context is cheap. So if the first test logs in, the second test, which uses the page fixture, opens the site as a stranger who is not logged in. Tests do not depend on each other or on the order.

Controlling the network

page.route catches the requests made by the page and lets the test decide: answer with made-up data, abort, or let it go. Replacing a third-party service with a fixed answer makes the test steady and independent of the internet. What is no longer checked is the real integration with that service, which needs another kind of test.

Smoke and regression tests

When you plug in a new machine, you first check that no smoke comes out. You do not test every function yet. A smoke test is the same: a small set of fast, shallow checks of the critical paths. Does the application start? Does it answer? Do the essential functions work? It runs right after a build or a deploy and works like a gatekeeper: if it fails, the version is rejected at once, and nobody wastes time on the long suites.

A regression is a loss of behaviour that used to be correct, caused by a change somewhere in the code. Regression testing means running the existing tests again after each change to see that nothing that worked has stopped working. With automated tests it is cheap and can run on every commit.

Suites are ordered from the cheapest to the most expensive so that failing happens as early as possible. The sanity test is a word that teams use in different ways, but it usually means a narrow check of the area that has just been changed.

What goes in the smoke suite is chosen by criticality, not depth: the paths without which the system is useless (open the site, sign in, the main operation). Rule details, edge cases and old bugs belong to the unit and regression suites. If the smoke suite grows too big, it loses the speed that justifies it.

Every bug becomes a test

A user reports that an order with a 100% coupon shows a negative total. What should the developer do first? Write a test that reproduces the bug and fails. Only then fix the code, and see the test turn green. The first red proves the test really sees the problem, and the test stays forever to protect against the return of that bug.

When the regression suite is too slow

If the full suite takes 3 hours, a team may run on each commit only the tests of the files that import the changed modules. This gives fast feedback, but it is a bet: indirect dependencies can let a change break something that looks far away. The usual protection is to keep running the full suite at regular times and before each release.

Coverage and mutation testing

How do you know if your tests are good? A first idea is to count how much of the code they run. This is called coverage.

export function shippingFee(total: number, isMember: boolean): number {
	let fee = 10;
	if (isMember) fee = 5;
	if (total >= 100) fee = 0;
	return fee;
}

test("big member order is free", () => {
	expect(shippingFee(150, true)).toBe(0);
});

This one test goes through every line, so line coverage is 100%. But both if conditions are only ever true. Each if has two branches, so there are 4 branches and the test took 2 of them: branch coverage is 50%. 100% of lines does not mean 100% of branches.

Mutation testing: testing the tests

Mutation testing asks the better question: would the tests notice if this line were wrong? A tool makes copies of the program, each with one small planted defect, called a mutant. The tool changes things like + into -, < into <=, one constant into another, or removes a statement. Then it runs the suite on each mutant.

original program --- change one small thing ---> mutant
                                                   |
                                          run the test suite
                                                   |
                  +--------------------------------+---------------+
                  |                                                |
       some test fails: KILLED                 all tests pass: SURVIVED
       the suite noticed the bug               the suite accepts the bug

A mutant that survived is a question the tests did not ask: it shows where an assertion or a test is missing.

The mutation score

The mutation score is the killed mutants divided by the mutants that could be killed. An equivalent mutant has different text but behaves exactly like the original for every input, so no test can ever kill it. Under the usual convention, equivalent mutants leave the bottom of the fraction. Example: a tool makes 30 mutants and the suite kills 24. Of the 6 survivors, a person decides that 2 are equivalent. The score is 24 / (30 - 2) = 24 / 28, about 85.7%. Many tools cannot spot equivalent mutants and just report killed over total, so always say which convention is used. Deciding if a mutant is equivalent is impossible in the general case, so a person usually reviews the survivors.

if (weightKg > 2) {                    // mutant: weightKg >= 2
	cents += (weightKg - 2) * 150;     // at exactly 2 kg this adds (2 - 2) * 150 = 0
}

Killing a boundary mutant

Take return age >= 18 and the mutant return age > 18. They differ only for the input 18. A test kills the mutant only if it uses that exact value and checks the result: isAdult(18) must be true. Tests with ages 10 and 30 let it live. This is the practical reason to test the boundary values.

The cost of mutation testing grows with the number of mutants times the time of the suite. Tools make it smaller by running for each mutant only the tests that cover the changed line, stopping at the first test that fails, mutating only the code changed since the last run, and sampling the mutants.

Property-based testing

An ordinary test is an example: 'sort [3,1,2] and get [1,2,3]'. A property-based test states a general rule, like 'sorting never changes the length of the list', and a generator invents hundreds of random inputs to try to break the rule. It finds cases you did not think of. But it only checks the inputs that were generated, so it is not a proof.

Shapes of properties

Shrinking

A random input that fails is often big and full of useless detail, like a list of 37 numbers. Shrinking searches by itself for a minimal failing input, for example [0, -1]. A small example makes the cause of the bug easy to see.

The seed

The randomness comes from a pseudo-random generator, whose whole sequence is decided by one number, the seed. When a property test fails in CI, the tool prints something like seed: 48213. Run the test again with the same seed and you get exactly the same inputs. Without this, a random test would be a flaky test.

A property that proves nothing

function average(a: number, b: number) { return Math.floor((a + b) / 2); }

for (let i = 0; i < 1000; i++) {
	const a = randomInt(-100, 100), b = randomInt(-100, 100);
	expect(average(a, b)).toBe(Math.floor((a + b) / 2));  // the same formula
}

The expected side repeats the formula of the code, so the test passes by construction and detects nothing. A good property states a relation required by the specification and independent of the code, such as 'the average lies between the smaller and the larger of the two numbers'.

Should properties replace examples? No. Properties explore many inputs, but good ones are hard to find and are general rules, weaker than an exact value. Examples document the behaviour with real numbers and guarantee that known edge cases always run. They work well together.

Flaky tests

A flaky test passes and fails on the very same code. Nothing changed between the green run and the red one. This happens when the result depends on something the test does not control: the clock, the order of the tests, shared state, the network, concurrency or random numbers. The typical sign is a failure that disappears when you run it again.

That is the real harm. The value of a suite is trust: red means a bug, green means it is safe to go on. Flaky tests break this in both directions, raise false alarms and teach the team to just re-run instead of investigating. After that a real bug can slip by.

Small chances add up

If each of n independent tests fails by luck with probability p, the suite passes completely with probability (1 - p) to the power of n. So it shows at least one failure with probability 1 - (1 - p)^n.

With 50 tests at 2% each, 0.98^50 is about 0.364, so 1 - 0.364 is about 0.636. The suite is red almost two runs out of three. The same happens to a suite of 300 end-to-end tests that fail by luck 0.5% of the time each: 0.995^300 is about 0.22, so around 78% of the runs show a failure with correct code.

The usual causes and their fixes

Shuffling the order of the tests and printing the seed is a way to find and repeat an order dependence, not to cure it. If B only passes when it runs after A, the cure is isolation: B creates its own data and cleans up after itself, so it passes alone and in any position.

What is not a fix

• Retry until green: it hides the symptom and keeps the cause.
• A longer sleep: it makes the race less likely and the suite slower.
• Deleting or skipping the test without understanding it: sometimes the product has a real race.
• Fixing the order of the tests so shared state keeps working: the dependence is still there and blocks running in parallel.

Test design: equivalence classes and boundaries

You cannot test every possible input. A number field alone has billions. Test design is the art of choosing a few inputs that tell you a lot.

Equivalence partitioning

Split the possible inputs into groups, called equivalence classes, where the program should treat every member in the same way. Then pick at least one representative of each class. The classes come from the specification, not from the code, and they include classes of invalid inputs.

Counting classes: one class for each range that the specification handles differently. For a numeric input with separate bands, that is one class per valid band, plus one invalid class below the smallest valid value and one above the largest. If a price table has 3 bands for weights, plus errors for too light and too heavy, there are 3 + 2 = 5 classes and at least 5 test cases.

Boundary value analysis

Programmers make mistakes right at the edges: < instead of <=, a loop that stops one step early. A value from the middle of a class goes through the faulty code and never notices. So besides the typical value, test the edge and its neighbour on the other side.

  rejected |      accepted range       | rejected
  ... -1  0 | 1  2  3 ... 18 19  20 | 21  22 ...
           ^  ^                  ^   ^
          a-1 a                  b  b+1
  test values: 0, 1, 20, 21

The convention in this quiz: for a closed range of whole numbers [a, b], the two-value method tests a-1, a, b and b+1. Imagine the code says grade >= 0 && grade < 10 when the specification says grades 0 to 10 are valid. Only the value 10 gives a different answer from the specification (false instead of true). Every other test value stays on the right side, so only the value on the boundary shows the bug.

Several inputs: one invalid at a time

When a function has several inputs, valid classes can be combined in one test. But each invalid class must be alone, with the other inputs valid. A program usually stops at the first check that fails. If a test gives an invalid age and an invalid income together, the income check is never reached, and you never learn that it exists and works.

Guidelines for sequences

Experience says where programs tend to break. For lists and other sequences: test the empty sequence, a sequence with one element, and different sizes in different tests. Also make the interesting element appear first, in the middle and last. For a function that returns the largest number, good inputs include [], [7], [9, 2, 3], [2, 9, 3] and [2, 3, 9].

Decision tables

When the result depends on several yes/no conditions, a decision table lists the combinations and the expected action for each one. With n independent yes/no conditions, the full table has 2^n columns, and each column becomes at least one test. If one condition decides alone, the columns that differ only in the other conditions can be merged, writing 'don't care' for them.

Tests in CI and test strategy

Continuous integration (CI) means that everybody shares their work with the main line often, and each share is checked by an automatic build and tests. The tests run after every push, so the feedback arrives fast and points at the change that was just made. When the build breaks, fixing it comes before new work, because everyone depends on the main line.

Order the stages by cost

A pipeline arranges its checks from the cheapest to the most expensive. With fail-fast, the run stops at the first stage that fails, so a mistake that a 50 ms check finds does not wait behind a stage of minutes.

  push --> [ lint + types + unit ] --> [ integration ] --> [ e2e ] --> merge
               seconds                    tens of seconds    minutes
               |                          |                  |
               fail: stop here            fail: stop here    fail: stop

A flaky test blocks everyone: quarantine it

A pipeline is useful only if red means a defect. Suppose a test fails in 1 of 10 runs and blocks merges, and the team cannot fix it today. Bad ways out hide the problem (silent retries, deleting the test, weakening its assertions) or keep the noise (re-running by hand). The good way is an explicit quarantine: the test leaves the blocking path, the debt is written down with an owner and a date, and it still runs somewhere so it is not forgotten.

Risk-based strategy

You cannot test everything, so spread the effort by risk: probability that a defect is there times its impact. Probability is higher in new or heavily changed code, complex logic and parts with a history of failures. Impact is bigger when it involves money, safety or many users. Coverage is only one input.

The price and tax module has the highest risk: it changed a lot and a mistake costs real money. It should get the effort even though its coverage is already the highest, because high coverage does not make a module safe.

Putting it all together

Each technique answers a different question. A healthy project uses several of them, because each one is blind to something that another sees.

Common mistakes

• Putting every check in end-to-end tests (the ice-cream cone).
• Counting calls to doubles instead of checking the result.
• Trusting a coverage number without asking if the tests check anything.
• Fixing a bug without writing the test that reproduces it.
• Letting tests share state or depend on the clock, the order or the network.
• Refactoring and changing behaviour in the same step.
• Forgetting await in an asynchronous assertion.