Performance

The big picture: measure, find, fix, measure

Fixing performance is like being a doctor. A doctor does not give medicine before checking what is wrong. In the same way, a good engineer follows a loop: measure how the system behaves, find the part that wastes the most time, change only that part, and measure again to see if it really got better.

  measure --> find the --> change one --> measure
     ^        slow part      thing          again
     |                                         |
     +-----------------------------------------+

Four layers of the same story

Across all layers you need two skills. First, you need good numbers: the right measurements, taken in a fair way (benchmarking). Second, you need to know where the time goes (profiling). The rest of this summary builds those skills step by step.

Latency, throughput and percentiles

Go back to the bakery. Two different things matter. One is how long one person waits from entering until leaving with the bread. The other is how many people the bakery serves per hour. Computers have the same two measures.

The time a client sees is the response time. In this summary we use the convention: response time = waiting time in the queue + service time (the time the server really spends working on the request). When the server is busy, the service time hardly changes. It is the waiting that grows.

Why the average lies

Imagine 10 requests: nine take 10 ms and one takes 910 ms. The mean (add everything and divide by 10) is 100 ms. But no request took 100 ms! The median (the middle value after sorting) is 10 ms, which describes the typical request much better. A few huge values drag the mean up, and latencies almost always have a few huge values.

Percentiles

A percentile answers: below which value are p% of the measurements? The p50 is the median. The p95 is the value that 95 out of 100 requests beat, so 1 request in 20 is slower than the p95. The p99 leaves 1 in 100 slower. The slow end of the distribution is called the tail, and high percentiles describe it. This is called tail latency.

To compute a percentile with the nearest-rank method: sort the values, then take the value at position ceil(p / 100 × n), counting from 1. Example with 10 sorted values:

values: 5 6 7 8 9 10 11 12 13 50     (n = 10)
p50 -> position ceil(0.50 x 10) = 5  -> 9
p90 -> position ceil(0.90 x 10) = 9  -> 13
p99 -> position ceil(0.99 x 10) = 10 -> 50

Throughput from numbers

The average throughput of a test is the total completed requests divided by the duration. In k6, http_reqs counts HTTP requests, while iterations counts complete runs of the script, and one iteration can make several requests. For example, 18,000 requests in 150 seconds is 120 req/s. Do not mix the two counts up.

In a closed model with no pause, each of U users sends one request at a time, so throughput is about U / response time. With 20 users and 100 ms responses: 20 / 0.1 = 200 req/s. If the server gets slower, the users also send less.

The tail grows when you fan out

If one call has a 1% chance of being slow (its p99), a page that makes many calls is slow much more often, because it only escapes if every call is fast. The chance of at least one slow call is 1 - 0.99^n.

Two traps

• Failures are fast. A request that fails with an error often answers in 5 ms. If errors rise from 0% to 60%, the median can drop and look like an improvement. Read latency and error rate together, and measure successful responses separately.
• Coordinated omission. A load generator that waits for the slow answer before sending the next request stops sending exactly during the slow moment, so it never records how bad it was. The percentiles come out too optimistic. The cure is an open model: requests leave at a fixed rate, whatever the answers do.

Benchmarking methodology

A benchmark is a fair race between programs. If one runner starts with a backpack and the other without, the race proves nothing. Benchmarking is the craft of making the race fair, so that a difference in the numbers really comes from the code.

Warm-up, repetitions and noise

• Warm-up. A program that just started is not in its normal state. Files are not in the system cache, the CPU caches are cold, and runtimes with a JIT compiler have not optimised yet. So a runner runs the work a few times and throws those timings away. If start-up time is what you care about, measure it separately and say so.
• Repeat. One run proves little. Another program on the same machine (a video call, an antivirus) can slow one run down. This is called a noisy neighbour. Run several times on a machine as idle as possible.
• Report the spread, not only the best run. Say the mean and the standard deviation, and ideally the minimum and maximum.

The sample standard deviation divides by n - 1 (not n), because the mean was estimated from the same runs. Example with runs 4, 6, 8 ms: mean = 6; distances to the mean are 2, 0, 2, so the squares are 4, 0, 4, which sum to 8; divide by 3 - 1 = 2 to get 4; the square root is 2 ms.

How to read a result: if A has mean 102 ms and B has mean 99 ms, but the standard deviation of both is about 6 ms, then the 3 ms difference is smaller than the noise. Repeating the test could flip the order. Report a tie, or measure more with a quieter machine.

Speed-up and Amdahl's law

Speed-up is the old time divided by the new time, for the same work. Going from 800 ms to 200 ms is a speed-up of 4. Careful: cutting the time by 50% is a speed-up of 2, and cutting it by 90% is a speed-up of 10. Say which measure you use.

Speeding up one part has a limit. If a part takes a fraction p of the time and you make that part s times faster, the whole program gets 1 / ((1 - p) + p / s) times faster. This is Amdahl's law. Example: a 10 s job has 5 s of computing. Making the computing 5 times faster gives 1 / (0.5 + 0.1) = 1.67, not 5. Even an infinitely fast computation leaves the other 5 s, so the ceiling is 2.

A fair comparison

• Change one thing at a time. To compare two runtimes, use the same workload, data size, concurrency and machine.
• Time the whole process (from outside) to know how long a user waits. It includes loading the runtime and reading input, a near-fixed cost that dominates with small inputs. Time only the section (inside the program) to know the cost of the algorithm. Both are useful if you say which one you show.
• Make every variant print a checksum of its result. If one variant is much faster but its checksum is different, it is not an optimisation, it is a bug.
• Consume the result. An optimising compiler may delete code whose result nobody uses, so a microbenchmark can print "0 ms" because it timed an empty loop. Add the results to a checksum and print it at the end.

Reproducible results

A published number is worth something only if another person can repeat it. Always record the machine (CPU, memory, system), the exact versions of compilers and runtimes, and the full command with its input and options. The tool hyperfine times terminal commands for you with warm-up runs, repetitions and statistics (mean, standard deviation, minimum, maximum), and exports JSON or Markdown. It measures the whole process, so start-up is included.

Profiling and flame graphs

You know the program is slow, but where? A profiler is a tool that watches a program and tells you where the time goes. It is like a security camera in a kitchen: afterwards you can see which step takes most of the minutes. Profile first, optimise later.

Sampling and instrumenting

Which profile for which problem

• CPU profile: where does the processor spend its time?
• Heap profile: who allocates memory, and which objects stay alive? Use it when memory grows for hours (a leak) while CPU is low.
• Wall-clock vs CPU time: wall-clock is the real time between start and end. CPU time is only the time the processor ran your program (user in your code, sys in the kernel). The difference is off-CPU time: waiting for disk, network, a database or a lock.

$ time ./script
real 5.0 s    # the clock on the wall
user 0.3 s    # CPU running your code
sys  0.1 s    # CPU running the kernel for you

Reading a flame graph

A flame graph draws the sampled call stacks. Each box is a function. A box sits on top of the function that called it, so height is how deep the stack is. The width of a box is the fraction of samples where that function was on the stack, so wide means "a lot of time". Identical stacks are merged, and boxes are sorted by name.

                [ escape  ]
  [ parse ]     [ render .............. ][ send ]
  [ handleRequest ............................ ]
  [ main ..................................... ]
   (the root is always 100% wide)

The total time of a function includes everything it calls. The self time is only its own code: total minus the sum of its direct children. If renderHtml has total 600 ms and its children take 450 + 100 ms, its self time is 50 ms. The CPU is really working where self time is high, which in a flame graph is the free edge at the top of a box. Look for the widest plateau at the top, not for the widest box (the root is always widest) and not for the tallest tower (that is just a deep stack).

Runtime performance: event loop, garbage collection, cluster

Node.js and Bun run the JavaScript of your server on one thread. A thread is like one cook. How can one cook serve thousands of customers? Because most of the time the cook is not cooking, only waiting.

The event loop

The event loop is a queue of ready tasks (callbacks) that runs them one at a time. When your code asks for a database answer, the waiting is handed to the operating system and the loop moves on to other requests. This is I/O-bound work: the time goes to waiting, not computing. One thread keeps thousands of such requests going, and the limit becomes the database or the connection pool.

I/O-bound:  [start query][ other requests run here ][answer]
                         (event loop is free)

CPU-bound:  [ computing 200 ms, nobody else runs ]
            /ping waits behind it

CPU-bound work is the opposite: the thread is busy calculating, so no other callback runs, no timer fires, no I/O result is read. If two requests of 300 ms of synchronous CPU arrive together, followed by a tiny /ping, the ping is answered only after about 600 ms. One slow endpoint ruins the latency of the whole server.

Using more cores

Cluster multiplies CPU capacity, so it helps when the CPU is the bottleneck. If one request needs 25 ms of CPU, one process serves at most 1 / 0.025 = 40 req/s. Four workers on four cores give up to 160 req/s (a bit less in real life). But for I/O-bound work (5 ms of CPU and 95 ms waiting), one process is free most of the time, so more workers change nothing. Before adding workers, compute how much of a core the process really uses: request rate × CPU time per request.

Garbage collection

Programs create objects and forget them. The garbage collector (GC) finds objects nobody uses and frees their memory. V8, the engine of Node.js, is generational, based on the observation that most objects die young. New objects go into a small area, the young generation, cleaned often by quick minor collections. Survivors move to the old generation, cleaned rarely by major collections, which are slower because the heap is much bigger.

Much GC work is incremental or runs on helper threads, but some phases are stop-the-world: JavaScript stops completely. Major collections over a big old heap can pause for a long time. Because they are rare, they hardly change the mean or the median, and they show up as periodic spikes in the p99. Do not confuse this with a leak: after a leak, the memory in use after each collection keeps growing. If memory returns to the same level after every collection, it is not a leak.

CPU cache and memory locality

Big O counts steps, as if every step cost the same. But reading from memory is not equally fast everywhere. Think of your desk, your bookshelf, and the city library. The pencil on the desk takes one second. A book on the shelf takes a minute. A book from the library takes an afternoon.

The memory hierarchy

registers < L1 < L2 < L3 < RAM < disk
smallest, fastest, most expensive  -->
                   <-- biggest, slowest, cheapest

Fast memory is small and expensive, big memory is cheap and slow. So the computer keeps the data in use right now in the small fast levels. In round numbers, L1 answers in about 1 nanosecond and main memory (RAM) in about 100, roughly a hundred times slower. A program is fast when most of its reads are served by the upper levels. A read that finds its data in the cache is a hit, and one that must go further is a miss.

Cache lines and locality

The cache does not copy single bytes. It copies blocks called cache lines, normally 64 bytes. If you read one byte that is missing, the processor loads the whole 64-byte line that contains it. Programs enjoy two habits, together called locality:

• Temporal locality: you use the same address again soon (a loop counter).
• Spatial locality: you use neighbouring addresses one after another (the elements of an array).

Count the misses of a sequential scan: it is the number of lines touched, not the number of elements. A double has 8 bytes, so a line holds 8 of them. Scanning 4,096 doubles reads 4096 × 8 / 64 = 512 lines: 1 miss for each 8 elements, and the next 7 are hits.

Row-major order

C, C++ and Rust store a matrix row after row (row-major). Element (i, j) lives at index i × n + j. Walking along a row uses neighbours. Walking down a column jumps a whole row each time.

/* A: inner loop on j, follows the memory order */
for (i = 0; i < N; i++)
    for (j = 0; j < N; j++) sum += a[i][j];

/* B: inner loop on i, jumps N values each step */
for (j = 0; j < N; j++)
    for (i = 0; i < N; i++) sum += a[i][j];

The same idea explains why summing 1 million numbers in an array beats a linked list with nodes scattered in memory. Both are O(n), but the list may pay one miss per element.

Matrix multiplication: interchange and blocking

Multiplying two n × n matrices has three nested loops and n³ multiplications, whatever the loop order. In the order i, j, k, the inner loop goes down a column of b, the worst walk. Loop interchange to i, k, j makes every inner access sequential or fixed. Same result, same Big O, very different speed.

Blocking (or tiling) goes further: do the work on small B × B tiles that fit in the cache, and reuse each loaded piece many times before moving on. Three tiles of doubles use 3 × B² × 8 bytes. For a 32 KiB (32,768 bytes) L1: B = 32 needs 24,576 bytes and fits, while B = 64 needs 98,304 and does not. The block size comes from the cache capacity, and blocking does not change the number of operations.

Database performance: indexes, N+1, connection pooling

Many slow apps are slow because of the database. A table with 10 million rows is like a library with 10 million books. Finding one book by walking along every shelf takes forever. The librarian needs a catalogue.

Indexes

An index is that catalogue. A B-tree index keeps the values of a column sorted in a balanced tree, so finding one value costs O(log n) instead of O(n). Each node holds hundreds of keys, so the tree is shallow even for huge tables. Without a useful index, PostgreSQL does a sequential scan: it reads every row and throws away the ones that do not match.

EXPLAIN shows the plan the database chose, without running the query. Each node shows an estimated cost (abstract units), the estimated rows and the row width in bytes. A Seq Scan with a Filter that returns few rows from a huge table is the classic sign of a missing index.

EXPLAIN SELECT * FROM orders WHERE customer_id = 42;
-- Seq Scan on orders (cost=0.00..23864.00 rows=95 width=64)
--   Filter: (customer_id = 42)
CREATE INDEX idx_orders_customer ON orders (customer_id);

• Indexes cost writes. Every INSERT, and every UPDATE or DELETE that touches indexed columns, must update each index, and each index uses disk and memory. Create indexes for queries that exist, and drop the unused ones.
• Selectivity. It is the fraction of rows a filter lets through. An index pays off when that fraction is small. On a column with three values where 'delivered' is 95% of rows, the planner prefers the sequential scan, because millions of random reads are worse than one straight read.
• Composite indexes and the leftmost prefix. An index on (tenant_id, status, created_at) is sorted by the first column, then the second inside each first value, like a phone book sorted by surname and then first name. It serves filters that fix a prefix starting from the left. A filter that skips the first column finds its rows scattered across the index and is the worst served.
• **SELECT *** brings every column, even the ones the app throws away. Wide columns (long text, JSON) cost disk reads, memory and network. Ask only for the columns you need.

The N+1 problem

The code fetches a list with 1 query, then makes more queries for each of the N items. Each query is fast, but each pays a round trip to the database. A list of 10 posts that loads the author of each post in a loop runs 1 + 10 = 11 queries, and 100 posts would run 101. The fix is a fixed number of queries: a JOIN, a filter WHERE id IN (...) with all the ids, or the eager loading of the ORM.

Pagination

With OFFSET 100000, the database must produce 100,000 rows and throw them away before the page, so deeper pages cost more. Keyset pagination remembers the last value seen: WHERE id > 100000 ORDER BY id LIMIT 20. The index goes straight to that point, and the cost is the same on every page. The trade-off: you can only go to the next or previous page, and the order needs indexed columns without ties.

Connection pooling

Opening a database connection costs several network round trips and, in PostgreSQL, a new process on the server. That can cost more than the query itself. A connection pool opens a limited number of connections, keeps them open and lends one to each request, which gives it back when done.

The capacity of a pool is the number of connections divided by the time each request holds one. With 20 connections and 40 ms each: 20 / 0.04 = 500 req/s. Above that rate requests pile up and latency grows without bound, even though each query still takes the same time. In a load test this appears as a knee: stable latency up to a point, then a steep rise while throughput stops growing. Shortening the time a connection is held raises capacity as much as enlarging the pool.

The limit max_connections of PostgreSQL belongs to the whole server, but every application instance has its own pool. What must fit is the sum: pool size × instances + reserve. With max_connections = 100, a reserve of 10 and 6 instances: (100 - 10) / 6 = 15, so at most 15 per instance. A huge pool is not the answer: too many active connections fight for CPU, memory and disk in the database.

Load, stress, spike and soak tests

A load test sends many fake users to a service and watches how it behaves. It is like a fire drill: you want to find the problems before the real fire. Different questions need different shapes of traffic, so there are different test types. They differ in two things: how much load, and for how long.

users
  |            spike                    stress
  |             /\                       ____
  |   load     /  \          load       /
  |  _____    /    \       _____      /
  | /     \__/      \_____/     \____/
  +----------------------------------> time
   (soak = the plateau of 'load' lasting hours)

What each one reveals

• Stress and the knee. While there is spare capacity, more users give more throughput at almost the same latency. At the saturation point some resource runs out (CPU, pool connections, threads): throughput levels off, and extra users only lengthen the queue, so latency shoots up. In a table: 100 users give 800 req/s at 30 ms, 150 users give 1200 req/s at 32 ms, but 200 users give only 1250 req/s at 190 ms. The elbow is at about 150 users.
• Spike. Watch three moments: before, during and after. Surviving the burst is not enough. If p95 stays ten times above normal for 15 minutes after the load drops, there are unbounded queues, long timeouts, retry storms, or pools slow to release. A shop opening a sale at an exact time needs a spike test.
• Soak. Memory at first grows and then stays flat? That is warm-up (caches filling), not a leak. A leak is growth that continues under constant load. That is why a soak test lasts hours: in a few minutes both curves look the same.
• Breakpoint. Unlike stress, it has no planned plateau. An arrival-rate executor is usually used, so the pressure keeps rising even when responses get slow.

Open and closed models, again

In a closed model the number of users is fixed, and each waits for its answer. When the server slows down, the generator slows down too, which relieves the server at the worst moment. In an open model arrivals follow a fixed rate, like real people who do not know the site is slow. So for stress and spike tests, an arrival-rate executor is the better choice.

k6: virtual users, stages, thresholds, checks

k6 is the tool of this area. A k6 script is a JavaScript file that says what one fake user does and how many fake users there are. Think of a puppet theatre: you write the play once, and k6 runs many puppets at the same time.

Virtual users and iterations

A virtual user (VU) is an independent executor that runs the default function of your script in a loop. Each complete pass through the function is one iteration. One iteration can contain several requests, checks and pauses, so k6 counts iterations and requests in different metrics.

import http from 'k6/http';
import { sleep } from 'k6';

export const options = { vus: 10, duration: '60s' };

export default function () {
  http.get('http://localhost:3000/products');
  sleep(1); // a pause, in seconds, like a person thinking
}

An iteration lasts as long as its requests plus its sleep. Here, if the request takes 500 ms, one iteration is 1.5 s, so the number of iterations is VUs × duration / iteration time, about 10 × 60 / 1.5 = 400.

Stages and executors

The stages option draws a ramp: each stage has a duration and a target, and k6 moves the number of VUs in a straight line from where it is to the target. A stage that repeats the previous target is a plateau, and a last stage with target 0 is the ramp-down.

export const options = {
  stages: [
    { duration: '1m', target: 20 }, // ramp-up
    { duration: '3m', target: 20 }, // plateau
    { duration: '1m', target: 0 },  // ramp-down
  ],
};

Example: 50 constant-vus with no sleep. At 100 ms per request they send 50 / 0.1 = 500 req/s. If the server slows to 500 ms, they send only 100 req/s: the test eases the pressure exactly when the system suffers. A constant-arrival-rate of 30 per second with a response of 1 s needs about 30 VUs. If it only has 20, k6 drops the rest.

Thresholds and checks

A threshold is a pass or fail rule on a metric: an aggregation (avg, p(95), rate, count), an operator and a limit. http_req_duration: ['p(95)<500'] means that the p95 of all request durations must be under 500 ms. When any threshold is crossed, k6 ends with exit code 99, so a CI pipeline sees a failure. By default k6 keeps running. The long form { threshold: 'rate<0.05', abortOnFail: true, delayAbortEval: '30s' } stops the test early, and delayAbortEval waits first, so a few early failures cannot abort the test before enough samples exist.

A check verifies something about a response, like "body has items". It does not stop the test or fail it: it only records the pass rate in the checks metric. To make wrong content fail the test, add a threshold on checks, for example checks: ['rate>0.99'].

The four phases and metric types

• Counter: adds values up (how much).
• Gauge: keeps the last value (how much right now).
• Rate: the share of true values (how often). http_req_failed and checks are Rates.
• Trend: statistics such as mean and percentiles (what distribution). Use it for the time of a login flow.

Other useful pieces: __ENV holds environment variables (k6 run -e NAME=value), so const BASE_URL = __ENV.BASE_URL || 'http://localhost:3000' makes the safe, local case the default. And handleSummary(data) receives all metrics at the end and returns an object whose keys are destinations (a file, stdout) and whose values are content, which is how JSON or HTML reports are made. It replaces the default summary, so include a stdout key if you want both.

Capacity planning and queueing

Capacity planning answers: how many servers do we need? To answer, we use a few simple laws from queueing theory, which is the maths of waiting in line. Picture a single ticket window at a cinema.

Two laws that always hold

The utilisation law: the fraction of time a server is busy is arrival rate × mean service time (same time units). A window that gets 10 customers per second and takes 50 ms each is busy 10 × 0.05 = 0.5, so 50% of the time. It is the first calculation of capacity planning: how much is already committed.

Little's law: in a stable system, L = λ × W, where L is the average number of items inside, λ (lambda) the arrival rate and W the average time each item spends inside (waiting plus service). It does not depend on how arrivals are distributed or how many servers there are. If 100 requests arrive per second and each spends 0.05 s in the system, on average 100 × 0.05 = 5 requests are inside. It converts between concurrency, throughput and latency.

Why latency explodes near 100%

Arrivals are never evenly spaced. When two customers arrive together, one must wait. The queue is emptied during idle time. At low utilisation, idle time is plentiful and a burst is absorbed fast. At high utilisation, the queue from one burst is not empty when the next one comes. For a simple model, the M/M/1 queue (one server, random Poisson arrivals, exponential service times), the mean response time is:

R = S / (1 - U)

S = mean service time, U = utilisation (between 0 and 1)
example: S = 20 ms, 40 req/s -> U = 0.8 -> R = 20 / 0.2 = 100 ms

Look at the shape: nearly flat at first, then very steep. Going from 50% to 90% utilisation (traffic not even twice as high) multiplies latency by 5. The formula depends on the model, but the shape is a good intuition for real systems. That is why we keep headroom: spare capacity.

Sizing from a load test

1. Measure the capacity of one instance within the latency target, for example 250 req/s.
2. Apply the headroom rule: if each instance may run at most 70%, usable capacity is 175 req/s.
3. Divide the expected peak by it and round up. For a peak of 1,000 req/s: 1000 / 175 = 5.7, so 6 instances.

This assumes load spreads evenly and scaling is linear. Check that: a shared database may saturate first.

The same method sizes a connection pool. The average number of busy connections is arrival rate × holding time. With 100 req/s holding a connection 30 ms: 3 busy on average. To keep at most 60% busy, divide by 0.6: 5 connections. A pool sized exactly for the average would be 100% busy, and the queue would grow long long before that.

Amdahl's law for scaling out

If a fraction s of the work cannot be parallelised, the speed-up with N processors is 1 / (s + (1 - s) / N). With 10% serial, it never exceeds 10, however many processors you add.

In a cluster of processes, the serial part is usually a shared resource: the database, or a lock.

Putting it all together

Now you have a toolbox. Here is how the pieces connect when a service is slow, step by step.

1. Define fast: pick a goal with a percentile, for example "95% of requests under 300 ms".
2. Produce load with a test of the right type (load, stress, spike, soak) in an open model, against a local target.
3. Read the result with percentiles, throughput and error rate together, and look for the knee.
4. Find the bottleneck: CPU high means a CPU profile and a flame graph. CPU low and latency high means waiting: database, pool, network.
5. Fix one thing, measure fairly again (warm-up, repetitions, same machine, checksum) and report the spread.
6. Size the service with utilisation, Little's law and headroom.

Common mistakes

• Optimising without profiling.
• Looking only at the mean, or averaging percentiles of different servers.
• Comparing two programs that do different work, or never checking the checksum.
• Timing code the compiler removed, or including start-up by accident.
• A closed-model test that hides the problem it should find (coordinated omission).
• Aiming a load test at a system you do not own.
• Adding workers or a bigger pool without checking where the bottleneck is.