Observability

The big picture: telemetry and vocabulary

Think of a doctor. The doctor cannot open you up to see inside, so she uses a thermometer, a heartbeat monitor and your story of how you feel. Software is the same: we cannot look inside a running program, so it sends out signals. All these signals together are called telemetry.

your code --> instrumentation --> telemetry
                                       |
                                       v
                         store (backend) --> screens
                                              and alerts
                                       |
                                       v
                                 a person understands

Monitoring against observability

Imagine a school with an alarm on each door. The alarm rings if a door opens at night. That is monitoring: you decide before what could go wrong and you put a watcher there. But what if something odd happens that nobody imagined, like a window opening because of the wind? No alarm was made for it.

The word observability comes from control theory, a branch of engineering. There, a system is observable if you can work out what is happening inside it by looking only at what it gives out. For software it means: you can understand and explain any state the system gets into, even one you have never seen, using the telemetry it already emits. If you must ship new code just to investigate, observability was missing.

Known-unknowns and unknown-unknowns

Cardinality and dimensionality

Cardinality is about one field. The field http_method has few values (GET, POST, and a few more): low cardinality. The field user_id in a service with millions of users has millions of values: high cardinality. The high-cardinality fields are the most useful for debugging, because they point to one single case among millions.

Dimensionality is about how many fields there are. A wide event is a record with hundreds of fields (customer, version, region, route, duration, and so on). Observability needs both: high cardinality and high dimensionality.

Why aggregates hide problems

An aggregate answers 'how is the system in general?'. If one customer, on one app version, fails on every request, the global error rate may still be 0.1% and the dashboard looks fine. The problem hides inside the average. To find it you must keep the data with its fields and be able to slice by any combination of them.

Pre-aggregation has the same problem in time. If you store only a counter of requests per endpoint and status, you decided before the incident which questions are allowed. A field that did not become a label at write time cannot be recovered later. Keeping the raw event, with all its fields, and aggregating when you ask, lets you invent the question during the investigation.

The core analysis loop

1. Start from the overall view of what looks wrong (for example, slow requests).
2. Check what you know so far.
3. Look for the fields whose values differ between the bad group and the rest.
4. Filter by those fields, so now you look at a smaller group.
5. Repeat until the cause is clear.

This loop is driven by data, not by guesses from experience. It does not need you to know the system beforehand, so it works even for brand-new failures.

Why an event store can take a customer id

A time series database stores data by series. Every new label value creates a new series, so the index grows with cardinality. An event store (usually columnar, which means it keeps each field in its own column) writes one row per event. The cardinality of a field does not change the cost of writing, and the grouping happens when you ask.

The three signals: logs, metrics and traces

Telemetry comes in three main kinds, called signals. Think of a car trip. The dashboard of the car shows speed and fuel: numbers over time. The travel diary says 'at 3 pm we stopped for lunch': things that happened. The map with your route shows each step of one trip. They answer different questions.

Metrics

A metric is a number over time: how many requests, how much memory, how long a request took. Each series is identified by a name and a set of labels. Because metrics keep aggregates and not each event, they are small and fast, which makes them ideal for trends and alerts.

The cost of a metric depends on the number of series and on how often it is read (the scrape frequency), not on traffic. If traffic grows ten times, a request counter costs almost the same. Logs and traces produce data for every request, so their cost follows traffic volume.

Traces

A trace tells the story of one request. Each operation becomes a span, with a start, a duration and a pointer to its parent span. All spans of the same request share a trace id. A trace shows the order of calls and where the time went, which no aggregate shows.

The chart uses the numbers of a trace that took 900 ms. A parent span includes the time of its children, so the root is always the longest. Self time is the duration of a span minus the duration of its direct children. The span with the biggest self time, usually a leaf of the tree, is where the time was really spent. Here it is stock.read, with 720 ms.

Logs

A log is a record of something that happened. Logs say 'why' and 'with which message'. They are the most detailed signal, and also the most expensive to keep, since there is a line for each event.

Making the signals work together

Signals become much stronger when they share identifiers. If the logger writes the trace id and the span id of the active context on each line, the log is no longer loose text: it belongs to one request, and tools can open the logs from a span and the trace from a log line.

A metric says 'something is slow', but not which request. An exemplar solves it: the metrics system stores the trace id of a request next to a sample. On the chart, you click the point and open the trace.

What a histogram can and cannot say

A cumulative histogram counts how many requests took up to 0.5 s, up to 2.5 s, and so on. From it you can tell how many and what share, which is enough for panels and alerts. You cannot tell which requests, where inside the request or with what message. For those you need traces and logs.

Sampling and estimating totals

Keeping every trace is expensive, so many systems keep only a fraction: this is sampling. Suppose we keep 1 trace in 100 and a query finds 480 stored traces of /pay. Each stored trace stands for about 100 real requests, so the estimate is 480 × 100 = 48,000. For that, the sample rate must be recorded on each stored event. Metrics do not have this problem, because they count every request before any sampling.

Structured logs

A free-text log is like a letter: 'Checkout request from alice for 2 pens'. A person reads it easily, but a computer cannot easily ask 'how many requests took more than 500 ms?'. You would write a regular expression that breaks as soon as someone changes the wording of the sentence.

A structured log is like a form with boxes. Each record is a list of name and value pairs, usually in JSON or logfmt. A tool can filter by status, add up duration_ms or group by route without caring about any sentence.

{"ts":"2026-10-07T17:03:22.418Z","level":"info",
 "route":"/orders","status":200,
 "duration_ms":1240,"correlation_id":"7f3a"}

The correlation id

When many requests happen at once, their lines mix together in the log. A correlation id (or request id) is a value created when the request enters the system and repeated on every log line about that request, in every service it passes through. Then one search for that value returns the whole story.

The id only works if it travels across each boundary between processes. Over HTTP it goes in a header, for example X-Correlation-Id. Through a queue it goes in a header or property of the message: the producer fills it and the consumer reads it before processing. Nothing does this by magic. Your code or an instrumentation library must copy the value.

Two sides must work: the caller injects the id into the call, and the receiver extracts it and uses it. A service creates a new id only when it is the entry point, meaning the request arrived without one. If every service creates its own, the logs are correct inside each service but cut apart between them.

Good habits for fields

• Time in UTC and ISO 8601 (RFC 3339), such as 2026-10-07T17:03:22.418Z, with milliseconds or better. All machines mean the same instant, and alphabetical order matches time order.
• Never write secrets: passwords, tokens, card numbers and personal data stay out, even as an example. Logs usually have weaker access control than the database.
• Write what helps an investigation: identifiers, route, status, duration.

Searching with LogQL

Loki uses a query language called LogQL. A query is a stream selector in braces followed by a pipeline, and each stage receives only what the previous one let through.

{service="orders"} |= "timeout" | json | status >= 500

• {service="orders"} picks the streams by their indexed labels.
• |= "timeout" keeps only lines that contain the text.
• | json turns the JSON fields into labels that can be compared. Nested fields are flattened with _, so http.status becomes http_status.
• | status >= 500 is a number comparison, not a fragile regular expression. The same works for durations, for example | duration_ms > 500.

Metric types and cardinality

Prometheus has a few types of metric, like different kinds of measuring tools in a kitchen: a counter is a click counter at the door, a gauge is a thermometer, a histogram is a set of jars sorted by size.

Counters and rate

The raw value of a counter says little. What matters is how fast it grows. The rate is the change divided by the time. If a counter was 1200 at t = 0 s and 1500 at t = 60 s, the increase is 300, and 300 / 60 = 5 requests per second. The rate() function does this over a window, with adjustments for resets and for the edges. increase() is the same without dividing by the time.

When the value of a counter goes down, Prometheus understands that the process restarted, and counts the whole next sample as increase. For samples 980, 1000, 15, 40: the steps are 20, then 15 (after the reset), then 25, so the total is 60. For this reason you apply rate() first, and sum after. If you add up the counters of several instances first, the resets are hidden.

Histograms in detail

A classic histogram has buckets whose limits you choose in the code. Each bucket is a series _bucket{le="bound"}, and it is cumulative: it counts every observation less than or equal to the bound (le means 'less than or equal'). The last bucket is le="+Inf". There are also _sum (the total of the observed values) and _count (the number of observations), and all are counters.

..._bucket{le="0.1"}    40
..._bucket{le="0.25"}   70
..._bucket{le="0.5"}    90
..._bucket{le="1"}     100
..._bucket{le="+Inf"}  100

To find the 80th percentile (quantile 0.8) from this table: the total is 100, so we look for rank 80. Up to 0.25 there are 70 and up to 0.5 there are 90, so rank 80 is inside the bucket from 0.25 to 0.5, which holds 90 - 70 = 20 observations. We assume the observations are spread evenly in the bucket: (80 - 70) / 20 = 0.5 of the way. The estimate is 0.25 + 0.5 × (0.5 - 0.25) = 0.375 s. It is only an estimate: the error depends on how wide the bucket is.

Histogram against summary

A summary computes quantiles in the application and exposes them ready, per instance. A histogram exposes counts per bucket, and the quantile is estimated later on the server. Percentiles are not additive: the average, sum or maximum of the p99 of each instance is not the p99 of the whole service. Bucket counts are additive. So, to get the percentile of a whole service, use a histogram, with this recipe:

histogram_quantile(0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

Cardinality: counting series

Each different combination of label values is its own time series, with its own memory and index cost. The maximum is the product of the numbers of values. A counter with method (4 values), status (5 values) and instance (3 instances) can make 4 × 5 × 3 = 60 series. Add a user_id label with 10,000 values and it becomes 600,000.

A classic histogram multiplies this even more, because each label combination carries one series per bucket (the finite bounds plus +Inf) plus _sum and _count. With 10 finite bounds, route (20 values) and method (3 values) you get (10 + 1 + 2) = 13 series per combination, 20 × 3 = 60 combinations, so 13 × 60 = 780 series per instance.

Metric instruments in OpenTelemetry

Distributed tracing: spans and context propagation

Imagine passing a note in class from the first desk to the last. Each child who touches it writes the time and their name on it. At the end you can see the whole path and who held the note for a long time. A distributed trace does this for a request that crosses many services.

Trace, span and the tree

A trace is the record of one request as it crosses the system. It is made of spans. A span is a unit of work with a start and a duration. Each span has three identifiers: the trace id (the same for the whole request), its own span id, and the span id of its parent. The backend groups spans by trace id and builds a tree by following the parent ids. Without a shared trace id, spans from different services would be loose events.

The only span with no parent is the root span. It represents the whole request, as seen where it entered the system. All others are its children, grandchildren, and so on.

span                      start    end
gateway   GET /checkout        0    900 ms
  orders    POST /orders       20    880
    orders    db.insert        30     90
    inventory GET /stock      100    860
      inventory stock.read    120    840

Reading it: gateway is the root. Its duration is 900 ms, but its self time is only 900 - 860 = 40 ms. Self time of POST /orders is 860 - (60 + 760) = 40 ms, of db.insert is 60 ms, of GET /stock is 760 - 720 = 40 ms and of stock.read is 720 ms. The slow part is stock.read, even though six lines look slow.

Context propagation

How do three different programs agree on the same trace id? The caller injects the trace id and its own span id into the headers of the call. The receiver extracts them and uses them as the parent of its spans. Inside one process, the context passes from function to function. Between processes, it travels in the headers of the protocol. This is called context propagation.

The standard is W3C Trace Context. Its header is called traceparent:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             |  |                                |                |
             |  trace id (32 hex digits)         |                flags (2)
             version (2)                         parent id (16 hex digits)

The parent id is the span id of the caller, which becomes the parent of the span the receiver creates. An id made only of zeros is invalid. The last field holds flags, and one of them is the sampled flag, which says whether the trace is being recorded.

Span kind

Attributes, baggage and links

• Attributes describe one span and are exported with it. They go nowhere else.
• Baggage is a set of key and value pairs that follows the context from service to service, so whoever is downstream can read it. Because it goes in headers to any destination of the call, never put sensitive data in baggage.
• Span links point to other span contexts, in the same trace or in other ones. They are for asynchronous work with several causes. A worker that processes 50 messages in one batch span links that span to the 50 requests that published them, because a normal parent has only one.

Clocks lie a little

The tree comes from ids, which are exact. But the position of each span on the timeline comes from the clock of the machine that made it. Clocks of different machines drift, so a child span can look like it starts before its parent. The duration of each span stays reliable, because it is measured on a single clock.

Sampling

Recording every trace costs a lot, so we keep some. If we keep 1 in 20 and write the rate 20 on each kept trace, then 350 stored error traces stand for 350 × 20 = 7,000 real requests with an error. Adding up the rates gives the estimate of the total.

With tail sampling and several collector instances, a layer before them must route by trace id, not round-robin. If spans of one trace go to different instances, each instance sees only part of the trace and decides wrong.

A parent-based sampler uses its own rule only for root spans. When there is a parent, it repeats the parent's decision, read from the sampled flag of traceparent. With the gateway at ratio 0.1 and orders at 0.5, orders still records only the traces the gateway chose, so 100 of every 1,000 requests, complete in both services. The decision made at the entrance holds for the whole trace.

OpenTelemetry

Imagine every country had a different plug for electricity. You would need a new charger for each trip. Before OpenTelemetry, each monitoring vendor had its own way to collect data. OpenTelemetry (often written OTel) is the universal plug: an open, neutral set of specifications and tools to produce and carry telemetry. It does not store or draw anything. Storing, querying and showing are done by a backend you choose.

API against SDK

Libraries depend only on the API, which is light and stable. The application decides whether and how to measure by configuring the SDK. If a database library creates spans through the API and the application never installs the SDK, the calls do nothing. That is the point: a library can be instrumented without forcing a cost on anyone.

Resource and semantic conventions

A resource is the set of attributes that describes who produces the telemetry, and it applies to all spans, metrics and logs of that program. The most important one is service.name. Without it the SDK uses unknown_service, and the data of different services gets mixed in the backend. You can set it with the OTEL_SERVICE_NAME variable.

If each team picks its own names (status, statusCode, http_status), a query that crosses services must know all of them. The semantic conventions fix a shared vocabulary, such as http.request.method, http.response.status_code and service.name. The same query and panel then work for any service.

OTLP and ports

OTLP has two transport variants. gRPC uses port 4317. HTTP uses port 4318, and each signal has its own path, such as /v1/traces. Pointing an HTTP exporter at the gRPC port, or the reverse, is a very common reason why telemetry never arrives.

The Collector

The Collector sits between applications and backends. Applications speak only OTLP to one address. Everything specific to a vendor lives in the Collector configuration. To change or add a backend you edit the Collector, and you do not deploy the services again.

receivers:
  otlp:
    protocols:
      http:
processors:
  batch:
exporters:
  otlp/tempo:
    endpoint: tempo:4317
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Automatic and manual instrumentation

Automatic (zero-code) instrumentation intercepts well-known libraries, such as HTTP servers, database clients and queues, and gives you the skeleton of the trace without changing code. It does not know your business: the total of an order, the plan of a customer, a calculation step. That data enters through manual instrumentation with the API. Both live together in the same trace.

Short programs: flush before exit

The batch span processor keeps finished spans in a queue and sends them in groups from time to time, so the application is not slowed on every span. The price: recent spans are still in memory when the process ends. In a short-lived script, shut the provider down before exiting (this empties the queue), or use the simple processor, which exports each span as it ends.

Logs in OpenTelemetry

For logs, OpenTelemetry does not ask you to rewrite your logging calls. The Logs API exists so that existing logging libraries connect to the SDK through a bridge. Every log record written while a span is active receives the trace id and span id of that context. With those two fields the backend jumps from a log line to its trace and back.

SLI, SLO and error budgets

Say a lemonade stand promises 'at least 99 of every 100 cups will be cold'. To keep the promise you must count the cups, you need a target, and you need to know how many warm cups you are still allowed to serve. Reliability works in the same way, with three words that look similar and are different.

The SLO is usually tighter than the SLA. This way the team notices the problem and acts before the contract is broken.

Error budget

The error budget is what the SLO allows to fail: 1 minus the SLO. With 99.9%, the budget is 0.1%. Think of it as a piggy bank of failures that you are allowed to spend on releases, experiments and surprises.

• Request-based SLO: with 1,000,000 valid requests and a 99.9% target, 0.001 × 1,000,000 = 1,000 requests may fail.
• Time-based SLO: 99.9% of good minutes in 30 days (43,200 minutes) allows 0.001 × 43,200 = 43.2 bad minutes.

Each extra nine divides the budget by ten. A team should not choose 100%. The budget would be zero, so no deploy, experiment or maintenance could ever go wrong, and the team would stop changing the system. Also, the user cannot tell 100% from 99.99%, because the network and the user's own phone fail more often than that.

When the budget runs out before the end of the window, the team puts effort on stability instead of new features. While there is budget left, the team may ship and experiment. The budget turns the argument between speed and reliability into a decision made with a number.

Budget consumed

An example: SLO 99.5%, with 2,000,000 valid requests and 6,000 failures in the window. Allowed failures are 0.005 × 2,000,000 = 10,000. So 6,000 / 10,000 = 60% of the budget is gone. The same answer comes from the error rate (0.3%) divided by the budget fraction (0.5%).

Burn rate

The burn rate tells how many times faster than allowed the budget is being spent: observed error rate divided by (1 - SLO). With burn rate 1, the budget lasts exactly the SLO window. For a 99.9% SLO and 0.5% errors in the last hour: 0.005 / 0.001 = 5. At that pace the 30-day budget is gone in 30 / 5 = 6 days.

With burn rate B, a whole budget lasts window / B. If 40% was already used and the burn rate is a steady 6, the remaining 60% lasts 0.6 × 720 hours / 6 = 72 hours. This calculation tells you whether a problem can wait for office hours or must wake someone up.

Choosing the unit of the SLI

Imagine a day with 1,440 minutes and 1,000,000 requests. During the night the service was down for 36 minutes, and only 2,000 requests arrived (all failed). By time: (1,440 - 36) / 1,440 = 97.5% good. By requests: (1,000,000 - 2,000) / 1,000,000 = 99.8% good. The same incident gives two numbers. Measured by time, every minute weighs the same. Measured by requests, every minute weighs what traffic it had, which follows the impact on users more closely.

Latency SLI and percentiles

A latency SLI is a ratio too: requests under the threshold divided by the total. With a cumulative histogram, the numerator is the bucket of the threshold and the denominator is the +Inf bucket. If 9,700 of 10,000 requests took up to 0.3 s, the SLI is 97%. Against a target of '95% within 300 ms', that passes.

A percentile says 'below which value do p% of measurements fall'. With the nearest-rank convention, the percentile p is the value at position ⌈p × n⌉ counting from 1. For 10 sorted values (20, 22, 25, 27, 30, 31, 35, 40, 48, 900 ms), the p90 is at position 9: 48 ms. The mean is 117.8 ms, a number no request ever had. For this reason latency SLIs use percentiles or the share under a threshold, not the mean.

Alerting

A smoke detector that rings every time you make toast soon gets ignored. An alert is a message that asks a person to act. If most alerts need no action, people stop reading them, and then the one real alert is lost in the noise. This is alert fatigue. So an alert that pages a person must be rare, urgent and actionable.

Symptoms against causes

Alerting on symptoms, with SLOs, gives fewer alerts and each one matches a real problem. The causes are left for the investigation.

Prometheus alerting rules and the for clause

- alert: HighErrorRatio
  expr: job:http_errors:ratio_rate5m > 0.05
  for: 5m
  labels:
    severity: page

An alert goes through three states. Inactive: the expression returns nothing. Pending: it returns something, but for less time than for. Firing: it returned something for at least for. If the expression became true 2 minutes ago and for is 5 minutes, the alert is pending. The for clause avoids paging for a spike that goes away alone, at the price of delaying every alert by that time.

for needs the expression to be true at every consecutive evaluation. A single false evaluation resets the wait. Look at a service where every 5 minutes there is a 60-second burst with half of the requests failing. Overall 10% of requests fail, yet with a 1-minute window and for: 10m the alert never fires, because it is false between bursts. That is 12 minutes of bursts per hour burning the budget with no alert.

Burn-rate alerts

A burn-rate alert asks: 'is the budget being spent too fast?'. Choose a share of the budget and a time, and compute the burn rate that matches. To open a ticket when 5% of the budget of a 99.9% SLO (30 days = 720 hours) is spent in 6 hours: the service must go 0.05 × 720 / 6 = 6 times faster than allowed. In error rate that is 6 × 0.1% = 0.6%. Another well-known case: spending 2% of the budget in 1 hour means burn rate 14.4.

- alert: ErrorBudgetBurn
  expr: |
    ratio_rate1h > (14.4 * 0.001)
    and
    ratio_rate5m > (14.4 * 0.001)
  labels:
    severity: page

Why two windows? The long window (1 hour) shows that a relevant part of the budget was spent and protects against brief spikes. Alone, it would stay above the threshold for up to an hour after the problem ended. The short window (5 minutes) confirms that the burn is happening now, so the alert clears within minutes of recovery.

A predictive alert measures the burn pace over a baseline window and asks whether, at that pace, the budget runs out within a lookahead window. Example: 2,000 failures left and 150 in the last hour. 2,000 / 150 = 13.3 hours, which is 13 h 20 min. A 4-hour lookahead would not fire. A 24-hour lookahead would.

Alertmanager

Prometheus decides which alerts are firing. Alertmanager decides what to notify. Three tools reduce noise:

RED, USE and golden signals

When you open a dashboard, what should be on it? Checklists help. There are three famous ones, and each looks at the system from a different place. Think of a restaurant: RED is how the customers feel (how many orders, how many wrong, how long they wait). USE is how the kitchen equipment is doing (stove busy, orders waiting, burnt dishes).

RED says what to measure in every request-driven service: the request rate, the number of errors and the duration (best as a histogram). These three are the base of standard dashboards and of SLOs.

USE is a checklist to find bottlenecks: for every resource, look at utilization (how much of it is busy), saturation (how much work is queued because it did not fit) and errors.

The golden signals are almost RED with other names: latency is duration, traffic is rate, errors are errors. The extra one is saturation, how close the service is to its limit. RED leaves saturation to USE, applied to resources.

Utilization is not saturation

Imagine a host with 4 CPU cores, all busy 100% of the time, and an average of 12 runnable threads waiting for a core. Utilization is 100%. The 12 is saturation: work that did not fit and waits in a queue. A resource at 100% with no queue is at its limit. A resource at 100% with a long queue is already delaying everyone who depends on it.

Which method for what

• A checkout HTTP API is a request-driven service: use RED or golden signals.
• The disks of the database host underneath are resources: use USE.
• In a real system, use both, each at its own layer.

Latency and errors contaminate each other

Suppose a dependency goes down. 40% of requests now return HTTP 500 in 5 ms, and the rest still take 300 ms. The average latency becomes 0.4 × 5 + 0.6 × 300 = 182 ms. The graph of the average shows an 'improvement' from 300 ms to 182 ms, while 40% of users are failing. The advice is to measure latency split by outcome (success against error) and to read the signals together: the 40% error rate explains the fake improvement.

Profiling

A trace tells you which service was slow. But inside that service, which line of code is the slow one? Think of a photographer who takes a picture of your busy classroom 100 times a second and writes who is doing what in each picture. After a while she knows where the class spends its time. That is a profiler.

A sampling CPU profiler interrupts the program many times per second and writes down the call stack at that moment (the list of functions currently running, one calling the next). Adding up the samples shows where the CPU time concentrates. It is cheap enough to run in production. A trace tells where a request was slow across services. A profile tells which code spent the CPU inside one process.

Flame graphs

A flame graph draws the samples. Each rectangle is a function in a stack. The vertical axis is stack depth. The width is the share of samples that contain that function. The horizontal axis is not time: equal stacks are merged and sorted to be easy to read. So look for what is wide, not for what is on the left.

root                      1000
+- handleSearch            520
|  +- parseQuery            60
|  +- compileRegex         230
|  +- rankResults          230
+- handleExport            400
|  +- compileRegex         210
|  +- writeCsv             190
+- gc                       80

The self time of a frame is its width minus the width of its children: the exposed top of the flame. A flame graph splits the same function when it is called through different paths. Here compileRegex appears twice, with 230 + 210 = 440 samples, which is 44% of the CPU. Adding up per function is what pprof top does.

Reading pprof: flat and cum

  flat   flat%     cum   cum%
 0.05s   0.50%   9.80s 98.00%  net/http.(*conn).serve
 0.30s   3.00%   9.40s 94.00%  main.buildRows
 6.50s  65.00%   6.50s 65.00%  main.normalizeKey
 1.20s  12.00%   1.20s 12.00%  runtime.mallocgc

• flat is the time spent in the function's own code.
• cum (cumulative) also includes everything the function calls.
• Functions near the root, such as the HTTP server loop, have a huge cum and a flat near zero. The hot spot is the function with a high flat. Here it is main.normalizeKey, with 6.50 s.

Prometheus, Grafana, Loki and Tempo basics

A school library has different shelves for books, magazines and maps, but one desk where you ask for anything. In an observability stack each signal has its own store, and one tool puts everything on one screen.

services --metrics--> Prometheus --+
         --logs-----> Loki       ----+--> Grafana
         --traces---> Tempo      ----+  (one screen)

Because Grafana queries all three, you can jump from a metric to a trace and to its logs on the same screen.

Prometheus: pull and scrape

By default Prometheus works in a pull model. Every configured interval it makes a scrape: an HTTP request to the metrics endpoint of each target. It stores the samples with the time of collection. The service only needs to expose its current values. A useful side effect: the scrape itself shows whether a target is alive, through the up metric.

Loki: index labels, scan content

Loki indexes only the labels of the streams, not the content of the lines. The index answers one question: which streams exist in a time range. Everything else, including |= and the parsers, is done by reading and scanning the content of the selected streams. Take the query:

{env="prod"} |= "4bf92f3577b34da6a3ce929d0e0e4736"

Going from 15 minutes to 7 days makes it much slower, because it must scan 7 days of lines of all prod streams. The cost depends on how much data the selector and the range let through. More specific selectors (add service) and short ranges make queries fast.

Tempo and TraceQL

In TraceQL, a pair of braces is a span selector, and the conditions inside apply to one and the same span. Attributes are scoped: resource. for the emitter, span. for the operation. Intrinsics such as duration, name and status come from the span itself.

{ resource.service.name = "inventory" && duration > 500ms }

Putting it all together

Now let us follow one real problem from the beginning. Users say checkout is slow, but only sometimes. Here is how the pieces work as a team.

1. An SLO alert fires. A burn-rate alert over two windows says the latency budget is being spent too fast (alerting, SLO).
2. A dashboard shows RED and USE. Rate and errors look normal, but the duration histogram has a fat tail (metrics, RED).
3. An exemplar opens a trace. The trace shows inventory stock.read has the largest self time (traces, self time).
4. The logs of that span explain it. Searching the trace id or correlation id in Loki gives the structured lines of that exact request (logs, correlation).
5. A profile points at the code. A flame graph shows one function with a wide frame (profiling).

Common mistakes

• Putting an id with unbounded values (user id, request id, trace id) in a metric label or a Loki label.
• Averaging or summing percentiles across instances instead of summing histogram buckets.
• Calling the longest span the culprit instead of the span with the most self time.
• Forgetting to forward the trace headers, which cuts one trace into two.
• Alerting on causes such as CPU, which wakes people without user pain.
• Choosing a 100% SLO, which leaves no error budget.
• Defining a Collector exporter but not listing it in a pipeline.