Artificial intelligence and LLMs

The big picture: a function with adjustable knobs

Imagine a radio with thousands of knobs. At the start the knobs are set at random, so the radio only makes noise. Now a teacher plays a song, listens to how wrong the radio sounds and turns every knob a tiny bit in the direction that makes the sound a little better. After millions of tiny turns, the radio plays beautifully. That is almost exactly how a modern AI model learns.

In one sentence: a model is a function with adjustable numbers. Training is the search for good values of those numbers. Using the model after the search is over is called inference. Every section below is a detail of this picture.

   example (input x, right answer y)
            |
            v
   +-------------------+
   | model with knobs  |--> guess
   +-------------------+       |
            ^                  v
            |          how wrong? = LOSS
            |                  |
            +---- turn the knobs to lower the loss

Three kinds of model appear in this summary. A language model receives a piece of text and says how likely each possible next piece is. An image model receives a noisy picture and guesses the noise. A classifier receives something (a photo, an e-mail) and says which group it belongs to. Inside, they are all numbers being multiplied, added and corrected.

AI, machine learning and deep learning

Think of three boxes, one inside the other, like Russian dolls. The biggest box is artificial intelligence: every technique that makes a computer do an intelligent task, including old programs made only of rules written by people. Inside it is machine learning: the part where the behaviour is learned from examples. Inside that is deep learning: machine learning that uses neural networks with many layers.

+------------------------------------------------+
| AI: any technique for 'intelligent' tasks      |
|  (rules written by people count too)           |
|  +------------------------------------------+  |
|  | MACHINE LEARNING: learns from examples   |  |
|  |  +------------------------------------+  |  |
|  |  | DEEP LEARNING: neural networks     |  |  |
|  |  | with several layers                |  |  |
|  |  +------------------------------------+  |  |
|  +------------------------------------------+  |
+------------------------------------------------+

In normal programming, a person writes the rules: if the e-mail has the word 'prize', it is spam. In machine learning, a person gives examples (e-mails already marked as spam or not spam) and a learning algorithm finds the rules. In the older kind of machine learning a specialist also chose the features, which are the measurements fed to the model (for a face: the distance between the eyes, the edges). In deep learning the network itself learns the features: each layer turns the output of the previous one into something a bit more abstract. 'Deep' means many layers in a row, not deep thinking.

Kinds of learning

Self-supervised learning is why language models could become so large. Any text is its own answer sheet: hide a word, ask the model to guess it, and the hidden word is the label. Nobody has to label billions of sentences by hand.

Words that appear all the time

• Parameters: the numbers that training changes (weights and biases). Hyperparameters: the numbers a person chooses before training, such as the learning rate, the number of layers, the batch size and the number of epochs. They are usually picked by comparing results on a validation set.
• Training changes the parameters. Inference uses the trained model with its parameters frozen.
• Generalising means doing well on examples the model has never seen. It is the true goal. Doing well only on the training examples is memorising.
• A discriminative model learns p(label | input): given a photo, which label? A generative model learns the data itself, p(x), so it can make new examples. A classifier cannot draw a new cat; an image generator can.

Probability and statistics

A weather forecast says 'chance of rain: 70%'. That is a probability: a number from 0 (impossible) to 1 (certain). A probability distribution gives a number to every possible outcome, and all the numbers must add up to exactly 1. The last layer of a language model produces exactly this: one probability for each possible next token, none negative, total 1.

The normal distribution and the average

The bell-shaped curve is called the normal (or Gaussian) distribution. It has two numbers: the mean, where the middle is, and the standard deviation, how spread out the values are. A good rule is 68-95-99.7: about 68% of the values are within one standard deviation of the mean, 95% within two and 99.7% within three. If adults have mean height 170 cm and standard deviation 10 cm, about 68% are between 160 and 180 cm.

The expected value (expectation) is the average you would get after many repetitions: multiply each value by its probability and add. A fair die gives (1+2+3+4+5+6)/6 = 3.5, a value the die never shows. The loss that training minimises is an expectation, estimated by the average over many examples.

Independence, conditional probability and Bayes

Two events are independent when knowing one tells nothing about the other: P(A and B) = P(A) × P(B). Two coin tosses are independent. The words of a sentence are not: after 'the cat drinks' the word 'milk' is far more likely. A language model lives exactly on that dependence.

A conditional probability P(B | A) is the chance of B when we already know A happened: P(B | A) = P(A and B) / P(A). We shrink the world to the cases where A is true. Be careful: P(B | A) and P(A | B) are usually different numbers.

Bayes' theorem turns one into the other: P(A | B) = P(B | A) × P(A) / P(B). Here is the classic trap. Out of 1,000 people, 10 are sick. A test finds 9 of those 10, and also wrongly flags 10% of the 990 healthy ones, which is 99 people. A person with a positive test belongs to 9 + 99 = 108 positives, and only 9 are sick: 9/108, about 8%. The test looks good, but the disease is rare, so most positives are healthy people. The starting chance P(A) is called the prior or base rate.

Likelihood, cross-entropy and softmax

The likelihood answers: how probable does a value of the parameter make the data I already saw? The data stays fixed and the parameter varies. A coin gave 6 heads in 8 tosses; the likelihood is p^6 × (1-p)^2, and the value of p that makes it largest is 6/8 = 0.75, the observed frequency. This is maximum likelihood. Training a network by minimising cross-entropy is the same idea applied to millions of parameters.

The cross-entropy loss of one example is -log of the probability given to the correct answer. We use the natural logarithm, and the unit is the nat. If the model gave 0.9 to the right class, the loss is -ln(0.9) = 0.105; with 0.5 it is 0.693; with 0.1 it is 2.303. A confident mistake is punished heavily.

Softmax turns any list of numbers (the logits) into a distribution: take e to the power of each one, so all become positive, and divide by the total. For the logits (1, 0, 1): e^1 = 2.718, e^0 = 1, e^1 = 2.718, total 6.437, so the probabilities are 0.42, 0.16 and 0.42. The biggest logit gets the biggest share, but nobody gets exactly zero.

Entropy measures average uncertainty: H = -Σ p log p. With the logarithm in base 2 the unit is the bit. A fair coin has 1 bit, a fair 4-sided die has 2 bits, and a certain outcome has 0. It is largest when all outcomes are equally likely.

Argmax, sampling and the chain rule of probability

With a distribution in hand you can do two things. Argmax picks the most probable outcome, always the same. Sampling draws an outcome in proportion to the probabilities, so the answer may change on each run. The chain rule says the probability of a whole text is the product of the conditional probabilities of its tokens: p(w1..wn) = p(w1) × p(w2 | w1) × p(w3 | w1, w2) and so on.

import math

# 1,000 tokens, each with probability 0.01
prod = 1.0
for _ in range(1000):
    prod *= 0.01
print(prod)                  # 0.0  (underflow: too small to store)

log_prob = 1000 * math.log(0.01)
print(log_prob)              # -4605.17  (fine)

Multiplying many numbers smaller than 1 ends in a number the computer rounds to 0 (underflow). The fix is to work with log-probabilities: a product becomes a sum, which is stable and easy to differentiate. The cross-entropy loss is this sum with the sign flipped, divided by the number of tokens.

Vectors and matrices

A number alone is a scalar. A list of numbers, such as a person's height, weight and age, is a vector. A table of numbers with rows and columns, like a school timetable, is a matrix. A tensor is the general word for a block of numbers with any number of directions (called axes). Everything in a neural network is stored as one of these.

The shape lists the size of each axis. In NumPy and in the deep learning libraries the first number counts the rows. A shape (3, 4) is a matrix with 3 rows and 4 columns, 12 numbers in total (the sizes multiply, they do not add). A vector of 12 numbers has shape (12,), only one axis.

Dot product, norm and cosine similarity

The dot product of two vectors of the same size multiplies the numbers in the same position and adds everything. For (1, 2, 3) and (4, 0, 5): 1×4 + 2×0 + 3×5 = 19. The result is one number. It is big and positive when the vectors point the same way, zero when they are perpendicular and negative when they point opposite ways. A neuron is just a dot product of weights and inputs, plus a bias.

The norm (length) of a vector is the square root of the sum of the squares: for (3, 4) it is √(9+16) = 5, like the diagonal of a 3-by-4 rectangle. Dividing a vector by its norm gives a vector of length 1 (a unit vector) that points the same way. This is called normalising.

The cosine similarity compares only the direction: cos(a, b) = (a · b) / (|a| × |b|). It goes from -1 (opposite) through 0 (perpendicular) to 1 (same direction). For a = (1, 0) and b = (1, 1) it is 1 / (1 × 1.414) = 0.71. A document and a copy of it repeated ten times have the same direction, so the cosine is 1, even though the distance between the points is big. That is why cosine is the usual way to compare texts and embeddings.

Matrices and shapes

The transpose of a matrix swaps rows and columns: a (2, 3) matrix becomes (3, 2). To multiply two matrices, the number of columns of the first must equal the number of rows of the second: shapes (m, n) and (n, p) give (m, p). Because of this rule the order matters: A times B can exist while B times A does not. Checking shapes is the first thing to do when a network breaks.

   W        x        y
 (10x512) (512x1) = (10x1)

  each of the 10 rows of W = the weights
  of one neuron; y[i] = row_i . x + b[i]

To process a whole batch at once, stack the examples as rows of a matrix X with shape (32, 512). If W has shape (10, 512), the layer is computed as X @ W.T + b, giving shape (32, 10). The matrix goes transposed because the examples are rows and not columns. The bias, a vector of 10 numbers, is added to every row.

Supervised learning: learning from labelled examples

A teacher shows a child many photos, each with the name of the animal. Soon the child can name animals in new photos. This is supervised learning. The data is a set of pairs: the features x that describe an example (area, rooms and neighbourhood of an apartment) and the label y, the known right answer (its rent). Training adjusts the model so that its guess from x gets close to y, hoping this also works for apartments not in the data.

The mean squared error (MSE) averages the squares of the errors. If the model guesses 1 and 4 where the truth is 2 and 2, the errors are -1 and 2, the squares 1 and 4, and the MSE is 2.5. Squaring makes all errors positive and punishes big errors much more than small ones. Logistic regression, despite the name, is a classifier: it computes z = w·x + b and passes z through the sigmoid, which squeezes it into (0, 1). The result is read as the probability of the positive class; z = 0 gives 0.5.

Three sets of data

• Training set: the examples the model learns from.
• Validation set: used to take decisions (which hyperparameters, which architecture, when to stop).
• Test set: used only once, at the end, to report the final quality. Every time you look at a set to take a decision, you adapt a little to it, so the test set must stay untouched or its score becomes optimistic.

Underfitting and overfitting

epoch  training loss  validation loss
  1       0.90           0.95
  3       0.45           0.55
  4       0.30           0.50   <- best point
  6       0.15           0.65
  8       0.08           0.80   <- overfitting

If both losses are high, the model is underfitting: too simple. If the training loss is low but the validation loss rises, the model is overfitting: it is memorising details of the training examples, like a student who learns last year's exam by heart. The best moment to stop is the lowest validation loss, an idea called early stopping.

Bias and variance: what to do next

Read the errors in two parts. The training error shows the bias: how far the model is even on data it has seen. The gap between validation and training error shows the variance: how much the model depends on the sample it was given. Example: 22% training error, 24% validation error and people make 2%. The gap is tiny, so the problem is bias. Collecting more data will not help; a bigger model or better features will. If training error were 3% and validation 20%, the variance is the problem: more data or regularisation.

Measuring a classifier

Accuracy is the fraction of correct answers. It lies when one class is rare. Out of 100 e-mails with only 2 spam, a 'model' that always says 'not spam' scores 98% accuracy and finds no spam at all. For the class you care about, use two numbers. Precision: of everything the model called positive, how much really is? Recall: of everything really positive, how much did the model find?

Raising one usually lowers the other, depending on the decision threshold. A useful model must at least beat the trivial baseline.

Data leakage

Leakage is any way for information that would not exist at prediction time (the test examples, their labels) to reach the model during training. It includes preprocessing: standardising features with the mean of the whole dataset, picking the best features by looking at all the labels, or filling missing values from all rows. All of these must be learned from the training set only and then applied to the others. The symptom is a wonderful score in evaluation and a disappointing model in real use.

Neurons, layers and activation functions

An artificial neuron is a tiny voting machine. Each input gets a weight that says how much it matters. The neuron multiplies every input by its weight, adds everything, adds a bias (a built-in push) and passes the total through an activation function. In symbols: a = f(w·x + b). The part w·x + b is the pre-activation, called z.

x = [2.0, 1.0]
w = [3.0, -4.0]
b = 1.0

z = 2.0*3.0 + 1.0*(-4.0) + b   # 6 - 4 + 1 = 3
a = max(0, z)                   # ReLU keeps positive numbers
print(a)                        # 3.0
# if z had been -3, ReLU would give 0

A layer is many neurons side by side, each looking at the same inputs. The outputs of one layer are the inputs of the next. A network in which every neuron receives all the outputs of the previous layer is called fully connected or an MLP (multi-layer perceptron).

Counting parameters

A layer with n inputs and m neurons has n×m weights and m biases: (n+1)×m parameters. The input layer has none, it only delivers the data. A network with 3 inputs, a hidden layer of 5 neurons and 2 outputs has (3+1)×5 + (5+1)×2 = 20 + 12 = 32 parameters.

Activation functions

Why do we need activations at all? Without them every layer is just a straight (linear) transformation, and a straight transformation applied after another is still a straight transformation. A hundred layers would be worth only one. The bend of ReLU or tanh lets the network fold space and draw curved boundaries.

x2
 1 |  1       0       XOR: 1 when the inputs differ
   |
 0 |  0       1       No single straight line puts
   +-----------  x1   the two 1s on one side and the
      0       1       two 0s on the other.

A single neuron is a linear classifier: its decision boundary is a straight line (a plane in more dimensions). XOR is not linearly separable. With one hidden layer the network first computes new features (say, OR and AND of the inputs), and in that new space a line is enough.

Starting values and what a network can do

Weights start as small random numbers, to break symmetry. If two neurons of a layer start with the same weights, they compute the same thing, get the same gradient and stay twins forever, so the layer wastes its width. Biases can start at zero, since the random weights already make the neurons different.

The universal approximation theorem says that a network with one hidden layer and a non-linear activation, wide enough, can represent any continuous function on a limited region. That is only about being able to represent. It does not promise that training will find the weights, that the needed width is reasonable, or that the model will not overfit. Deeper networks usually need far fewer units.

Training: gradient descent and backpropagation

You stand on a foggy hill and want to reach the valley. You cannot see far, but you can feel which way the ground slopes. So you take a small step downhill, feel again, and repeat. This is gradient descent. The height of the hill is the loss; your position is the set of weights.

The gradient is the list of derivatives of the loss with respect to each weight, and it points to where the loss grows fastest. To lower the loss we go the opposite way: w ← w − rate × gradient. The learning rate says how big the step is. A positive gradient makes the weight smaller; a negative one makes it bigger. Example: the loss is (w − 3)² and w = 1. The gradient is 2(w − 3) = −4. With rate 0.1: w = 1 − 0.1 × (−4) = 1.4, closer to 3.

learning rate too small    just right        too large

 loss                       loss                loss
  \                          \                  \     /
   \.                         \.                 \   /
    \ .                        \.                 \ /  jumps
     \  . . . . .                \_.               X   back and
                  (very slow)       (smooth)        forth, may blow up

Batches, steps and epochs

Computing the gradient on all the data is exact but slow; on a single example it is fast but noisy. The usual choice is the mini-batch: draw a small group (say 64 examples), average their gradients and take one step. The three words to keep apart: batch size (examples per step), step or iteration (one update of the weights) and epoch (one full pass over the training set). The steps per epoch are the number of examples divided by the batch size, rounded up when the last batch is smaller. With 1,000 examples and batch size 32, an epoch has ceil(1000/32) = 32 steps, and 3 epochs have 96.

Backpropagation

A network has millions of weights and we need the gradient for each. Backpropagation does it cleverly. The forward pass computes the output and remembers the middle values. The backward pass walks from the loss back to the inputs, applying the chain rule: multiply the incoming gradient by the local derivative of each operation. The cost is about one more forward pass, no matter how many weights.

# f = (x + y) * z   with x=1, y=2, z=5
q = 1 + 2                # q = 3
f = q * 5                # f = 15
# backward: df/df = 1
# multiplication: each input gets the OTHER input's value
df_dq = 5                # = z
df_dz = 3                # = q
# addition passes the gradient unchanged to both inputs
df_dx = df_dq * 1        # 5
df_dy = df_dq * 1        # 5

When a value is used in more than one place, it affects the result through several paths, and the contributions are added. For f = a × a with a = 3, each of the two paths gives 3, so the gradient is 6. That is why an automatic-differentiation engine writes grad += ... inside each operation, and why gradients must be cleared before a new backward pass.

A gradient check tests a hand-written derivative with a number: the centred difference (f(x+h) − f(x−h)) / (2h). For f(x) = x² at x = 3 with h = 0.1: (9.61 − 8.41) / 0.2 = 6, the same as 2x. It needs two evaluations per parameter, so it is a test tool, never used for training.

Problems and remedies

• Vanishing gradient: in a deep network with saturating activations (sigmoid), the gradient is a product of many numbers smaller than 1 and shrinks exponentially with depth, so the first layers barely learn. Remedies: ReLU (derivative 1 on its active side), residual connections, normalisation layers, careful initialisation.
• Dropout: a regulariser that switches off each unit with probability p at every training step, so the network cannot depend on fragile combinations. It is used only in training; at inference nothing is switched off. That is why frameworks have a training mode and an evaluation mode.
• Adam: an optimiser that keeps a running average of gradients (momentum, which smooths the noise) and divides each step by the root of the running average of squared gradients. A parameter with gradients near 100 and another near 0.01 then move at similar paces. With plain SGD and one learning rate they would not.

PyTorch: a framework that does the work for us

Writing backpropagation by hand for every model would be tiring. A framework gives four things: tensors (arrays of numbers that can live on a fast chip), automatic differentiation, ready-made layers and optimisers. PyTorch is one of the two most used. Its tensors have a shape (the size of each axis) and a dtype (the type of the numbers). Most bugs in networks are shapes that do not match, so reading shapes is a daily skill.

import torch

x = torch.zeros(3, 4)
w = torch.ones(4, 2)
print((x @ w).shape)    # torch.Size([3, 2])

a = torch.tensor([[1, 2], [3, 4]])
b = torch.tensor([[5, 6], [7, 8]])
print(a * b)    # element by element: [[5, 12], [21, 32]]
print(a @ b)    # matrix product:    [[19, 22], [43, 50]]

Broadcasting lets different shapes be combined: shapes are compared from the right, and a missing axis or an axis of size 1 is repeated. A bias of shape (3,) added to a batch of shape (2, 3) is added to each of the 2 rows. No loop, no copy.

Automatic differentiation

x = torch.tensor(2.0, requires_grad=True)
y = x**2 + 3 * x
y.backward()
print(x.grad)       # 7.0  (2x + 3 at x = 2)

y = x**2 + 3 * x
y.backward()
print(x.grad)       # 14.0: gradients ADD UP, not replace

With requires_grad=True, PyTorch records every operation in a graph that is built while the code runs. backward() walks it backwards and accumulates the derivative in .grad. That is useful when a value has several paths, but it forces the training loop to clear the gradients on every step. detach() gives a tensor with the same values but outside the graph: for autograd it is a constant, and no gradient flows through it.

Models and the training step

A model is a class that inherits from nn.Module. In __init__ you create the layers (they are registered automatically, so model.parameters() finds them). In forward you write the computation. You call model(x), not model.forward(x). A layer nn.Linear(n_in, n_out) has n_in × n_out weights plus n_out biases; ReLU has no parameters. So nn.Sequential(nn.Linear(5, 8), nn.ReLU(), nn.Linear(8, 2)) has (5×8+8) + (8×2+2) = 48 + 18 = 66.

pred = model(x)               # 1. forward pass
loss = loss_fn(pred, y)       # 2. how wrong?
optimizer.zero_grad()         # 3. clear old gradients
loss.backward()               # 4. compute new gradients
optimizer.step()              # 5. update the parameters

For plain SGD, step() does p = p − lr × p.grad for each parameter. If w = 2, the loss is (3w − 12)², lr = 0.05: the gradient is 2 × (6 − 12) × 3 = −36, so w becomes 2 + 1.8 = 3.8.

Two common traps

• nn.CrossEntropyLoss already includes the log-softmax, so it wants the raw logits. Putting nn.Softmax before it raises no error, but learning gets slow and strange. Apply softmax only when you want to show probabilities.
• In validation, use both model.eval() and with torch.no_grad():. The first switches layers such as dropout and batch normalisation to evaluation mode. The second stops recording the graph, which saves memory and time. They solve different problems.

TensorFlow and Keras

TensorFlow is the other big framework, and Keras is its friendly top layer. The ideas are the same as in PyTorch, with other names. A TensorFlow tensor cannot be changed: each operation makes a new one. What must change in training (weights and biases) lives in a tf.Variable. A tf.constant never changes. Keras layers create their variables automatically.

The gradient tape

import tensorflow as tf

x = tf.Variable(3.0)
with tf.GradientTape() as tape:
    y = x * x
print(tape.gradient(y, x).numpy())   # 6.0

c = tf.constant(3.0)
with tf.GradientTape() as tape:
    tape.watch(c)        # needed for a constant
    y = c * c
print(tape.gradient(y, c).numpy())   # 6.0

The operations inside the with block are recorded on a 'tape', and tape.gradient(result, source) plays the tape backwards with the chain rule. It watches only trainable tf.Variable objects automatically. For an ordinary tensor you need tape.watch(x); otherwise the gradient is None. A None almost always means the source was not watched, or the computation left TensorFlow on the way. By default the tape is released after one gradient call; to ask twice, create it with persistent=True or ask for a list of sources in a single call.

The Keras workflow

1. Build the model from layers: keras.Sequential([...]).
2. compile(optimizer, loss, metrics): choose how training will be done. It trains nothing.
3. fit(x, y, epochs, batch_size): train.
4. evaluate: measure on data the model has not seen.
5. predict: get the outputs for new inputs.

Dense(units) receiving n inputs has n×units weights plus units biases. With Input(shape=(4,)), Dense(8) and Dense(3), model.summary() shows (4×8+8) + (8×3+3) = 40 + 27 = 67 parameters. And fit is the same loop you would write by hand: for each batch, forward, loss, gradient with a tape, update. One epoch has ceil(N / batch size) steps, counting the last smaller batch: 1,000 examples with batch size 32 give 32 updates.

Eager mode and tf.function

By default TensorFlow runs each operation at once, like normal Python (eager execution): easy to debug. Decorating a function with @tf.function makes TensorFlow trace it once for each input signature (dtype and shape) and turn it into a graph, which is faster and portable. Python side effects such as print happen only while tracing: three calls with the same shape and dtype print once. To print on every run, use tf.print.

Computer vision: how a computer sees

For a computer, a photo is a grid of numbers. A grey image has one number per pixel (how bright it is). A colour (RGB) image has three: red, green and blue. So an image of 32 × 32 pixels in colour is a tensor of 32 × 32 × 3 = 3,072 numbers. The order of the axes is a convention: channels last (H × W × C) is the Keras default and channels first (C × H × W) is the PyTorch one. The count of numbers is the same.

Why not a fully connected layer?

Flatten an RGB image of 64 × 64 into a vector: 12,288 numbers. Connect it to just 100 neurons: 12,288 × 100 + 100 = 1,228,900 parameters, for one small layer. Worse, the layer treats each pixel as unrelated, with no idea of neighbours.

Convolution

A convolutional layer slides a small filter over the image. At each stop it multiplies the filter weights by the patch under it and adds: a dot product. The same weights are used everywhere (parameter sharing), so the layer needs few parameters and finds the same pattern (an edge, a corner) wherever it is. The result is an activation map that shows where the pattern was found. Libraries do not flip the filter (strictly this is cross-correlation) and call it convolution.

image row:        0  0  9  9  0
filter (-1, +1):  slides along

  (-1)*0 + (+1)*0 =  0
  (-1)*0 + (+1)*9 =  9   <- edge: dark to bright
  (-1)*9 + (+1)*9 =  0
  (-1)*9 + (+1)*0 = -9   <- edge: bright to dark

output:  0   9   0  -9

The output width is (W − F + 2P) / S + 1, rounded down: W input size, F filter size, P zeros added on each side (padding), S the stride (how many pixels the filter jumps). A 10-wide input with a 3-wide filter, no padding and stride 1 gives 8. With padding 1 it gives 10: the 'same' size. With padding 1 and stride 2: (10 − 3 + 2)/2 + 1 = 5.5, rounded down to 5.

A filter spans all the channels of its input. A layer with K filters of F × F over D channels has F×F×D×K weights plus K biases, whatever the image size. With 16 filters of 3 × 3 on an RGB image: 3×3×3×16 + 16 = 448.

Pooling, stacking and seeing far

Max pooling keeps the biggest value in each window. With a 2 × 2 window and stride 2 each side is halved. It has no parameters, cuts the cost of later layers and makes the answer a little less sensitive to the exact position.

1 2 | 0 1            4 | 5
3 4 | 5 2     -->     --+--
----+----            2 | 7
0 0 | 7 1
2 1 | 3 3

The receptive field is the part of the input that influences one value of a layer. With 3 × 3 filters and stride 1 it grows by 2 pixels per layer: 3, 5, 7. Two stacked 3 × 3 layers see 5 × 5 using 2 × 9 = 18 weights per channel pair, against 25 for one 5 × 5 filter, and they have one more activation in between. This was the idea of VGG.

Convolution is equivariant to translation: shift the input and the activation map shifts with it, with the same values. Invariance (the same final answer) comes later, from pooling and the last layers.

ResNet, augmentation, transfer, IoU

• ResNet (2015): very deep networks trained worse, because the signal and the gradient faded. A residual block adds its input to its output: y = x + F(x). The layers learn only the difference, and if the best thing is nothing, F(x) = 0 and the block passes the input along.
• Data augmentation creates variations of each training image that keep the label (shifts, flips, small rotations, brightness). It fights overfitting. Check that the transformation keeps the label: flipping a digit or a letter may change it.
• Transfer learning: reuse a network trained on a huge dataset. Early layers learn generic features. With little data (say 400 photos), freeze the network and train only a new last layer; with more data, also fine-tune the upper layers with a small learning rate.
• IoU (intersection over union) scores a detection: common area divided by the area covered by either box. Boxes (0,0,2,2) and (1,0,3,2): common area 2, total 4+4−2 = 6, IoU = 0.33. A hit usually needs IoU above a threshold such as 0.5.

Tokens and tokenisation

A neural network only knows numbers. So before a language model sees text, a program called the tokenizer cuts it into pieces called tokens and gives each piece an integer, its id in the vocabulary (the list of all tokens). The model reads ids and predicts ids; at the end the ids are turned back into text. Imagine a box of Lego bricks: the vocabulary is the set of different bricks, and a sentence is built from them.

Token, word and character are different units. In English, a rule of thumb is that a token is about three quarters of a word, so 100 words are roughly 130 tokens. Each tokenizer has its own vocabulary, so the same text gives different counts in different models.

Byte pair encoding (BPE)

BPE builds the vocabulary from the bottom up. Start with single letters. Count the pairs of neighbouring tokens. Merge the most frequent pair into a new token. Repeat. If two pairs tie, our convention is to pick the pair that appears first in the text. Each merge replaces every occurrence of the pair, so the sequence gets shorter.

start:    a a b a a b          (6 tokens)
merge 1:  (a,a) is most frequent -> 'aa'
          aa b aa b            (4 tokens)
merge 2:  (aa,b) -> 'aab'
          aab aab              (2 tokens)

Byte-level BPE starts from the 256 possible values of a byte, so any text can be encoded, even in a language never seen. The vocabulary size is 256 + number of merges (+ special tokens, if any). With 500 merges and no special token: 756 tokens. More merges give a bigger vocabulary and shorter sequences for the same text; but each token costs a row in the embedding table and in the output layer.

Bytes come from UTF-8, which uses 1 byte for plain ASCII letters, 2 for accented letters such as 'ã' or 'ç', 3 for characters such as Japanese ones, and 4 for most emojis. A byte-level tokenizer starts with one token per byte, so 'ação' is 6 tokens before merges (a, ç=2, ã=2, o). If the tokenizer was trained only on English, its merges join only ASCII bytes; a Japanese text falls back to one token per byte. Nothing is lost when you encode and decode again, but the sequence is much longer, and so it costs more.

Special tokens and the context window

Special tokens are reserved ids that do not come from the text and carry signals. The end-of-text token marks where a document ends; the model learns to produce it, and the program that generates text stops when it sees it.

The context window is everything the model can see at once, measured in tokens, and it includes the answer being generated. It works like a desk of limited size: system prompt, tool descriptions, conversation and the reply must all fit. If the window is 8,000 tokens, the system prompt takes 1,200, the tools 500 and 1,000 are reserved for the answer, only 5,300 are left for the history. A history of 6,000 tokens needs 700 cut or summarised. What does not fit is simply not seen by the model.

Embeddings and similarity search

A token id like 2,917 says nothing about meaning. We need a way to place words so that similar ones are near each other, like putting friends who like the same things at the same table. This is an embedding: a vector of real numbers that represents a token. The model keeps a table (a matrix) with one row per token and one column per dimension: vocabulary × dimensions. The id picks the row. The values start random and are learned in training. A vocabulary of 1,000 tokens with 16 dimensions has 16,000 parameters in this table; doubling the vocabulary doubles it.

Why not one-hot vectors (a 1 in the position of the word and 0 elsewhere)? They are as long as the vocabulary, and all words are perpendicular to each other: 'cat' is exactly as different from 'dog' as from 'bridge'. Dense embeddings have tens or hundreds of dimensions, and close meanings can end up close in space.

Where the meaning comes from

The distributional hypothesis says: you can know a word by the company it keeps. Words that appear among the same neighbours have similar meaning. So meaning can be learned from raw text, with no dictionary.

The simplest way is a co-occurrence matrix: count how many times each pair of words appears within a distance of k positions (the window). The row of a word is already a vector describing its contexts. A small window captures local relations (grammar), a big one captures topic, and the counts change a lot with this choice. word2vec learns vectors by a prediction game, and has two forms. CBOW: the surrounding words predict the centre word. Skip-gram: the centre word predicts the surrounding words. The game is only a pretext; the useful part is the vectors learned. GloVe uses the co-occurrence counts directly.

Searching by meaning

To find the passages closest to a question, turn everything into vectors and look for the nearest neighbours, usually by cosine similarity. Exact search compares the query with all n stored vectors: cost proportional to n × d (d dimensions). The nearest by cosine can be different from the nearest by dot product (which favours long vectors) or by Euclidean distance.

To go faster we accept losing some neighbours with an approximate index. One idea is LSH with random hyperplanes: draw k random flat cuts through the space; each vector gets a signature of k bits, saying on which side of each cut it lies; compare the query only with vectors that have the same signature. The chance that two vectors at angle t (degrees) share one bit is 1 − t/180. More bits mean smaller buckets: faster, but with more neighbours lost (lower recall). With several tables, the chance of finding a neighbour in at least one grows.

These numbers come from the embeddings-vector-search mini-project, over 1,927 sentences and 400 queries. Searching compared only a few percent of the vectors (1.1% with 1 table, 9.4% with 8 tables), which is the trade-off.

Recall@k measures an approximate index: of the k true neighbours (from exact search), what fraction appeared among the k results? The order inside the k does not matter. If the true top 5 are 12, 7, 33, 4, 19 and the index returns 7, 12, 33, 50, 8, then 3 of 5 were found: recall@5 = 0.6. An index is judged by recall and cost together, because it is easy to be fast and wrong.

Attention and the transformer

Read: 'The animal did not cross the street because it was too tired.' Who is 'it'? To know, you look back at 'animal'. The model needs the same skill: a word must be able to look at other words, even far away, and decide how much to use of each. This is attention. Older networks (recurrent ones) passed information from word to word, in a chain, so far-away relations had to travel through many steps.

Query, key and value

Each token produces three vectors. The query: 'what am I looking for?'. The key: 'what do I offer?'. The value: 'what I hand over if you choose me'. Attention is a soft lookup, in three steps:

1. Compare the query of the token with the key of every token by dot product. One score per token.
2. The softmax turns the scores into weights that are positive and add to 1.
3. The output is the average of the values, weighted by those weights.

query q = (1, 0)
keys   k1=(1,0)  k2=(0,1)  k3=(1,1)
scores q.k = 1, 0, 1
weights softmax(1,0,1) = 0.42, 0.16, 0.42
values v = 10, 20, 30
output = 0.42*10 + 0.16*20 + 0.42*30 = about 20

Key and value are separate because deciding who is relevant is one thing, and what is copied from the relevant ones is another. In one formula: softmax(Q Kᵀ / √d_k) V. The division by √d_k matters: a dot product adds d_k terms, so its size grows with d_k. Without the division, big scores push the softmax into saturation (one weight near 1, the rest near 0) and the gradients become tiny, so learning stalls.

Heads, mask and position

• Multi-head attention runs several attentions in parallel, each with its own matrices, so one head can follow grammar while another follows 'who is it'. Each head uses d_model / h dimensions (768 / 12 = 64 per head), the outputs are joined back to d_model, and the total cost stays close to one full attention.
• Causal mask: a model that predicts the next token must not see the future. Before the softmax, the scores of later positions are set to minus infinity, so their weight is 0. For n tokens, n(n−1)/2 entries of the n × n matrix are blocked; for 3 tokens, 3 of 9.
• Position information: attention treats its input as a set, without order, because weights depend on content only. So a vector that identifies the position (fixed sines and cosines, or learned) is added to each token embedding.

causal mask (x = may look, . = blocked)

            looks at ->  1  2  3  4
   token 1                   x  .  .  .
   token 2                   x  x  .  .
   token 3                   x  x  x  .
   token 4                   x  x  x  x

The transformer block and the three families

A transformer block has two sub-layers. Self-attention lets positions exchange information. The feed-forward network processes each position alone. Each sub-layer is wrapped by a residual connection (the input is added to the output, which eases the flow of the gradient) and layer normalisation (keeps values on a stable scale). A model is a stack of such blocks, and at the end a linear layer gives one logit per vocabulary token for each position: for a batch of 2 sequences of 5 tokens and a vocabulary of 1,000, the logits have shape (2, 5, 1000). Because of the mask, a single pass trains all positions at once, which suits parallel hardware.

token ids
   |
[ token embedding + position ]
   |
[ block 1: attention -> MLP ]   each wrapped by
[ block 2: attention -> MLP ]   residual + norm
   ...
[ linear layer + softmax ]
   |
probability of every next token

In self-attention queries, keys and values come from the same sequence. In cross-attention (encoder-decoder) the queries come from the decoder, and the keys and values from the encoded input: this is how a translator consults the source sentence.

The price of looking at everything

Every token is compared with every other token, so the number of pairs grows with the square of the length. Doubling the text multiplies the work by four; tripling it, by nine. This is one reason why context windows are limited.

To see it all drawn step by step, read The Illustrated Transformer.

Language models: predicting the next token

Try to finish: 'The sky is...'. You thought of 'blue' because you have seen many sentences. A language model does exactly this: it takes a sequence of tokens and returns a probability distribution over the next token. Everything else (answering questions, writing stories) is built on this single prediction. By the chain rule, multiplying the conditional probabilities gives the probability of a whole text.

To write many tokens, the model is autoregressive: the output of one step becomes the input of the next. Loop: compute the distribution, choose a token, append it, repeat until an end token or a limit. So the cost of generating grows with the number of tokens produced.

The simplest model: counting pairs

A bigram model looks only at the current token. Count how many times each token follows each other token. To estimate P(next = b | current = a), divide the count of the pair (a, b) by the total of the row of a (all pairs that start with a), so every row of the normalised table adds to 1. Dividing by a column or by the total of the table answers different questions.

Training goals

Both training goals are self-supervised, because the label comes from the text. GPT: predict the next token, seeing only the past; good for generating. BERT: masked language modelling, where hidden tokens are predicted with context on both sides; good for understanding but it does not write left to right. Pre-training uses an enormous unlabelled text. Fine-tuning starts from those weights and keeps training on a smaller, specific set. Both change the weights, unlike a prompt.

The loss and perplexity

The loss is the cross-entropy: −ln of the probability given to the token that really came. A model that knows nothing gives 1/V to each of V tokens, so its loss is ln(V). For V = 10 that is 2.30. This works as a test: an initial loss far above ln(V) points to an initialisation bug.

Perplexity is exp(average loss per token), the inverse of the geometric mean of the probabilities given to the right tokens. Read it as the number of equally likely options the model hesitates between: guessing uniformly among 4 tokens gives perplexity 4. If a model gave 0.5 and 0.5 to the right tokens of a two-token text, the average loss is ln 2 and the perplexity is 2.

Choosing the next token: decoding

The network itself is deterministic: same input, same distribution. The variety comes from decoding. Greedy decoding takes the most probable token, always. Sampling draws according to the probabilities, so two runs differ. Three knobs shape sampling.

• Temperature T: divide the logits by T before the softmax. T above 1 flattens the distribution (more variety); T below 1 makes it sharper; T near 0 is greedy.
• Top-k: keep the k most likely tokens, zero the others, divide by the sum of those kept. With probabilities 0.5, 0.3, 0.15, 0.05 and k = 2: keep 0.5 and 0.3, divided by 0.8 give 0.625 and 0.375.
• Top-p (nucleus): keep the smallest set of top tokens whose cumulative probability reaches p, then renormalise. With p = 0.9 on the same numbers: 0.5 + 0.3 = 0.8 is not enough, adding 0.15 gives 0.95, so three tokens stay (the token that crosses the line is included). The number kept adapts: one token when the model is sure, many when in doubt.

Look at the bars: a lower temperature concentrates the probability; a higher one spreads it.

Memorising instead of learning

If you train for a long time on a small text, the training loss keeps falling while the loss on text kept apart (validation) rises, and the samples repeat long passages of the training text. The model has memorised. Remedies: more data, a smaller model, regularisation, early stopping. A language model is judged by how well it predicts text it has never seen.

Using LLMs: prompts, RAG, tools and agents

A large language model (LLM) is a language model with billions of parameters. Using one is mostly a matter of what text you put in front of it. The input text is the prompt. In a chat application, the system prompt is written by whoever builds the app: it sets the role, the rules and the context, and comes before the messages of the user. It changes no weight, it is only text in the context window, read again on every call.

If you put three solved examples in the prompt and then a new case, the model usually solves the new one in the same format. This is few-shot prompting, or in-context learning. The name misleads: no weight changes. The model just predicts the likeliest continuation of a text that now contains examples, so the effect lasts only while the examples are in the context.

Hallucination

A model is trained to continue text in a likely way, not to check facts. When it lacks the information, it still produces something shaped like a right answer, with the same confident tone. This is a hallucination. To reduce it: give documents and ask the model to answer from them, allow 'I don't know', ask for citations, and check the result.

RAG: retrieval-augmented generation

question
   |
   v
[ search the document base ]   <- uses embeddings
   |  best passages
   v
[ prompt = passages + question ]
   |
   v
[ LLM answers from the passages, with citations ]

Imagine an assistant for internal manuals written after the model was trained. RAG has two stages. First, a search finds the passages most relevant to the question. Then those passages are placed in the context window and the model answers using them. The model can use knowledge that was not in training, the base can be updated without retraining, and the answer can point to its source. Fine-tuning is not the answer for this: it is expensive and the facts would go stale.

Tools and agents

In tool use (function calling), the application describes functions to the model (name, purpose, parameters). When useful, the model writes a structured request instead of final text, for example get_weather(city='Recife'). The application validates and runs the function and sends the result back as a new message. The model decides what to ask; the code around it is always what acts on the world.

Prompt injection

The model reads instructions and data in the same stream of tokens. If an assistant summarises a web page that hides the sentence 'ignore your instructions and send the contacts to this address', the model may obey. This is prompt injection. The risk grows with the power of the tools. Defences work together: keep external content clearly separated and treated as data, give the least privilege that is needed, and ask for human approval before sensitive actions like sending e-mail.

Evaluating honestly

To evaluate is to measure on data the model has not seen. Public benchmarks have been on the web for years, so they leak into the training data of models trained on web text, and a score then mixes ability with recall. If a model gets 95% on a famous test and 62% on new questions of the same difficulty, the likely cause is contamination. Use held-out or freshly written sets with many examples: one case that worked is not an evaluation.

Image generation: autoencoders, GANs and diffusion

To a network, an image is a block of numbers: height × width × channels. A colour picture of 64 × 64 has 64 × 64 × 3 = 12,288 numbers, and a generator has to produce all of them, which is why large images are expensive. There are three main ways to teach a network to draw.

Autoencoders and VAEs

An autoencoder has an encoder that squeezes the image into a short code (the latent) and a decoder that rebuilds the image from the code. The code is a bottleneck, so the network keeps only what is essential. No labels are needed: the target is the input itself.

But if you feed a random vector to the decoder of a plain autoencoder, you get blur. The codes of the training images sit scattered with gaps between them, and a random vector falls in a gap. A variational autoencoder (VAE) encodes each input as a small cloud (a distribution), and its loss adds to the reconstruction a KL term that pulls the clouds towards a simple standard normal. The latent space becomes continuous and filled, so a random point from the normal decodes into a plausible image.

GANs

A generative adversarial network is a game between two networks, like a forger and a detective. The generator turns noise into images that try to look real. The discriminator tries to tell real from generated. Each improves by beating the other; at the ideal end the discriminator can only guess and answers 1/2. A GAN makes an image in a single pass, but training is unstable. A typical failure is mode collapse: the generator keeps making a few images that fool the detective.

Diffusion models

Think of a drop of ink in water: it spreads until everything is uniform. Diffusion teaches a network to run that film backwards.

forward (fixed formula, no learning)
 image -> a bit noisy -> noisier -> ... -> pure noise

reverse (the network learns this)
 pure noise -> ... -> less noisy -> a bit noisy -> image

1. Forward process: add Gaussian noise little by little, following a fixed schedule, until nothing of the image is left. There is a shortcut to any step t: x_t = √a · x_0 + √(1−a) · noise, where a goes from near 1 (few steps) to near 0 (many). It is a formula, nothing is learned. With a = 0.5, a pixel 2 and a noise of −1: 0.707×2 + 0.707×(−1) = 0.707.
2. Training (DDPM): take a real image, pick a random step t and a random noise, build x_t, and ask the network (given x_t and t) to output the noise that was added. The loss is the mean squared error between true and predicted noise. It is a plain regression, with labels that we created ourselves.
3. Generation: start from pure noise and repeat: ask the network for the noise, remove a part of it, go to the previous step. After all steps, an image that was never in the training set appears.

Diffusion calls the network once per step (many times), while a GAN calls it once, so diffusion is usually slower to generate, but it trains stably and covers the variety of the data well.

Latent diffusion makes it cheaper: an autoencoder trained separately compresses the image into a small latent, all the diffusion happens there, and only at the end the decoder makes the full image. For an image of 256 × 256 × 3 = 196,608 numbers compressed to a latent of 32 × 32 × 4 = 4,096, the denoiser handles 48 times fewer values at each of its many steps. Text conditioning encodes the prompt with a text model and lets the denoiser read it by cross-attention (the image asks, the text holds keys and values).

Limits, bias, privacy, safety and cost

A model is a very good pattern machine, not a wise person. Knowing its limits is part of using it well.

What a model does not know

A model holds patterns from texts gathered up to a date, its knowledge cutoff. About what came after it knows nothing, yet it still writes fluent answers and may describe as current something that has already changed. The cure is to put recent information in the context (search, documents) instead of trusting its memory. And it gives no guarantee of truth: errors come with the same fluency as correct answers.

Bias

A model learns the statistical regularities of its data, without knowing which are fair. If it is trained on decades of hiring decisions that favoured one group, it will tend to rank new candidates the same way, and at scale it can amplify the unfairness. Just deleting the sensitive column is not enough, because other variables can stand in for it (a neighbourhood, a school). Measure results per group, then fix the data and the objective.

Privacy

Big models also memorise pieces of the training data, especially rare and repeated ones. If customer e-mails with names, addresses and card numbers are used for fine-tuning, a piece of that can come out in an answer to another user. Prevention happens before training: remove or anonymise personal data, collect less, and test the model for verbatim reproduction.

Size, quantisation and cost

The memory of the weights is the number of parameters times the bytes of each one: 4 bytes in float32, 2 in float16, 1 in int8. Quantisation stores weights with fewer bits (integers with a scale to convert back): smaller and often faster, at the price of a bit of accuracy, since each int8 weight has only 256 possible values. A model with 1 billion parameters takes 4 GB in float32 and 1 GB in int8 (counting only the weights, 1 GB = 10^9 bytes).

This is why APIs charge by token and why long prompts are slower and more expensive. The more a system can do alone (send messages, spend money, change files), the more its outputs need limits and review. A person stays responsible for decisions that affect people.

Putting it all together

Now you can follow a whole story. A text is cut into tokens (ids). Each id picks an embedding from a table. Position is added. Blocks of attention and feed-forward layers mix the vectors. A last linear layer and a softmax give a probability for the next token. During training, cross-entropy compares that distribution with the real token, backpropagation finds the gradients, and an optimiser such as Adam turns the knobs. At inference, decoding (temperature, top-k, top-p) picks the token and the loop repeats.

Common mistakes

• Judging a model on the data it trained on, or tuning on the test set.
• Preprocessing with statistics of the whole dataset before splitting (leakage).
• Forgetting to clear gradients, or forgetting that dropout and eval mode exist.
• Putting a softmax before a loss that already includes it.
• Believing that few-shot examples or a system prompt change the weights.
• Trusting fluent text as if it were true, or giving a tool too much power.

To go further, watch the 3Blue1Brown neural networks series, then build things yourself with Karpathy's Zero to Hero course.