Topic 16.13
Tensors and PyTorch: A First Look
In one line
A tensor is a NumPy-like array that can run on a GPU and remember how it was computed, so PyTorch can find gradients automatically. Training a neural network is five repeated steps: forward pass, loss, gradients, update, repeat. You can run the whole idea with plain NumPy first.
Think of it like this
Tuning a radio dial by ear. You turn the knob a little, listen, and decide which way to turn next. A neural network has millions of knobs. Training turns each knob a tiny bit in the direction that makes the sound (the error) better, and repeats very many times. The "which way to turn" is the gradient.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Tensor
- A grid of numbers with any number of dimensions. PyTorch's version of an array.
- GPU
- A graphics card that does huge amounts of simple arithmetic at once.
- Gradient
- For each adjustable number, which direction and how strongly changing it changes the loss.
- Autograd
- Automatic computation of gradients by recording the operations.
- Loss
- A number measuring how wrong the predictions are. Training tries to make it small.
- Learning rate
- How big each update step is.
- Epoch / batch
- One pass over all the data / a small group of examples processed together.
- Fine-tuning
- Training an already-trained model a little more on your own data.
Step by step
01Training is a loop: see it in NumPy
Real PyTorch cannot run in this page, so the idea runs as a tiny NumPy version. It is the same loop, only with the gradients worked out by hand. The first example fits y = w·x + b to data generated by y = 2x + 1. It starts with w = 0 and b = 0. Each of 200 steps does the five parts: predict, measure the squared error, compute gradients (done by hand here), update, repeat. The printed loss falls from 33 to almost 0 while w approaches 2 and b approaches 1.
02What autograd replaces
In the first example the gradient formulas were derived by hand. For a network with millions of parameters that is impossible, which is why autograd exists. The second example shows the idea of a gradient with a simple loss: the exact (analytic) value matches the numeric one. At w = 2 the gradient is 0: the minimum.
03Tensors in PyTorch
Not runnable here (PyTorch does not run in the browser). On your machine, after installing PyTorch:
import torch
x = torch.tensor([[1.0, 2.0, 3.0],
[4.0, 5.0, 6.0]]) # shape (2, 3)
print(x.shape, x.dtype) # torch.Size([2, 3]) torch.float32
device = "cuda" if torch.cuda.is_available() else "cpu"
x = x.to(device) # move to the GPU if there is one
y = x @ x.T # matrix multiply, shape (2, 2)04Autograd in one example
Mark a tensor with requires_grad=True. After a computation, call .backward() on the loss and read .grad.
w = torch.tensor(3.0, requires_grad=True)
loss = (3 * w - 6) ** 2
loss.backward()
print(w.grad) # tensor(18.) same as 2 * (3*3 - 6) * 305The training loop in PyTorch
The five steps map to five lines. optimizer.zero_grad() clears gradients from the previous step, because PyTorch adds them up.
model = torch.nn.Linear(1, 1) # y = w*x + b
optimizer = torch.optim.SGD(model.parameters(), lr=0.05)
loss_fn = torch.nn.MSELoss()
for epoch in range(200):
optimizer.zero_grad()
prediction = model(x_batch) # 1. forward pass, x_batch shape (N, 1)
loss = loss_fn(prediction, y_batch) # 2. loss
loss.backward() # 3. gradients
optimizer.step() # 4. update
# 5. repeat06Loading a Hugging Face model
Names below are placeholders: choose a model from the Hub for your task, and check its licence and size first. A model reads numbers, not letters, so the tokenizer (Topic 16.4) turns your text into token IDs, which are the numbers input_ids. The logits are the model's raw scores, one per possible class.
from transformers import AutoTokenizer, AutoModelForSequenceClassification
name = "your-model-name-from-the-hub"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name)
inputs = tokenizer("I liked this film", return_tensors="pt") # tensors: input_ids, attention_mask
outputs = model(**inputs)
print(outputs.logits.shape) # (1, number_of_classes): one row per input textTry it yourself
- 1
Too big a step
In the first example set
learning_rate = 0.3. What happens to the printed loss? Explain why in a sentence. - 2
Change the rule
Change the data rule to
y = -3x + 5. Dowandbstill converge? What do they converge to?
Code & diagrams
Change learning_rate to 0.2, then to 0.5, and watch what happens to the loss. Too large a step overshoots.
Expected output
step 1: loss 33.0000 w 1.400 b 0.500
step 10: loss 0.0137 w 2.069 b 0.804
step 50: loss 0.0012 w 2.020 b 0.942
step 200: loss 0.0000 w 2.000 b 0.999Expected output
w=0.0: analytic -36.000 numeric -36.000
w=1.0: analytic -18.000 numeric -18.000
w=2.0: analytic 0.000 numeric 0.000
w=3.5: analytic 27.000 numeric 27.000Expected output
input shape: (4, 3)
hidden shape: (4, 5)
output shape: (4, 2)
hidden units that stayed at 0: 12Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
A batch with the wrong number of features
Pass a batch with 2 features into a layer built for 3.
import numpy as np
W = np.ones((3, 5))
batch = np.ones((4, 2)) # 2 features, but the layer expects 3
print((batch @ W).shape)Myth vs fact
Myth
Deep learning is magic that nobody understands.
Fact
The mechanics are matrix multiplications, simple non-linear functions and gradient descent, as in these examples. What is hard to explain is why huge models behave as they do, not how they are trained.
Myth
You need a GPU to learn PyTorch.
Fact
Small models and tensors run fine on a CPU. A GPU matters for large models and large data.
Myth
You must train models yourself to work in AI.
Fact
Most applied work loads or calls existing models. Training from scratch is rare and costly.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
PyTorch tensors default to 32-bit floats (
float32), while NumPy defaults to 64-bit (float64). Converting between them copies or reinterprets, and mixed dtypes cause subtle bugs or errors. - ▸
Gradients are stored on the parameter tensors and accumulate across
backward()calls, which is why loops callzero_grad(). Wrap evaluation intorch.no_grad()to save memory and time. - ▸
Reproducibility needs seeds for Python, NumPy and PyTorch, and note that some GPU operations are non-deterministic by default. Record library versions and hardware.
- ▸
Switch modes for inference:
model.eval()turns off training-only behaviour such as dropout and uses stored statistics in batch normalisation, andtorch.no_grad()(ortorch.inference_mode()) stops gradient tracking. Forgettingeval()gives noisy, wrong predictions with no error. The model and its input tensors must also be on the same device and have compatible dtypes, otherwise PyTorch raises a runtime error. Call.item()to take a Python number from a one-element tensor, and.detach()before turning a tensor that tracks gradients into NumPy. - ▸
Gradient descent step size and batches interact: using mini-batches gives a noisy but cheap estimate of the gradient, which is why loss curves are jagged. Optimisers like Adam keep running averages per parameter and are the usual default for neural networks, while plain SGD is what the first example uses. Watch the validation loss as well as the training loss: training loss falling while validation loss rises is overfitting.
Remember this
- 1
A tensor is a grid of numbers with any number of dimensions: a single number, a list (1-D), a table (2-D), a stack of images (4-D). PyTorch tensors look like NumPy arrays (shape, dtype, slicing, broadcasting) and convert to and from them. Two things make them different: they can live on a GPU (a device with thousands of small processors, ideal for matrix arithmetic), and they can track operations to compute gradients.
- 2
A neural network layer is, at its heart, what you saw in Topic 14.5: a matrix multiplication plus a bias (a number added to each result), followed by a simple bending step called a non-linear function, such as ReLU (replace negatives with zero). The matrix of adjustable numbers is the layer's weights; together with the biases they are the network's parameters. The third example shows shapes: a batch of 4 examples with 3 features times a 3-by-5 weight matrix gives (4, 5); another layer takes it to (4, 2). Most beginner bugs in deep learning are shape mismatches, so read shapes first.
- 3
Autograd is automatic differentiation. You write the forward computation, and the library records it and computes the gradient of the loss for every parameter. (It uses the chain rule from calculus: if a result depends on a step that depends on another step, you multiply the slopes along the way. You do not need to do this yourself.) The second example compares an exact gradient from calculus with a numeric one found by nudging the input: they agree. Autograd produces the exact kind, for any computation you write, without you doing the calculus.
- 4
A training loop has five steps. (1) Forward pass: run the inputs through the network to get predictions. (2) Loss: a number saying how wrong they are. (3) Backward pass: compute the gradient of the loss for each parameter. (4) Update: move each parameter a small step against its gradient (gradient descent, scaled by a learning rate). (5) Repeat over many batches (small groups of examples) and epochs (full passes over the data). The first example performs exactly these steps for a straight line using only NumPy.
- 5
Choose the learning rate carefully. Too small and training crawls. Too large and the loss bounces or explodes. Real training also watches the loss on held-out data (Topic 16.12) to catch overfitting.
- 6
Hugging Face hosts thousands of trained models. With the
transformerslibrary you can load a model and its tokenizer by name, and run text through it. Models are large: check the size, memory and licence before downloading, and run them on a GPU if you can. You will rarely train from zero; you usually load, use, or lightly adapt (fine-tune) an existing model. A full Hectal course on deep learning is planned.
Explain it without notes
What are the five steps of a training loop?
What is a tensor, and how does it differ from a NumPy array?
What does autograd do, and why do we need it?
Your PyTorch model gives different outputs for the same input on repeated calls, and memory grows during evaluation. What are the likely causes and fixes?
Practice
Write mse(y_true, y_pred) using NumPy.
Write gradient_step(w, grad, learning_rate) and use it three times to minimise w squared from w = 10.
Write linear_forward(batch, W, b) and check the output shape for a batch of 4 with 3 features and 2 outputs.
Trade-offs
- ↔
Train your own versus use a pretrained model. Training gives control and needs data, compute and time. A pretrained model is quick and strong, but you inherit its limits, size and licence.
- ↔
CPU versus GPU. A CPU is available everywhere and fine for small jobs. A GPU speeds large matrix work enormously and adds cost and setup.
Done when you can
Done when you can describe a tensor and read its shape.
Done when you can list the five steps of a training loop.
Done when you can explain a gradient and what autograd saves you from doing.
Done when you can trace shapes through two layers by hand.
Done when you know a full Hectal course on deep learning is planned.