Aggregations, Statistics and Random Numbers

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Aggregation (reduction)

An operation that collapses many numbers into fewer, like sum or mean.

Mean

The average: the sum divided by the count.

Standard deviation (std)

How spread out numbers are around their mean.

argmax / argmin

The position of the largest or smallest value.

np.where

A vector if/else: choose between two values per element, based on a condition.

Seed

A starting number for a random generator. The same seed gives the same sequence.

Generator (rng)

The object from np.random.default_rng that produces random numbers.

keepdims

Keep the reduced axis as size 1 so the result can broadcast back.

1. 1

   Rank the scores

   With s = np.array([72, 55, 90, 41]), print np.argsort(-s). What does the first number mean? (It is the position of the best score.)

2. 2

   Roll a die 6000 times

   Use rng.integers(1, 7, size=6000) and np.bincount on the result (np.bincount counts how many times each whole number appears). Each face should come up near 1000 times, but not exactly.

3. 3

   Same seed, same shuffle

   Call np.random.default_rng(1).permutation(5) twice in a row and compare. Then create one rng and call permutation(5) twice on it. Why do the two calls differ?

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

import numpy as np

a = np.random.default_rng().random(2)
b = np.random.default_rng().random(2)
print(np.array_equal(a, b))

1. 1

   The core aggregations are sum, mean, min, max, std and var. The mean is the usual average: add everything up, divide by how many there are. The standard deviation (std) says how spread out the numbers are around the mean. Picture two classes that both average 50 marks. In class A everyone scored 48 to 52, so the std is small. In class B half scored 10 and half scored 90, so the std is large. The variance (var) is the std squared. Each works on the whole array by default and returns one number, or along one axis, returning an array with that axis removed. They exist as methods (m.sum()) and as functions (np.sum(m)).

2. 2

   argmax and argmin return the position of the largest or smallest value, not the value. Along an axis they give one position per row or column. np.argsort(a) gives the positions that would sort a, which is how you rank items and how top-k search works in Topic 14.5.

3. 3

   np.where(condition, x, y) is a vector if: it picks x where the condition is true and y elsewhere. With only a condition it returns the positions where it is true. Use it to label data ("high" or "low") or to replace values without a loop.

4. 4

   keepdims=True keeps the reduced axis with size 1. That makes the result broadcast back against the original array, for example m - m.mean(axis=1, keepdims=True) centres every row. Without it you would need a reshape.

5. 5

   Random numbers: create a generator with rng = np.random.default_rng(seed), then call rng.random(3) (floats from 0 to 1), rng.integers(1, 7, size=5) (dice rolls, stop excluded), rng.normal(loc, scale, size) (numbers that cluster around loc and thin out symmetrically, the bell-curve distribution; scale is the std), rng.choice(items, size, replace=False) (note that the default is replace=True, so the same item can be drawn twice unless you say otherwise) and rng.permutation(n) (a shuffled 0..n-1). The same seed gives the same numbers on every run, which is how you make experiments reproducible. Keep the NumPy version fixed too: NumPy does not promise that a Generator produces the same stream in every release.

6. 6

   Prefer default_rng over the older np.random.seed(...) and np.random.rand global functions. A generator is an object you pass around, so two parts of a program cannot disturb each other's random streams. For machine learning, default_rng(seed) is how you shuffle data into train and test sets repeatably (see the exercises).

7. 7

   Statistics with a catch: np.std divides by n by default (it treats your numbers as the whole population), while pandas Series.std divides by n - 1 (it treats them as a sample drawn from a bigger group). They give different numbers for the same data. Know which one a library uses.

8. 8

   Recall from Topic 14.2: axis is the direction you work along, and the axis you name disappears from the result's shape. Everything below uses that rule.

• ↔

  Population versus sample statistics. ddof=0 and ddof=1 differ; for tiny samples the gap is large.

• ↔

  Reproducibility versus unpredictability. Seeds are ideal for tests and papers and wrong for security.

• ↔

  Simple aggregates hide structure. A mean can hide a distribution (the way values are spread out) with two humps, such as a class where half scored 10 and half scored 90. Look at a histogram (Topic 14.9) too.

• Done when you can use sum, mean, std, min and max with and without axis.

• Done when you can explain argmax and rank items with argsort.

• Done when you can write an np.where that labels or replaces values.

• Done when you can create a seeded generator and draw floats, integers, normals, choices and shuffles.

• Done when you can explain why a seed makes results repeatable but not secure.