Command Palette

Search for a command to run...

Hectal
Phase 14Intermediate15 of 18 in Python

Working with Data: NumPy and pandas

Arrays, vectors, broadcasting, tables and charts: the two libraries under almost all data science and machine learning, and the maths behind embeddings.

What this phase relies on. Lists, tuples and dictionaries (Phase 4), functions and arguments (Phase 5), import and installing packages (Topics 6.1 and 6.5), and reading error messages (Topic 7.1). Topic 14.8 also uses dates (Topic 7.11) and CSV text (Topic 7.7). If you skipped those phases, go back to them first, or read each "Recall" line in this phase and follow its topic number. You do not need any maths beyond school arithmetic: every maths word (vector, matrix, dimension, axis, mean, standard deviation, dot product) is explained in everyday terms where it first appears.

Plain Python lists are flexible but slow for numbers. NumPy gives Python a fast, fixed-type array, and nearly every AI library (scikit-learn, PyTorch, the tools that store embeddings) is built on its ideas. This phase teaches arrays from the first line: shape, indexing, slices that share memory, boolean masks, broadcasting and random numbers you can repeat.

Then it turns to the maths that AI actually runs on: vectors, dot products, cosine similarity and matrix multiplication. You will build a small similarity search over embeddings and see a neural-network layer as one line, X @ W + b. Topic 16.8 builds on this base.

The second half is pandas, the library for tables. You will load a CSV, clean missing values, duplicates, text and dates, group, join and reshape data, and meet the wider ecosystem (Matplotlib, Polars, DuckDB, Arrow, Parquet). The browser here runs NumPy 2.2 and pandas 2.3. Every runnable example is repeatable: random numbers come from a seeded generator and tables are built from literal text.

0/10 · 0%
10 topics ~5 h 33 code blocks & diagrams
Start with the first topic
1
14.1

NumPy Arrays

A NumPy array (ndarray) is a grid of numbers that all share one type, stored side by side in memory. Because of that, maths on a whole array runs in fast compiled code instead of a slow Python loop.

29 min 1 diagram 2 code practice

2
14.2

Indexing, Slicing and Reshaping Arrays

You pick elements out of an array with indices, slices, boolean masks and lists of positions. Slices are views that share memory with the original, and reshape changes the shape without changing the numbers.

31 min 2 code practice

3
14.3

Vectorisation and Broadcasting

Vectorisation means writing one array expression instead of a Python loop. Broadcasting is the rule that lets arrays of different shapes combine, by stretching size-1 dimensions to match.

26 min 1 diagram 3 code practice

4
14.4

Aggregations, Statistics and Random Numbers

Aggregations like sum, mean, std and argmax collapse an array along an axis into fewer numbers. A seeded random generator, np.random.default_rng(seed), gives random-looking data that you can repeat exactly.

30 min 3 code practice

5
14.5

Vectors and Matrices: The Maths AI Runs On

A vector is a list of numbers that can describe anything from a point on a map to the meaning of a sentence. The dot product, norm, cosine similarity and matrix multiplication (@) are the four operations behind embeddings, search and every neural-network layer.

34 min 4 code practice

6
14.6

pandas: Series and DataFrames

pandas gives Python tables: a DataFrame is a table with labelled columns and rows, and a Series is one column. You select with [], loc and iloc, filter with boolean masks and add columns by assignment.

31 min 4 code practice

7
14.7

Loading and Cleaning Data

Real data arrives messy: missing values, duplicates, stray spaces, text where numbers should be and dates stored as strings. read_csv loads it, and a handful of pandas methods clean it before analysis or model training.

30 min 3 code practice

8
14.8

Grouping, Joining and Reshaping

groupby splits a table into groups and summarises each, merge joins two tables on a shared key like a SQL join, concat stacks tables, and pivot_table turns rows into a grid. Together they answer most business questions about a table.

31 min 4 code practice

9
14.9

Charts with Matplotlib

Matplotlib draws charts from NumPy and pandas data: line, bar, scatter and histogram, with titles and labels, saved as a PNG file. A chart answers questions that a table of numbers hides.

24 min 3 code practice

10
14.10

The Wider Data Ecosystem

NumPy and pandas are the base, but real data work also uses Polars and DuckDB for big tables, Apache Arrow and Parquet for fast storage, and Jupyter notebooks for exploration. Underneath PyTorch and scikit-learn, NumPy-style arrays are still the language.

26 min 3 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Turn a vector into a unit vector

Write a function normalise(v) that takes a NumPy vector (a 1-D array of numbers) and returns a new vector that points the same way but has length 1. To do that, divide the vector by its length. The length of a vector is its norm, which np.linalg.norm(v) gives you (Topic 14.5).

Beginner
2

Scale each column with broadcasting

Write scale_columns(m, factors). m is a 2-D array (a table of numbers) and factors is a 1-D array with one number per column. Return a new table where column 0 is multiplied by factors[0], column 1 by factors[1], and so on. Use no loop: broadcasting (Topic 14.3) does it.

Beginner
3

Find the most similar items

Write top_k(embeddings, query, k). embeddings is a 2-D array with one item per row (each row is that item's embedding, a list of numbers describing it). query is a 1-D vector. Return a plain Python list of the row numbers of the k rows most similar to the query, best first. For the example below it should print [0, 1, 4].

Intermediate
4

Summarise sales with groupby

Write city_totals(df). df is a pandas table (DataFrame) with columns city, qty (how many sold) and price (price of one). Each row's revenue is qty * price. Return a pandas Series with one number per city: that city's total revenue over all its rows, sorted from highest to lowest. For the example below the cities come out in the order Pune, Delhi, Mumbai.

Intermediate
5

Clean a small CSV with pandas

Write clean(text) that takes CSV text with columns name,age,city and returns a cleaned DataFrame (the starter already loads the text for you). Do these four steps in order:

Intermediate
6

Split data into train and test sets by hand

In machine learning you keep some rows back to test the model (Topic 14.7). Write split(X, y, test_fraction, seed) where X is a 2-D array (one example per row) and y a 1-D array with one label (the right answer) per row of X. Shuffle the row numbers with np.random.default_rng(seed).permutation(len(X)). Put the first round(len(X) * test_fraction) shuffled row numbers in the test set and the rest in the train set.

Intermediate