PHASE 14Intermediate ~31 min· topic 6 of 10

Topic 14.6

pandas: Series and DataFrames

In one line

pandas gives Python tables: a DataFrame is a table with labelled columns and rows, and a Series is one column. You select with [], loc and iloc, filter with boolean masks and add columns by assignment.

Think of it like this

A school register. Each row is a student, each column is a fact (name, age, city). You can ask "show me the Pune students" or "add a column for next year's age". A DataFrame is that register inside Python, and a Series is one column of it.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

DataFrame: A table with labelled columns and rows.
Series: One labelled column of values with a single type.
Index: The labels of the rows.
Column: One named field in the table.
loc: Select rows and columns by label.
iloc: Select rows and columns by integer position.
Boolean filter: Keep only the rows where a condition is true.
object dtype: The type pandas uses for text, and for columns of mixed things.
describe: A summary of count, mean and spread for numeric columns.

Step by step

01A Series and a DataFrame

A Series is a labelled 1-D array: pd.Series([10, 20, 30], index=["a", "b", "c"]). Maths on it is vectorised, and s["b"] finds by label.

A DataFrame is a dictionary of Series that share an index. Printing it shows the index on the left, the column names on top and each column right-aligned.

main.pywhole filepython

import pandas as pd

df = pd.DataFrame({
    "name": ["Asha", "Ben", "Chen", "Dia"],
    "age": [23, 31, 27, 45],
    "city": ["Pune", "Delhi", "Pune", "Mumbai"],
    "salary": [52000.0, 71000.0, 64000.5, 90000.0],
})
print(df.shape)

A Series and a DataFramediagram

Rendering diagram…

02A first look at any table

head(2) shows two rows. count() gives non-null counts per column, so any column with fewer than the total has missing values. describe() summarises numeric columns only. value_counts() counts the values of a text column, sorted by frequency.

df.info() prints a text summary with column types and non-null counts and returns nothing. It prints to the output and is best run in a notebook or terminal on your own data.

03Pick columns, rows and cells

df["name"] is a Series; df[["name", "city"]] is a DataFrame. With the row labels set to r1 to r4, df.loc["r2", "name"] is Ben, while df.iloc[2]["name"] is the third row, Chen. loc["r1":"r2"] includes r2.

04Filter and add columns

df[df["city"] == "Pune"] keeps rows 1 and 3. A new column is one assignment: df["bonus"] = df["salary"] * 0.1. A comparison creates a column of True/False, which is handy for flags.

main.pywhole filepython

pune = df[df["city"] == "Pune"]
df["bonus"] = df["salary"] * 0.1
df["senior"] = df["age"] >= 30

05Assign through loc, not through a chain

sub = df[df["a"] > 1] makes a new DataFrame, so sub["b"] = 0 changes only that copy and pandas warns you. To change the original rows, use df.loc[df["a"] > 1, "b"] = 0: one operation, with the rows and column named together.

Try it yourself

1
Slice by label and position
With the r1 to r4 index, compare df.loc["r1":"r3"] with df.iloc[0:3]. Both give 3 rows but for different reasons. What about df.loc["r1":"r3"] versus df.iloc[0:2]?
2
Two conditions
List the names of people from Pune with a salary above 60000. Write it with & and brackets first, then try it without brackets and read the error.
3
Add a column
Add a column net equal to salary - bonus. Then print df.describe() and find the new column in it.

Code & diagrams

Build a DataFrame and look at it New tab

Add a fifth person and rerun. Print df.shape to confirm the row count. The same code works for a shop's orders, a list of students or a set of support tickets: one row per thing, one column per fact.

Expected output

   name  age   city   salary
0  Asha   23   Pune  52000.0
1   Ben   31  Delhi  71000.0
rows, columns: (4, 4)
columns: ['name', 'age', 'city', 'salary']
non-null counts: {'name': 4, 'age': 4, 'city': 4, 'salary': 4}
        age   salary
count   4.0      4.0
mean   31.5  69250.1
std     9.6  15903.3
min    23.0  52000.0
25%    26.0  61000.4
50%    29.0  67500.2
75%    34.5  75750.0
max    45.0  90000.0
city
Pune      2
Delhi     1
Mumbai    1
Name: count, dtype: int64

Select, filter and add columns New tab

Change Pune to Delhi in the filter. Then add a third condition with & and brackets.

Expected output

['Asha', 'Ben', 'Chen', 'Dia']
    name    city
r1  Asha    Pune
r2   Ben   Delhi
r3  Chen    Pune
r4   Dia  Mumbai
loc row: Ben
iloc row 2, name: Chen | first two cells: Chen 27
loc slice: [23, 31]
    name  age  city   salary
r1  Asha   23  Pune  52000.0
r3  Chen   27  Pune  64000.5
['Ben', 'Chen', 'Dia']
    name    bonus  senior
r1  Asha  5200.00   False
r2   Ben  7100.00    True
r3  Chen  6400.05   False
r4   Dia  9000.00    True

Assign safely with loc New tab

The first frame stays unchanged after the filtered copy is edited with a copy; the second uses loc.

Expected output

original after editing a copy:
   a  b
0  1  4
1  2  5
2  3  6
   a  b
0  1  4
1  2  0
2  3  0

Labels, not positions: index alignment and loc against iloc New tab

pandas lines values up by their index labels before doing maths. Missing partners give NaN, so a shop open in only one month vanishes from a plain +. add(..., fill_value=0) treats a missing side as 0. The last lines show why loc[0] fails on an index of 10, 20, 30 while iloc[0] works.

Expected output

feb + mar (aligned by label, not position):
Delhi     410.0
Goa         NaN
Mumbai    290.0
Pune        NaN
dtype: float64
with fill_value=0:
Delhi     410.0
Goa        50.0
Mumbai    290.0
Pune      120.0
dtype: float64
loc[10]: 5 | iloc[0]: 5
KeyError: 0

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Chained assignment on a filtered frame

Set a column on the result of a filter.

main.pywhole filepython

import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
sub = df[df["a"] > 1]
sub["b"] = 0
print(df)

terminal

$ python main.py

── what you'll see ──

main.py:5: SettingWithCopyWarning:

A value is trying to be set on a copy of a slice from a DataFrame.

Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

sub["b"] = 0

a b

0 1 4

1 2 5

2 3 6

Myth vs fact

Myth

A DataFrame is a spreadsheet.

Fact

It looks like one, but columns have strict types, operations apply to whole columns without loops, and it can hold millions of rows.

Myth

loc and iloc are interchangeable.

Fact

loc uses labels and includes the end of a slice; iloc uses positions and excludes it. They agree only when the index is the default 0, 1, 2.

Myth

Filtering changes the original table.

Fact

A filter returns a new DataFrame. Use loc to assign into the original.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Under the hood, a DataFrame stores columns in blocks of same-typed NumPy arrays. Text columns are arrays of Python objects, which are slow and memory-hungry; the category dtype and the dedicated string dtypes help.
▸
Prefer df.loc[mask, "col"] = value over chained indexing. Copy-on-write means a result behaves as if it were its own copy, so changing one DataFrame can never change another that was derived from it. It is the default in pandas 3 and an opt-in in pandas 2. Under it, chained assignment such as df[mask]["col"] = 0 never updates df, and the SettingWithCopyWarning of this course's pandas goes away. Write the loc form, which is correct under both behaviours.
▸
Index alignment. Arithmetic between two Series or DataFrames matches rows by index label, not by position, and a label found on only one side gives NaN. That is powerful (two tables in different row order still add correctly) and a classic surprise (two Series with different indexes add to mostly NaN). Use .add(other, fill_value=0) when a missing side should count as zero, or .to_numpy() when you really want positional maths. See the last runnable example.
▸
df["a"] returns a Series and df[["a"]] returns a one-column DataFrame, which matters when passing data to libraries that expect 2-D input. Reading df.iloc[2]["name"] first builds the whole row as a Series, which is slow and mixes dtypes; df.at[label, col] and df.iat[i, j] fetch a single cell directly. Row-by-row loops over a DataFrame (iterrows) are slow: prefer column maths, map or apply on a column.
▸
A DataFrame is great up to what fits in memory (roughly a few gigabytes on a laptop). For larger data, see Topic 14.10: Polars, DuckDB and Parquet.

Remember this

1
pandas is built on NumPy. Recall from Topic 14.1: a NumPy array has one dtype for all its elements. Each column of a DataFrame is a Series backed by such an array, but different columns may have different dtypes (text, integers, decimals, dates). That is the key difference from a NumPy array: a table of mixed types. Import with import pandas as pd. The browser has pandas 2.3.
2
Create a DataFrame from a dictionary (Topic 4.4) of columns: pd.DataFrame({"name": [...], "age": [...]}). Every list must have the same length. The leftmost numbers are the index: a label for each row, 0, 1, 2, ... by default, but it can be anything (names, dates). df.shape is (rows, columns), df.columns the column names and df.dtypes the type of each column. In the pandas of this course (2.x), text columns show as object; newer pandas versions can show a dedicated string type instead, so read dtypes rather than assuming.
3
To look at an unfamiliar table, use df.head(n) for the first rows, df.shape, df.dtypes, df.describe() (count, mean, std, min, quartiles, max for numeric columns), df["city"].value_counts() (how often each value appears) and df.info(), which prints column types and non-null counts ("non-null" means "not missing"). Recall from Topic 14.4: mean is the average and std says how spread out the numbers are. The quartiles (25%, 50%, 75%) are the values a quarter, half and three quarters of the way up when you line the numbers up in order. The 50% line is the median, the middle value. This tour should be the first thing you do with any new dataset.
4
Selecting: df["name"] gives a Series; df[["name", "city"]] (a list inside the brackets) gives a smaller DataFrame. **loc selects by label**: df.loc["r2", "name"], and its slices include the end label. **iloc selects by position** like a list: df.iloc[2, 1], and its slices exclude the end. Using the wrong one is a common mistake with a non-default index.
5
A boolean filter works like the NumPy masks of Topic 14.2: df[df["city"] == "Pune"] (the inner part is a column of True/False, one per row). Combine conditions with &, | and ~, each in brackets: df[(df["age"] > 25) & (df["salary"] > 60000)]. Every filter returns a new DataFrame; the original is untouched.
6
Add or change a column by assignment: df["bonus"] = df["salary"] * 0.1. Column arithmetic is vectorised, exactly like NumPy. To assign into a filtered subset, use a single loc step: df.loc[mask, "col"] = value. Assigning to a filtered copy (df[mask]["col"] = value) triggers a SettingWithCopyWarning and does not change df.
7
pandas is the standard tool for tabular data in machine learning: you load, inspect and clean a table here (Topic 14.7), then turn it into a NumPy array for a model. Knowing what the columns contain is half of any data job.

Explain it without notes

What is the difference between a Series and a DataFrame?

When do you use loc and when iloc? What is the difference in slice behaviour?

How do you filter rows by two conditions? What common mistakes appear?

Why does df[mask]["col"] = 0 not do what you want, and what is the fix?

Two Series with different index labels are added and most results are NaN. Why, and how do you get a sensible answer?

Practice

Make a DataFrame with columns item and price for three items and print the item with the highest price.

From the register example, print the average salary of people older than 25.

Add a column price_with_tax that is price times 1.2, and print the frame.

Give every person from Pune a raise of 5000 in place, with a single loc assignment, and print the frame. Start from city = Pune, Delhi, Pune and salary = 52000.0, 71000.0, 64000.5.

Trade-offs

↔
Convenience versus memory. pandas is quick to start with but keeps everything in memory, and object columns are heavy.
↔
Labels versus positions. Labels make code readable and robust to reordering; positions are faster to type but break when rows move.
↔
Method chaining versus step-by-step. Chains are compact (Topic 14.8) but harder to debug than named steps.

Done when you can

Done when you can build a DataFrame from a dictionary and read its shape, dtypes and describe().
Done when you can select columns, and rows by loc and iloc, and explain the slice difference.
Done when you can filter with two conditions using & and brackets.
Done when you can add a column and explain why chained assignment fails.
Done when you can say when pandas is the wrong tool (data too large for memory).

Command Palette

Words you'll meet

Step by step

01A Series and a DataFrame

02A first look at any table

03Pick columns, rows and cells

04Filter and add columns

05Assign through loc, not through a chain

Try it yourself

Slice by label and position

Two conditions

Add a column

Code & diagrams

Break it on purpose

Chained assignment on a filtered frame

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can