PHASE 14Intermediate ~26 min· topic 10 of 10

Topic 14.10

The Wider Data Ecosystem

In one line

NumPy and pandas are the base, but real data work also uses Polars and DuckDB for big tables, Apache Arrow and Parquet for fast storage, and Jupyter notebooks for exploration. Underneath PyTorch and scikit-learn, NumPy-style arrays are still the language.

Think of it like this

A kitchen. A home cook with one good knife (pandas) makes most meals. A restaurant adds a big oven (DuckDB), a freezer for stock (Parquet files) and a conveyor between stations (Arrow). The skills are the same, the tools scale. Knowing what exists tells you when your laptop-sized tool is no longer enough.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Polars: A fast DataFrame library written in Rust that uses all CPU cores.
DuckDB: A database that runs inside your program and queries files with SQL.
Columnar: Storing all values of one column together, so a query reads only the columns it needs.
Parquet: A compressed, typed, columnar file format for tables.
Apache Arrow: A standard way to lay out columns in memory so tools can share data without copying.
Jupyter notebook: A document of code cells, output and notes you run step by step.
Lazy evaluation: Describe the work first and run it later, so the library can plan the fastest way.
Chunk: A piece of a big file that you process one at a time.

Step by step

01Rows versus columns

Row layout stores one record after another: city and temperature of record 1, then record 2. Column layout stores all cities together and all temperatures together. To average the temperature, a column layout reads only the temperature list; a row layout has to step over every city name too.

The example builds both layouts in plain Python so you can see the idea.

Rows versus columnsdiagram

Rendering diagram…

02What CSV loses

A DataFrame with a date column is written to CSV text, and read back. The date has become object (plain text) and must be parsed again with parse_dates. CSV stores no types. Parquet stores the schema (the list of column names and types) with the data, so the date would still be a date when you read it.

03Parquet and DuckDB in practice

On your own computer, a typical flow is: convert a big CSV to Parquet once, then query it with SQL. The snippet is not run here (the browser has neither DuckDB nor a Parquet writer).

main.pywhole filepython

import duckdb

con = duckdb.connect()
rows = con.sql("""
    SELECT city, avg(temp) AS avg_temp
    FROM 'weather.parquet'
    GROUP BY city
    ORDER BY avg_temp DESC
""").df()
print(rows)

04Notebook to module

Explore in the notebook. When a cell works, move its logic into a function in a .py file, add a test and import it back. The notebook becomes a thin report, and the function is reusable and testable.

Notebook to modulediagram

Rendering diagram…

05A map of the ecosystem

Small data: NumPy and pandas. Bigger single-machine data: Polars, DuckDB, Parquet. Plots: Matplotlib. Models: scikit-learn and PyTorch, which take NumPy-like arrays. Clusters: Spark and warehouses. The links between them are arrays, Arrow and Parquet.

A map of the ecosystemdiagram

Rendering diagram…

Try it yourself

1
Compare file sizes
On your computer, save a DataFrame of a million random rows with to_csv and to_parquet (install pyarrow), then compare the file sizes and the reading times. Which is smaller and faster to read?
2
Read only some columns
With the CSV round trip example, add usecols=["city", "temp"] to read_csv and print back.columns. The unused column is never kept.
3
Match the tool to the job
Choose a tool for each: 5,000 rows to chart; a 40 GB log file to summarise by hour; a trained model's weights; a SQL-friendly question over Parquet files.

Code & diagrams

Row layout against column layout New tab

Add a fourth record and print columns again. Notice that the column layout needs one list append per field.

Expected output

row layout   : {'city': 'Pune', 'temp': 31.5}
column layout: {'city': ['Pune', 'Delhi', 'Pune'], 'temp': [31.5, 18.0, 29.5]}
mean temp reads one column only: 26.333333333333332

CSV forgets types New tab

A typed format such as Parquet keeps the datetime column on the way back. CSV does not.

Expected output

{'day': 'datetime64[ns]', 'city': 'object', 'temp': 'float64'}
day,city,temp
2024-01-01,Pune,31.5
2024-01-02,Delhi,18.0
{'day': 'object', 'city': 'object', 'temp': 'float64'}
{'day': 'datetime64[ns]', 'city': 'object', 'temp': 'float64'}

Query a Parquet file with SQL (not run in the browser)python

Needs duckdb and a Parquet engine such as pyarrow installed (pip install duckdb pyarrow). It prints a two-row table with one average per city.

import duckdb
import pandas as pd

df = pd.DataFrame({"city": ["Pune", "Delhi", "Pune"], "temp": [31.5, 18.0, 29.5]})
df.to_parquet("weather.parquet")

print(duckdb.sql("SELECT city, avg(temp) AS avg_temp FROM 'weather.parquet' GROUP BY city ORDER BY city").df())

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Trusting the CSV round trip

Write a datetime column to CSV and read it back without parse_dates.

main.pywhole filepython

import io
import pandas as pd

df = pd.DataFrame({"day": pd.to_datetime(["2024-01-01"])})
back = pd.read_csv(io.StringIO(df.to_csv(index=False)))
print(back["day"].dt.year)

terminal

$ python main.py

── what you'll see ──

AttributeError: Can only use .dt accessor with datetimelike values. Did you mean: 'at'?

Myth vs fact

Myth

Polars or DuckDB replace pandas.

Fact

They overlap but each has strengths. pandas has the largest ecosystem; most libraries accept it. Many teams use pandas for small data and the others for big data.

Myth

Notebooks are bad.

Fact

They are excellent for exploration and teaching. The problem is using them as production code.

Myth

Big data needs a cluster.

Fact

Many datasets that sound big fit in one machine's memory or on its disk when stored as Parquet and queried with DuckDB or Polars.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Columnar formats compress well because values in one column are alike, and Parquet also stores per-chunk statistics (min and max), letting a query skip chunks entirely. This is why a filtered query on Parquet can read a tiny part of the file.
▸
Arrow-backed data lets libraries exchange tables without copying or serialising. pandas can use Arrow-based dtypes, and DuckDB and Polars read Arrow directly. It is a quiet reason the ecosystem works together.
▸
Pin versions of data libraries in your project (Phase 6 covered environments and packages). Their APIs change between major versions, and numerical results can differ slightly across versions and platforms.
▸
Parquet files are written once and read many times: you add data by writing new files (often one folder per day or per month, a layout called partitioning), not by editing rows inside a file. A query can then skip whole folders and read only the needed columns. Reading and writing Parquet from pandas needs an engine package such as pyarrow to be installed.
▸
NumPy and PyTorch share memory on the CPU: torch.from_numpy(arr) makes a tensor that looks at the same bytes, so changing one changes the other. A tensor on a GPU must be copied back with .cpu().numpy(). This is the same view-versus-copy idea as in Topic 14.2. Likewise df.to_numpy() may return a view or a copy depending on the dtypes, so do not rely on either.
▸
Choose a tool by where the data lives and how it is queried: a lazy engine (Polars lazy mode, DuckDB) plan the whole query and push filters and column selection down into the Parquet reader, so they skip data automatically. With pandas you must ask for that yourself (columns= and filters= in read_parquet) or it loads first and filters after. Measure on your own data before moving a working pipeline.

Remember this

1
Recall from Topic 14.6 and Topic 14.8: a DataFrame is a table, and groupby and merge summarise and join tables. The tools below do the same jobs at bigger scale.
2
pandas limits. pandas keeps the whole table in memory and runs most operations on one CPU core (one worker inside the processor). That is fine for millions of rows, and painful at tens of gigabytes. Text columns cost a lot of memory. The usual first steps are cheap: read only the columns you need, use smaller dtypes, process in chunks, or store the data in a better format than CSV.
3
Polars is a DataFrame library written in Rust. It uses all CPU cores and can run in a lazy mode, where you describe the query first and Polars plans it before running. Its API is different from pandas (expressions instead of mutating columns), so it is a new skill rather than a drop-in. DuckDB is an embedded analytical database: it runs inside your Python process, queries CSV and Parquet files with plain SQL, and can query a pandas DataFrame by name. If you know SQL (see the Database course at /database-design), DuckDB is the fastest way to apply it to files.
4
CSV is a text format. It has no types (dates and numbers are guessed), is slow to parse and large. The round-trip example shows a date column coming back as text. Parquet is a binary, columnar file format: values of one column are stored together, with types and compression. A query that needs two columns reads only those two. Parquet is the standard for analytics and machine-learning datasets.
5
Apache Arrow is a standard in-memory columnar layout. Many tools (pandas, Polars, DuckDB, Spark, Parquet readers) can share Arrow data without copying or converting, which makes passing tables between them cheap. You rarely use Arrow directly, but it is why the tools fit together.
6
Jupyter notebooks mix code, output, charts and notes in cells. They are excellent for exploring data and explaining results. They are weak for production: hidden state from running cells out of order, hard code review, hard testing. The professional pattern is to explore in a notebook, then move working logic into functions in .py files with tests (Phase 11) and import them back.
7
Where NumPy sits. scikit-learn takes NumPy arrays or DataFrames and returns NumPy arrays. PyTorch tensors (its name for n-dimensional arrays, Topic 14.1) have almost the same shape rules, slicing and broadcasting as NumPy arrays, and convert to and from them cheaply. Everything you learned about shape, dtype and broadcasting in this phase transfers directly. Embeddings from LLM APIs arrive as lists of floats that you turn into arrays (Topic 16.8).
8
Choosing: a few million rows and a laptop means pandas. Larger files or SQL-friendly questions mean DuckDB or Polars. Data that lives on a cluster (many computers working together) means Spark or a cloud warehouse. Pick the simplest tool that works, and keep your data in Parquet so you can switch tools later.

Explain it without notes

What are the limits of pandas, and what would you try first when a file is too big?

Why is Parquet better than CSV for analytics?

What are Polars and DuckDB, and when would you choose each?

What is the right role for a Jupyter notebook in a professional project?

Practice

Make the column layout of the rows [{"a": 1, "b": 2}, {"a": 3, "b": 4}] and print it.

Write the SQL (as text) that gives the number of rows per city from a table weather.

List which tools you would use for a 30 GB set of Parquet files that you want to summarise with SQL.

Trade-offs

↔
Familiarity versus scale. pandas is known by everyone; Polars and DuckDB are faster on big data but are another API or language (SQL) to learn.
↔
Exploration versus reproducibility. Notebooks are fast to try things in; scripts and tests are reliable to run again.
↔
Text versus binary formats. CSV opens in any editor; Parquet is smaller, typed and faster but not human readable.

Done when you can

Done when you can explain row versus columnar storage and why analytics prefer columns.
Done when you can say what CSV loses and what Parquet keeps.
Done when you can name Polars, DuckDB and Arrow and say what each is for.
Done when you can describe a notebook-to-module workflow.
Done when you can say where NumPy arrays appear inside scikit-learn and PyTorch.

Command Palette

Words you'll meet

Step by step

01Rows versus columns

02What CSV loses

03Parquet and DuckDB in practice

04Notebook to module

05A map of the ecosystem

Try it yourself

Compare file sizes

Read only some columns

Match the tool to the job

Code & diagrams

Break it on purpose

Trusting the CSV round trip

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can