Huge Repositories: Shallow, Partial & Sparse Checkouts

In one line

Monorepos with millions of files and years of history need different clone strategies: shallow clones for CI, partial (blobless) clones for developers, sparse checkout to see only your part, and Scalar to set it all up.

0/4 · 0%

Think of it like this

A national library. You don't carry home every book ever printed. You might take only this year's editions (shallow), take the catalogue and fetch books on demand (partial clone), or only visit the floors for your subject (sparse checkout).

Key ideas

01
SHALLOW CLONE: git clone --depth 1 fetches only recent history. Great for CI builds that just need the latest code. Limits: no full git log, blame, or merge-base computations without deepening (git fetch --unshallow / --deepen).
02
PARTIAL CLONE: git clone --filter=blob:none downloads all commits and trees but fetches file contents (blobs) only when needed (on checkout or diff). Full history for log and blame, but a much faster clone. --filter=tree:0 goes further (treeless), which is best for CI-like, build-once uses.
03
SPARSE CHECKOUT: git sparse-checkout set services/checkout libs/common populates only those directories in the working tree, so your editor, builds, and git status see a small slice of a huge monorepo. Cone mode (directory-based) is fast; combine it with partial clone for the best experience.
04
SCALAR (shipped with Git) configures large-repo best practices in one command: partial clone, sparse checkout, the file-system monitor (fsmonitor) for fast git status, commit-graph, and background maintenance. Monorepos also need build tooling that only builds what changed (CI/CD course, CI at scale).

Code & diagrams

clone strategiesbash

git clone --depth 1 https://github.com/org/mono.git                  # CI: latest snapshot only
git clone --filter=blob:none https://github.com/org/mono.git         # dev: full history, lazy file contents
cd mono && git sparse-checkout set --cone services/checkout libs/common
git config core.fsmonitor true                                        # fast git status
scalar clone https://github.com/org/mono.git                          # all of the above, preconfigured

Explain it without notes

Shallow clone vs partial clone: which would you use for CI and which for developers?

Practice

Your CI computes 'files changed since main' but fails on shallow clones. Fix it.

Trade-offs

↔
Monorepos simplify atomic cross-project changes and shared tooling but need scale techniques (partial/sparse clones, affected builds); polyrepos scale naturally but make cross-repo changes and dependency updates harder.

Done when you can

I can choose between shallow, partial, and sparse strategies
I know when a monorepo needs Scalar-style setup