Memory Internals: Encodings, Overhead, and Fragmentation

In one line

Redis memory is your data plus per-key overhead, encoding choices, allocator fragmentation, buffers and the fork's copy-on-write. Knowing these explains why used_memory and RSS differ, and how to shrink a dataset by 2–10×.

0/4 · 0%

Think of it like this

A storage unit. The boxes you packed are your data, but the unit also holds the shelves, labels, gaps between odd-sized boxes, and empty space left behind when you took boxes out. The rent is for the whole unit (RSS), not just the boxes (used_memory).

Key ideas

01
Every key costs overhead beyond its bytes: a dictEntry in the main hash table, a redisObject header (16 bytes), the key string (an SDS with a small header), and an expiry entry if it has a TTL. That's roughly 50–90 bytes per key before the value. With 100M small keys, overhead can be larger than the data.
02
Encodings adapt to size: strings as int, embstr, raw; hashes, small sets and sorted sets as listpack below thresholds; integer sets as intset; lists as quicklist (linked listpacks); big sets and hashes as hash tables; big sorted sets as skiplist + hash table. OBJECT ENCODING key shows it; MEMORY USAGE key SAMPLES 0 measures it exactly.
03
INFO memory fields to know: used_memory (what Redis's allocator gave out for data and buffers), used_memory_rss (what the OS sees), used_memory_peak, used_memory_dataset, used_memory_overhead, mem_fragmentation_ratio (≈ RSS / used_memory), allocator_frag_ratio, mem_clients_normal, mem_replication_backlog, and used_memory_lua/used_memory_scripts.
04
Fragmentation: Redis uses jemalloc, which allocates in size classes. After lots of writes and deletes of varying sizes, freed memory is scattered in pages the allocator can't return to the OS, so RSS stays high. A ratio of 1.0–1.5 is normal; much higher means fragmentation; below 1.0 means the OS swapped Redis memory out (very bad for latency). activedefrag yes moves allocations in the background to compact memory, at some CPU cost.
05
Fork and copy-on-write: BGSAVE and AOF rewrite fork the process. Parent and child share pages until the parent writes to one, which then gets copied. With heavy writes during a snapshot, memory can temporarily grow by up to the size of the dataset. That's why maxmemory should leave headroom, vm.overcommit_memory = 1 is recommended, and transparent huge pages should be disabled (a 2 MB page copied for a 1-byte write).
06
MEMORY DOCTOR gives plain-English advice; MEMORY STATS gives a breakdown; redis-cli --memkeys and --bigkeys find the biggest keys by memory and by element count.

Code & diagrams

info-memory.redisredis

127.0.0.1:6379> INFO memory
used_memory:8589934592
used_memory_human:8.00G
used_memory_rss:15032385536
used_memory_rss_human:14.00G
used_memory_peak_human:11.20G
used_memory_dataset:7730941132
used_memory_overhead:858993460
mem_fragmentation_ratio:1.75
allocator_frag_ratio:1.68
mem_clients_normal:1200000
mem_replication_backlog:1048576
maxmemory_human:12.00G
maxmemory_policy:allkeys-lru
127.0.0.1:6379> MEMORY DOCTOR
High allocator fragmentation: This instance has an allocator external fragmentation greater than 1.1 ...
You may want to enable active defragmentation: CONFIG SET activedefrag yes

encodings.redisredis

127.0.0.1:6379> SADD ids 1 2 3
(integer) 3
127.0.0.1:6379> OBJECT ENCODING ids
"intset"
127.0.0.1:6379> SADD ids "abc"
(integer) 1
127.0.0.1:6379> OBJECT ENCODING ids
"listpack"
127.0.0.1:6379> MEMORY USAGE ids SAMPLES 0
(integer) 72
127.0.0.1:6379> CONFIG GET *listpack*
 1) "hash-max-listpack-entries"   2) "128"
 3) "hash-max-listpack-value"     4) "64"
 5) "set-max-listpack-entries"    6) "128"
 7) "zset-max-listpack-entries"   8) "128"
 9) "list-max-listpack-size"     10) "-2"

memory-anatomy.mermaiddiagram

Rendering diagram…

Interview problem

The problem

used_memory = 8 GB, RSS = 14 GB. Why?

Redis reports used_memory of 8 GB but the process RSS is 14 GB, and the container is close to its 16 GB limit. Explain the possible causes and what you'd do.

You're given

Workload: session writes and deletes all day
Nightly BGSAVE
maxmemory 12 GB

The interviewer follows up

What does a fragmentation ratio below 1.0 mean?

When it breaks

Transparent huge pages enabled on the Redis host

What you see

Fork copy-on-write copies 2 MB pages instead of 4 KB, so memory jumps during BGSAVE and latency spikes appear; Redis logs a THP warning at startup.

Fix & prevent

echo never > /sys/kernel/mm/transparent_hugepage/enabled (persist via your OS config); Redis also tries to disable THP for its own process (disable-thp yes).

Container limit equals maxmemory

What you see

Fragmentation, buffers and fork copy-on-write push RSS over the limit; the kernel OOM-kills Redis during a snapshot.

Fix & prevent

Set maxmemory to 60–80% of the container limit depending on write rate and persistence; alert on RSS, not just used_memory.

Explain it without notes

Explain what mem_fragmentation_ratio measures and what high and low values mean.

Why does forking for persistence need extra memory headroom?

Practice

Store 1M keys as user:{id} → email strings, measure memory, then store the same data bucketed into hashes of 100 fields and compare.

Run MEMORY STATS and identify the three largest categories of overhead in your lab.

Trade-offs

↔
Compact encodings save memory but cost CPU on large structures; raising thresholds too high slows operations.
↔
Active defragmentation reclaims memory at the cost of background CPU; restarts reclaim it instantly but need failover planning.

Done when you can

I can read INFO memory and explain used_memory vs RSS.
I know the main encodings and when conversions happen.
I can diagnose fragmentation, swapping and fork memory growth.

Command Palette