Security: Authentication, Authorization, Encryption

In one line

Secure Kafka in layers: TLS for encryption in transit, SASL (SCRAM, OAUTHBEARER, GSSAPI) or mutual TLS for authentication, ACLs with least privilege per principal on topics, groups and the cluster, encryption at rest for disks and backups, and secret and certificate rotation.

0/5 · 0%

Think of it like this

A secure office building. Badges prove who you are (authentication), door permissions decide which rooms you can enter (authorization), sealed envelopes protect documents moving between rooms (TLS), and locked cabinets protect files at rest (disk encryption).

Key ideas

01
Listeners: separate listeners for clients, inter-broker and controllers, each with a security protocol: SASL_SSL for clients is typical; SSL with client certificates for mTLS; never PLAINTEXT across untrusted networks.
02
Authentication: SASL/SCRAM-SHA-512 (username/password stored as salted hashes in the cluster), SASL/OAUTHBEARER (tokens from an identity provider, good for short-lived credentials and central identity), SASL/GSSAPI (Kerberos, common in enterprises), mTLS (certificate subject as principal). Avoid SASL/PLAIN without TLS.
03
Authorization: KRaft uses StandardAuthorizer; ACLs grant a principal operations (Read, Write, Create, Describe, Alter...) on resources (Topic, Group, Cluster, TransactionalId) with literal or prefixed names. Set allow.everyone.if.no.acl.found=false. Consumers need Read on the topic and on their group; transactional producers need Write on the TransactionalId.
04
Least privilege by prefix: service payments gets Write on payments. topics and Read on commerce.orders with group payments- prefixed; nothing else.
05
Data protection: encrypt disks (cloud volume encryption), protect backups and mirrored clusters equally, keep secrets and card data out of payloads (or encrypt fields end to end), and rotate credentials and certificates on a schedule with overlapping validity.
06
Audit: authorizer logs for denied operations, and alerts on unexpected principals or spikes in authorization failures.

Code & diagrams

acls.shbash

# Producer service: write its own topics (prefixed), idempotent/transactional writes
kafka-acls.sh --bootstrap-server $B --command-config admin.properties --add \
  --allow-principal User:payments --operation Write --operation Describe \
  --topic payments. --resource-pattern-type prefixed
kafka-acls.sh --bootstrap-server $B --command-config admin.properties --add \
  --allow-principal User:payments --operation Write --operation Describe \
  --transactional-id payments- --resource-pattern-type prefixed

# Consumer: read orders with its own group prefix
kafka-acls.sh --bootstrap-server $B --command-config admin.properties --add \
  --allow-principal User:payments --operation Read --operation Describe --topic commerce.orders
kafka-acls.sh --bootstrap-server $B --command-config admin.properties --add \
  --allow-principal User:payments --operation Read --group payments- --resource-pattern-type prefixed

client-sasl-ssl.propertiesproperties

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="payments" password="<from secret manager>";
ssl.truststore.location=/etc/kafka/tls/truststore.p12
ssl.truststore.type=PKCS12
ssl.endpoint.identification.algorithm=https

Interview problem

The problem

Multi-tenant Kafka security

Tenants A, B and C share a Kafka cluster and must be isolated. Design topics, ACLs, consumer groups, authentication and quotas.

When it breaks

allow.everyone.if.no.acl.found=true

What you see

Any authenticated principal can read or write any topic without ACLs, including newly created sensitive topics.

Fix & prevent

Set it to false and grant explicit ACLs; test access from an unprivileged principal.

Explain it without notes

Which ACLs does a transactional consumer-producer service need?

Practice

Enable SASL/SCRAM and ACLs in the lab and verify an unauthorized consumer is rejected.

Trade-offs

↔
TLS costs broker CPU and disables zero-copy; the protection is essential outside fully trusted networks.

Done when you can

I can secure Kafka with TLS, SASL or mTLS, least-privilege ACLs, and rotation.

Command Palette