Command Palette

Search for a command to run...

Hectal
← All projects

Project 12 of 12 · project brief

Capstone: A Production-Grade Platform, End to End

Everything combined: developer → GitHub → secure CI → ECR → Terraform-built EKS → Argo CD + Helm → observability → alerts → incident → runbook → recovery. Then prove it with a game day.

Advanced 2–3 weeks 5 milestones

The scenario

You're the founding platform engineer at ShopLite. Leadership wants a platform they can trust for a big sale in six weeks: developers ship safely several times a day, the system scales and survives failures, the team knows within minutes when something breaks and has a runbook for it, and cost is under control.

This capstone joins Projects 1–11 into one system and adds what separates 'it works' from 'it's production': architecture decisions written down, SLOs, disaster recovery, cost, and a rehearsed incident. It's the portfolio piece that demonstrates you can take a Java microservice from commit to production.

Before you start

Stack

TerraformAWS (VPC, EKS, RDS, ElastiCache, MSK, ECR, Route 53, CloudFront, WAF)GitHub Actions + OIDCArgo CD + Argo RolloutsHelmExternal SecretsKarpenterKyvernoOpenTelemetryPrometheus/Grafana/Loki/TempoAlertmanagerk6Chaos Mesh

Target architecture

Project 12 architecturediagram
Rendering diagram…

Deliverables and requirements

You will hand in

  • Three repos (infra, app, config) with READMEs and a top-level architecture document with ADRs (architecture decision records)
  • Infrastructure fully in Terraform, including DNS, CDN/WAF, EKS, data services, and IAM
  • Secure CI (Project 11) + GitOps CD with canary (Project 9) + observability and SLOs (Project 10)
  • Load-test report (k6) at 2× expected sale traffic, with the bottleneck found and fixed
  • DR plan with measured RTO/RPO, a game-day report, and a cost breakdown per service

Functional

  • A new service can be added by following a documented golden path in under a day
  • A code change reaches prod through PR → CI → dev → promotion PR → canary, with no manual cluster access
  • An induced failure (pod, dependency, AZ) is detected by an SLO alert, and the runbook leads to recovery

Non-functional

  • Checkout SLO: 99.9% availability, p95 < 400 ms at 2× peak load
  • Rebuild of the whole environment from code documented and timed
  • No static credentials anywhere; all secrets from Secrets Manager via External Secrets
  • Monthly cost estimate with top 5 cost drivers and an optimisation plan

Milestones

  1. 1

    Architecture and decisions

    Done when: A written design the team could review before you build.

    • Architecture diagram and request flow (Production Architecture guide)
    • ADRs for key choices: EKS vs ECS, managed vs self-hosted Kafka, Argo CD vs Flux, Helm vs Kustomize, DR strategy
    • SLOs and RPO/RTO agreed per service
    Stuck? Hints
    • An ADR is short: context, decision, consequences. Write down what you rejected and why; it's what interviewers ask about.

    Prove it works

    terminal
    $ ls docs/adr
    ── expected output ──
    0001-eks-over-ecs.md
    0002-managed-data-services.md
    0003-argocd-gitops.md
    0004-warm-standby-not-active-active.md
    0005-kyverno-for-policy.md
  2. 2

    Foundation: infra, delivery, security

    Done when: Terraform-built AWS + EKS, secure CI, GitOps CD, guardrails.

    • Combine Projects 7 and 8 into the infra repo (edge: Route 53, CloudFront, WAF)
    • Add Karpenter, Kyverno policies, External Secrets, Gateway/Ingress, cert-manager, ExternalDNS
    • Wire Projects 9 and 11: signed images only, canary to prod

    Prove it works

    terminal
    $ argocd app list -o name | wc -l; kubectl get clusterpolicy -o name | wc -l
    ── expected output ──
    14
    6
  3. 3

    Observe and load test

    Done when: Full telemetry, SLO alerts, and proof it holds 2× peak.

    • Project 10 across all services, with a checkout SLO dashboard
    • k6 test at 2× forecast peak with arrival-rate executor (Stateful course, Unit 5.1)
    • Find the first bottleneck with USE/RED (Unit 5.3), fix it, re-run, and write the report

    Prove it works

    terminal
    $ k6 run --out experimental-prometheus-rw sale-2x.js | grep -E 'http_req_duration|http_req_failed'
    ── expected output ──
    ✓ http_req_duration p(95)=318ms
    ✓ http_req_failed rate=0.07%
  4. 4

    Game day: incident → runbook → recovery

    Done when: A rehearsed failure, handled end to end, with a blameless postmortem.

    • Write runbooks for the top 5 alerts
    • Run a scheduled game day: inject payments latency and an AZ failure (Chaos Engineering guide)
    • Run incident command (SRE course), recover using runbooks, and write the postmortem with action items

    Prove it works

    terminal
    $ cat docs/postmortems/2026-10-gameday.md | head -8
    ── expected output ──
    # Game day postmortem — payments latency + AZ-a loss
    Detected: 2m 10s (CheckoutLatencyBurnFast)
    Mitigated: 11m (circuit breaker opened; traffic shifted from AZ-a)
    Customer impact: 0.4% of checkouts slowed > 2s; 0 failed orders
    What went well: alert → trace → runbook in < 5 min
    What didn't: runbook step 3 referenced a removed dashboard
    Action items: 4 (owners + dates)
  5. 5

    DR and cost

    Done when: Measured recovery and an understood bill.

    • Execute the DR runbook (restore/fail over per your chosen strategy) and measure RTO/RPO
    • Tag everything; OpenCost + Cost Explorer breakdown per service
    • Optimisation plan: Savings Plans baseline, Spot for workers, VPC endpoints, right-sized requests (Cost guide)

    Prove it works

    terminal
    $ cat docs/dr-rehearsal.md | grep -E 'RTO|RPO'
    ── expected output ──
    Measured RTO: 38 min (target 60)
    Measured RPO: 1 min (target 5)

Would you run this in production?

  • ☐Every component redundant across AZs; single points of failure reviewed
  • ☐Everything as code: infra, config, policies, dashboards, alerts, runbooks
  • ☐Secure supply chain: scanning, SBOM, signing, admission verification
  • ☐SLOs with burn-rate alerts; runbooks linked from alerts; on-call rota
  • ☐Backups and DR tested with measured RTO/RPO
  • ☐Cost visibility per team and a documented optimisation plan
  • ☐Game day performed and postmortem actions tracked

Stretch goals

  • Backstage portal with a golden-path template for new services (Platform course, Stage 1)
  • Crossplane self-service databases (Platform course, Mission 2.2)
  • Multi-region warm standby with Route 53 failover
  • DORA metrics dashboard (Platform course, Mission 2.4)

Show it off

Résumé bullet

Designed and built a production-grade AWS/EKS platform end to end: Terraform infrastructure, secure CI with signed images and SBOMs, GitOps canary delivery, OpenTelemetry-based observability with SLO alerting, load-tested at 2× peak, and validated through a chaos game day and a timed DR rehearsal.

Demo script

  • 10-minute walkthrough: architecture diagram → one commit flowing to prod as a canary
  • Live incident: inject a failure, show the alert, the trace, the runbook, and recovery
  • Show ADRs, the load-test report, the DR rehearsal results, and the cost breakdown

Interview questions about this project

01

Walk me through the platform you built.

02

What was the hardest problem and how did you solve it?

03

How would you evolve this platform for 10 teams?