Command Palette

Search for a command to run...

Mission 0.2 · Stage 0 — Platform Foundations: The Paved Road on Kubernetes

Traffic In: Gateway API, cert-manager, and ExternalDNS

Goal: Any team can expose a service at <name>.shoplite.dev with a valid certificate and a DNS record by committing one HTTPRoute, with no tickets.

60 min ~$0.03/hr (EKS load balancer) + Route 53 hosted zone 5 steps 2 break-it drills

By the end of this mission

  • Explain Gateway API's roles (GatewayClass, Gateway, HTTPRoute) and why it's replacing Ingress
  • Install a Gateway API implementation and share one Gateway across teams
  • Issue certificates automatically with cert-manager and Let's Encrypt
  • Create DNS records automatically from routes with ExternalDNS

Part 1

Understand it first

Why Gateway API

Ingress (Kubernetes course, Ingress topic) is one resource that mixes infrastructure concerns (which load balancer, which certificates) with application concerns (which paths go where), and anything advanced lives in controller-specific annotations. GATEWAY API splits it by ROLE: the infrastructure provider supplies a GatewayClass; the PLATFORM team owns a Gateway (listeners, hostnames, certificates, which namespaces may attach); APP teams own HTTPRoutes in their own namespaces (paths, headers, traffic weights). That split is exactly what a platform needs: the platform controls the front door, and teams self-serve their routes.

It's portable across implementations (Envoy Gateway, Istio, Cilium, NGINX Gateway Fabric, the AWS Load Balancer Controller), and has first-class traffic splitting, header matching, and cross-namespace references (via ReferenceGrant). The Ingress-NGINX controller is being retired, which makes Gateway API the default choice for new platforms.

cert-manager

cert-manager is a controller that obtains and RENEWS certificates automatically. An Issuer/ClusterIssuer describes where certificates come from (Let's Encrypt via ACME, a private CA, AWS Private CA); a Certificate resource (or an annotation on a Gateway) requests one; cert-manager proves domain control (HTTP-01 or DNS-01 challenge), stores the result in a Secret, and renews before expiry. Expired certificates are one of the most common avoidable outages, and cert-manager removes the human from that loop.

ExternalDNS

ExternalDNS watches Services, Ingresses, and Gateway API routes and creates matching records in your DNS provider (Route 53 here), pointing at the load balancer. It records ownership in TXT records so it only touches records it created. With policy: upsert-only it never deletes, which is a safe default while you gain confidence (Networking course, DNS records).

Who owns whatdiagram
Rendering diagram…

Part 2

Your project after this mission · 6 files change

shoplite-platform/
  • shoplite-gitops/
    • apps/
      • checkout/
        • base/
          • httproute.yamlnew
  • shoplite-platform/
    • addons/
      • cert-manager/
        • cert-manager-app.yamlnew
        • cluster-issuer.yamlnew
      • envoy-gateway/
        • envoy-gateway-app.yamlnew
      • external-dns/
        • external-dns-app.yamlnew
      • gateway/
        • shop-gateway.yamlnew

Part 3

Build it, step by step

  1. 1

    Install Envoy Gateway through Argo CD

    Envoy Gateway is a CNCF Gateway API implementation built on Envoy (Networking course, HAProxy & Envoy). The chart installs the Gateway API CRDs too. The pattern for every add-on is the same: an Application pinned to a chart version.

    shoplite-platform/addons/envoy-gateway/envoy-gateway-app.yamlwhole fileyaml
    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: envoy-gateway
      namespace: argocd
      annotations: { argocd.argoproj.io/sync-wave: "-5" }
    spec:
      project: default
      source:
        repoURL: docker.io/envoyproxy
        chart: gateway-helm
        targetRevision: v1.5.1
      destination:
        server: https://kubernetes.default.svc
        namespace: envoy-gateway-system
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true, ServerSideApply=true]
  2. 2

    Install cert-manager and a Let's Encrypt ClusterIssuer

    Add cert-manager-app.yaml the same way (chart cert-manager from https://charts.jetstack.io, crds.enabled: true, and config.enableGatewayAPI: true). The ClusterIssuer uses the DNS-01 challenge through Route 53, which works for wildcard certificates and private load balancers. cert-manager's service account gets an IAM role (Pod Identity) allowed to change only the shoplite.dev zone.

    Use Let's Encrypt's STAGING server first: production has strict rate limits, and a misconfigured issuer retrying in a loop can lock you out for a week.

    shoplite-platform/addons/cert-manager/cluster-issuer.yamlwhole fileyaml
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: letsencrypt
    spec:
      acme:
        server: https://acme-v02.api.letsencrypt.org/directory
        # server: https://acme-staging-v02.api.letsencrypt.org/directory   # test first
        email: platform@shoplite.dev
        privateKeySecretRef: { name: letsencrypt-account-key }
        solvers:
          - dns01:
              route53: { region: ap-south-1 }
            selector:
              dnsZones: ["shoplite.dev"]
  3. 3

    Install ExternalDNS for Route 53

    Chart external-dns from https://kubernetes-sigs.github.io/external-dns/. The values that matter: watch Gateway API routes, limit to your zone, and mark ownership.

    shoplite-platform/addons/external-dns/external-dns-app.yamlwhole fileyaml
    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: external-dns
      namespace: argocd
    spec:
      project: default
      source:
        repoURL: https://kubernetes-sigs.github.io/external-dns/
        chart: external-dns
        targetRevision: 1.19.0
        helm:
          valuesObject:
            provider: { name: aws }
            sources: [service, gateway-httproute]
            domainFilters: [shoplite.dev]
            policy: upsert-only
            txtOwnerId: shoplite-prod-eks
            serviceAccount:
              name: external-dns     # bound to an IAM role with route53:ChangeResourceRecordSets on this zone
      destination:
        server: https://kubernetes.default.svc
        namespace: external-dns
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true]
  4. 4

    The shared Gateway, owned by the platform

    One Gateway with an HTTPS listener for *.shoplite.dev. The cert-manager.io/cluster-issuer annotation makes cert-manager create the wildcard certificate Secret the listener references. allowedRoutes lets only namespaces labelled shoplite.dev/expose: "true" attach routes, which is a guardrail the platform controls.

    shoplite-platform/addons/gateway/shop-gateway.yamlwhole fileyaml
    apiVersion: gateway.networking.k8s.io/v1
    kind: GatewayClass
    metadata: { name: envoy }
    spec:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    ---
    apiVersion: gateway.networking.k8s.io/v1
    kind: Gateway
    metadata:
      name: shop-gw
      namespace: gateway
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt
    spec:
      gatewayClassName: envoy
      listeners:
        - name: https
          protocol: HTTPS
          port: 443
          hostname: "*.shoplite.dev"
          tls:
            mode: Terminate
            certificateRefs: [{ name: wildcard-shoplite-dev }]
          allowedRoutes:
            namespaces:
              from: Selector
              selector:
                matchLabels: { shoplite.dev/expose: "true" }
        - name: http
          protocol: HTTP
          port: 80
          hostname: "*.shoplite.dev"
          allowedRoutes: { namespaces: { from: Same } }
  5. 5

    A team exposes its service with one HTTPRoute

    This file lives in the TEAM's config (the GitOps repo from the GitOps course). The platform handles the certificate, DNS, and load balancer. The team only states hostname, path, and backend.

    shoplite-gitops/apps/checkout/base/httproute.yamlwhole fileyaml
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
      name: checkout
    spec:
      parentRefs:
        - { name: shop-gw, namespace: gateway, sectionName: https }
      hostnames: ["checkout.shoplite.dev"]
      rules:
        - matches: [{ path: { type: PathPrefix, value: / } }]
          backendRefs: [{ name: checkout, port: 80 }]
    terminal
    $ kubectl label ns checkout shoplite.dev/expose=true
    kubectl -n checkout get httproute checkout -o jsonpath='{.status.parents[0].conditions[?(@.type=="Accepted")].status}'; echo
    kubectl -n gateway get certificate
    kubectl -n external-dns logs deploy/external-dns | grep -m1 checkout
    curl -sI https://checkout.shoplite.dev/healthz | head -1
    ── expected output ──
    True
    NAME READY SECRET AGE
    wildcard-shoplite-dev True wildcard-shoplite-dev 6m
    level=info msg="Desired change: CREATE checkout.shoplite.dev A" profile=default zoneName=shoplite.dev.
    HTTP/2 200

Checkpoint — you should now have

  • ✓A Gateway owned by the platform serves *.shoplite.dev with a cert-manager certificate that shows READY True.
  • ✓Committing an HTTPRoute in a team namespace creates DNS and serves HTTPS with no tickets.
  • ✓Namespaces without the shoplite.dev/expose label can't attach routes.

Part 4

Break it on purpose

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.

Break #1

Route not accepted

The catalog team adds an HTTPRoute in namespace catalog, which doesn't have the expose label.

terminal
$ kubectl -n catalog get httproute catalog -o jsonpath='{.status.parents[0].conditions[0]}' | jq
── what you'll see ──
{
"type": "Accepted",
"status": "False",
"reason": "NotAllowedByListeners",
"message": "No listeners included by this parent ref allowed this attachment."
}

Break #2

Certificate stuck pending

The cert-manager IAM role lacks route53:ChangeResourceRecordSets for the zone.

terminal
$ kubectl -n gateway describe challenge | tail -3
── what you'll see ──
Reason: Error presenting challenge: failed to change Route 53 record set: AccessDenied: User: arn:aws:sts::123456789012:assumed-role/cert-manager/... is not authorized to perform: route53:ChangeResourceRecordSets on resource: arn:aws:route53:::hostedzone/Z0ABC...
State: pending

Part 5

Interview questions from this mission

01

Gateway API vs Ingress?

02

How does cert-manager get and renew certificates?

03

What does ExternalDNS do and how does it avoid clobbering records?

Before you stop

Clean up

terminal
$ # Keep for the next missions. To remove the load balancer cost:
kubectl -n gateway delete gateway shop-gw
0/4 · 0%