Test Strategy in Pipelines & Taming Flaky Tests

In one line

Put the right tests at the right stage: many fast unit tests on every commit, fewer integration and contract tests on every PR, a few end-to-end tests before release. Then treat flaky tests as bugs, because a pipeline people don't trust gets ignored.

0/4 · 0%

Think of it like this

Airport security. Everyone goes through the quick metal detector (unit tests); bags get X-rayed (integration tests); a few passengers are selected for a thorough search (end-to-end tests). Searching every passenger thoroughly would stop the airport.

Key ideas

01
The TEST PYRAMID: lots of UNIT tests (milliseconds, isolated), a solid layer of INTEGRATION tests (real database, queue: Testcontainers, Docker course), CONTRACT tests between services (Pact, or schema checks, so providers don't break consumers without spinning up everything), and a thin layer of END-TO-END/UI tests (slow, brittle, valuable for critical journeys only).
02
Pipeline placement: unit + static analysis on every push (< 5 min); integration and contract tests on PRs; E2E, performance smoke (k6), and security scans before deploy to production or nightly. Fail fast: order stages so the cheapest, most-likely-to-fail checks run first.
03
FLAKY TESTS pass and fail without code changes (timing, shared state, test order, real network calls, time zones, random data). They destroy trust: people start re-running pipelines until green and stop reading failures. Detect them (automatic retry-and-flag, historical pass rates), QUARANTINE (run but don't block, with an owner and a deadline), fix the root cause (deterministic clocks, isolated data, await conditions instead of sleeps), and track the flaky rate as a metric.
04
Speed techniques: parallelise and shard tests across runners (Phase 1, matrix), run only tests affected by the change (test impact analysis), cache dependencies and builds (Phase 1, caching), and keep the PR pipeline under ~10 minutes.

In your stack

→
JUnit 5 with Maven Surefire/Failsafe separates unit (*Test) from integration (*IT) tests; Testcontainers gives real Postgres/Kafka in integration tests; Gradle's test retry plugin can flag flaky tests; Pact JVM provides consumer-driven contract tests.

Code & diagrams

tests by stagediagram

Rendering diagram…

GitHub Actions: shard tests and flag flakesyaml

jobs:
  test:
    strategy:
      matrix: { shard: [1, 2, 3, 4] }
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-java@v5
        with: { distribution: temurin, java-version: "21", cache: maven }
      - run: ./mvnw -B verify -Dsurefire.rerunFailingTestsCount=1 -Dshard=${{ matrix.shard }}/4
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: reports-${{ matrix.shard }}, path: target/surefire-reports }

Explain it without notes

Why is automatically re-running failed tests until they pass dangerous?

Practice

A team's PR pipeline takes 40 minutes and fails randomly 1 in 5 runs. Propose a plan.

Trade-offs

↔
More E2E tests catch integration issues but slow and destabilise pipelines; contract tests give most of the safety at a fraction of the cost.

Done when you can

I can place tests at the right pipeline stage
I have a process for detecting, quarantining, and fixing flaky tests

Command Palette