Cutting a forty-minute CI pipeline down to six

A forty-minute pipeline does not read as an emergency on any dashboard. It reads as a number that has slowly grown, and everyone has adapted around it. What it actually is is a change to how the team works, arrived at without anyone deciding.

Pull requests get batched, because waiting forty minutes for one small change feels wasteful. Batched changes are harder to review and harder to bisect. A failure on main is expensive to confirm, so a revert becomes a judgement call rather than a reflex. None of that shows up as a CI problem in a retrospective. It shows up as the team feeling slow.

Measure before you optimise

Almost every slow pipeline we have looked at was slow for two or three reasons, and the team's intuition about which ones was wrong. Test suites get blamed because they are the part everyone watches; the actual time is usually spent before a single test runs.

A cache that misses is worse than no cache

The most common finding is a dependency cache that was configured once, looked correct, and has been missing on nearly every run since. It costs the full install plus the time to upload a fresh archive afterwards, and because the step is green nobody looks at it.

The usual cause is a cache key that includes something volatile — a commit sha, a timestamp, an OS image tag that moves. The key should be the lockfile hash and nothing else that changes more often than the dependencies do.

- name: Cache dependencies
  uses: actions/cache@v4
  with:
    path: ~/.npm
    # The lockfile hash is the only thing in the key. A new lockfile
    # misses; anything else — a new commit, a rerun, a different branch
    # — hits.
    key: npm-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      npm-${{ runner.os }}-

- name: Install
  # ci over install: it honours the lockfile exactly and skips
  # resolution, which is most of what install spends its time on.
  run: npm ci --prefer-offline --no-audit --fund=false

The restore-key matters as much as the key. Without it, one dependency change means a cold install. With it, a changed lockfile still restores the previous cache and installs only the difference.

Docker builds reward one specific ordering

Layer caching is invalidated from the first changed instruction downwards. So the ordering rule is simply: least frequently changed at the top. Source code changes on every commit and belongs at the bottom; dependency manifests change weekly and belong above it.

FROM node:22-slim AS deps
WORKDIR /app
# Only the manifests. This layer is reused until a dependency changes.
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci --prefer-offline

FROM node:22-slim AS build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
# Source last: a code change invalidates from here down and no further.
COPY . .
RUN npm run build

The cache mount is worth adding on top. It survives across builds independently of the layer cache, so even a genuine dependency change re-downloads only what is new rather than the entire tree.

Run independent things at the same time

Linting does not depend on tests. Typechecking does not depend on either. Running them serially in one job is the default in most pipeline templates and there is rarely a reason for it beyond that.

• Split lint, typecheck, and tests into separate jobs sharing one build artefact.
• Shard the test suite across runners — most frameworks support this natively.
• Build the image once and pass it to integration tests rather than rebuilding.
• Fail fast on the cheap jobs, so a formatting error does not wait behind a full test run.

None of this required rewriting a test or adopting a new CI provider. It was a cache key, a Dockerfile reordering, and four jobs where there had been one — roughly a day's work against something the team had lived with for two years.