Designing CI/CD Pipelines at Scale

February 28, 2024

When a CI/CD platform serves a handful of teams, almost any pipeline design works. When it serves hundreds of teams and thousands of daily deployments, the design choices that didn't matter before suddenly matter a lot.

Standardize, but leave an escape hatch

Provide opinionated pipeline templates that cover 90% of use cases out of the box. For the remaining 10%, allow teams to extend or override stages rather than forking the whole pipeline.

Cache aggressively, invalidate carefully

Dependency installs and build artifacts are usually the biggest time sinks. A well-tuned cache layer can cut pipeline duration dramatically - but stale caches cause some of the hardest-to-debug failures, so invalidation rules need to be explicit and tested.

Make security scanning non-negotiable but fast

SAST, dependency scanning, and container image scanning should run on every pipeline, but they need to be fast enough that teams don't try to bypass them. Run scans in parallel with tests, not after them.

Measure deployment frequency and lead time

If you can't see how often teams deploy and how long changes take to go live, you can't improve it. Dashboards for these DORA-style metrics make platform investment decisions much easier to justify.


This is a placeholder post - replace it with your own write-up, or add new .md files to content/blog/.