What ships here is not an internal tool. Customers' websites run on these services, publicly, indexed, with their phone numbers and their prices on them. A broken deploy is somebody's business offline while they are asleep and I am asleep.

Multitenancy makes that worse in a specific way. The services are shared, so there is no such thing as breaking one customer. A bad render path breaks every site at once.

And there is nobody else. No reviewer, no second pair of eyes on the diff, no colleague who notices the thing I stopped seeing three hours ago. The pipeline is the review step, and it is the only one that exists.

Every service reads its configuration from environment variables, and those variables are the secrets: the database connection string, the cloud credentials, the model API keys. A service handed an incomplete environment does not run badly. It fails to start.

That is why the first job checks that every secret the deploy will need actually exists. Without it, a missing key surfaces at the very end, after every test has run and every container has been built and pushed, and the whole run is wasted on a condition that took two seconds to detect.

The second job detects changes. The platform is seven services, separately built and separately deployed: the console customers work in, the frontend that renders their public websites, the API, the router that steers each domain to the right site, the file service, the AI service, and the error-triage service.

Redeploying all seven for a one-line fix is not merely slow. Every service that redeploys is a service that restarts, and a restart is when a process is most likely to fail, on a bad environment variable or a migration that has not landed. This job diffs the push against what is live and decides which services the change actually touched, so a fix to the file service leaves the other six running untouched.

Then twelve test jobs fan out at once, on parallel machines. Lint and format verification. Unit tests, the fast checks of individual functions. Three integration suites that open a live connection to a real Postgres and exercise the API the way the product does, because a test against a fake database proves nothing about the real one. An RBAC suite checking every declared permission against what the routes enforce. And six Python suites for the Python services and their shared libraries.

They run in parallel because they are independent, and running them in sequence would put the whole crossing well past an hour. A reporting job collects the twelve verdicts. Any red and nothing ships.

All green, and the deploy jobs start, one per service that detect-changes flagged. Each one builds its service into a container, an image that packages the program with everything it needs to run, and hands it to Google Cloud, where the new version replaces the old one.

Then seven health checks, one per service, and they exist because a successful deploy and a running service are different facts. The deploy step reports that the container was accepted. It does not report that the process inside it started, found its configuration, reached the database and began answering. A container that crashes on boot is a successful deploy by every measure the deploy step has.

So each health check asks its service to answer a request before the pipeline calls the push finished. If a service cannot answer, the run goes red and I know within the same twenty minutes, rather than finding out when a customer's site stops loading.

Twenty minutes, several times a day, and the bill grows every week as the test half does. I have fought with this file more than with any other part of the platform, and the fight is economics rather than correctness.

The obvious saving is to stop running everything on every push, since a change to the file service does not need the RBAC suite to pass. What stops me is that the moment a rule decides which tests a change needs, that rule can be wrong quietly, and a test that was skipped for a bad reason looks exactly like a test that passed. So it all runs, and the decision gets revisited every time the invoice arrives.

In exchange, no change reaches a customer's website without crossing twenty-nine jobs. A green run means the change is live and every check passed. A red run means the change stopped in the pipeline and production is still serving what it was serving before.