Roll out Kubernetes component upgrades in stages with a scheduled agent

Brendan Neutra, an engineer on Benchling's production infrastructure team, describes how the company keeps more than thirty cluster components current: Helm charts such as Karpenter and Argo CD, node machine images, and container images like the Datadog agent. Versions live in checked-in constants files keyed by stage, so an upgrade is a one-line change per stage. A Cursor Cloud Agent automation wakes on a schedule and reads a rollout config for each component. The config says where releases are published, how long a release must age before it is picked, what to check in the release notes, which dependencies to verify, how to test health in a dev cluster with commands or Datadog metrics, and how many days each stage must bake. The agent works out where each component stands from the constants files and the git log, then either does nothing or opens the next PR. It can run Terraform plans and open PRs in GitHub, with AWS and Datadog access limited by policy, but cannot change production on its own. A human approves every PR. The agent also writes an expected Terraform plan pattern into the PR so an internal service can apply matching runs after merge.

The setup has run since April. In that time the agents opened and merged hundreds of rollout PRs across more than thirty components, roughly five times the team's earlier cadence, and the team stopped skipping releases. In one case the agent read release notes for a network controller update, found that Benchling relied on a default the new version changed, kept the old value in the same PR, explained why, and confirmed in a dev cluster that network settings held. Most scheduled runs end with no change. When dev verification fails, no PR is opened and the owning team is told in Slack.

This is the company's own engineering blog, not an outside review. It only works because the infrastructure was already declarative Terraform with staged environments and required PR review; the staged rollout is the safety net if the agent misses a breaking change. The agent is stateless and holds no production credentials. The post does not give cost or failure counts.