Fix cluster memory alerts with agents gated by human approval

A homelab owner runs kagent agents that turn Grafana memory alerts into pull requests and gated pod restarts; a Prometheus fix took one merge and one approved delete.

What they did

Arigsela runs ten AI agents on kagent, an open-source agent runtime, inside a three-VM k3s homelab that Argo CD deploys from a public Git repo. When Grafana fires a memory alert, a small dispatcher written as plain code hands it to an implementer agent. That agent reads the pod, its events, logs and manifest, and if the limit is simply too low it raises it by half and opens a GitHub pull request. Its tool list holds 11 read and propose tools and no merge tool, and its GitHub App identity cannot touch CI. If the problem is a leak, or a merged raise has not reached the running pod, a second restarter agent gets one tool, a pod delete, which pauses for the author to approve on a web page or with Slack buttons. Before offering the approval, the dispatcher checks that the pending call names the alert's pod and namespace, refuses protected namespaces and caps restarts at two a day.

How it went

For Prometheus running at 90% of its limit, the implementer opened a one-file pull request with its evidence, passed 30 checks and waited 10 minutes until the author merged it. The pod kept the old limit until a restart; the restarter paused with the namespace and pod name shown, the author approved, and the pod came back with the new limit. A replay against a healthy pod gave an inconclusive verdict and no restart. On live alerts since, the dispatcher refused a second restart of the same pod within 24 hours, and refused a restart after the implementer wrongly called a problem in the kagent namespace a leak.

Worth knowing

Merging a fix and running an action are two separate approvals: with an OnDelete StatefulSet a merged limit does nothing until someone deletes the pod. The author's point is that limits must live in the agent's tools, permissions and dispatcher code, not in its prompt, because the model will sometimes misdiagnose. This is one person's homelab, though the agents act on a real cluster and the config is public.

Try it

With Custom agent· 5 parts to fill in

When [my alerting tool] fires a memory alert for a pod in [my cluster], read the pod, its events, recent logs and its manifest in [my Git repo]. If usage sits steadily near the limit with no OOM kills, raise the limit by [percentage] and open a pull request that quotes your evidence; never merge it yourself. If it looks like a leak, or a merged change has not reached the running pod, propose deleting exactly that one pod and pause so I can see the namespace and pod name before anything runs. Never touch [protected namespaces], and stop without acting if the evidence is inconclusive.

Square brackets are the parts to fill in. The prompt is ours; the story is theirs.

Where they said it

Related