What they did
Maikon Marquez built a practice orders platform on AWS (a FastAPI API, PostgreSQL, an SQS queue and worker on k3s) with Prometheus SLOs and burn-rate alerts, all in Terraform. When Alertmanager fires, a webhook calls a custom agent. It collects evidence with read-only Kubernetes access: pods, restarts, events, env vars, logs, Prometheus metrics and SQS queue depth. Code computes obvious signals first, such as a failure flag being on or a deployment at zero replicas. Then it sends the evidence to Amazon Bedrock's Nova Micro model with an SRE playbook, asks for a cause, confidence and proposed command in fixed JSON, and opens a GitHub issue labelled as needing approval. Marquez then ran two chaos sessions, injecting failures under steady traffic.
How it went
On the first try of each failure scenario the diagnosis came back wrong or vague, three times in all. Marquez never swapped the model; cleaning the evidence fixed it, by moving computed signals to the top of the prompt, dropping events from healthy pods and passing only error log lines. When evidence was weak the model answered with medium confidence. The tests also turned up 11 silent bugs in the alerting chain, and the agent once leaked the AWS account id into a public issue. Both sessions cost US$1.17; each investigation used about 5,000 tokens and took about a second.
Worth knowing
The agent can only get and list, never change anything, and its prompt tells it to treat logs as data, not instructions. Code now strips account ids before publishing. The write-up is in Portuguese, and the full project, post-mortems and the agent's issues are in the author's open repo.