Trace planted AWS faults to a root cause and rollback plan

Bebych worked through a set of AWS workshop labs where faults are injected on purpose: ten ECS failures, five EKS failures, a serverless stack, SageMaker and a Step Functions, Glue and EMR data pipeline. For each one he handed AWS DevOps Agent a short plain-language problem, such as investigating a named CloudWatch alarm or asking why a service would not start, and let it read logs, metrics, configuration, deployment history and CloudTrail on its own. He gave it Kubernetes read permissions for the EKS labs, connected GitHub code indexing so it could look at source changes, and tried custom skills that either spelled out an investigation procedure or described the system's services and dependencies. One lab wired CloudWatch alarms through EventBridge and Lambda so an investigation started by itself when a tagged alarm fired.

The agent found the causes he describes in detail. A DynamoDB throughput limit cut to a tiny value. A Secrets Manager permission removed from an ECS execution role, which only surfaced once a new deployment had to fetch the secret again. A sidecar that pushed a node near 100% CPU and added network latency. On the code side, an N+1 query in a report generator. Its plans covered checks, the change itself and a rollback, and after one suggested rollback the CPU graph fell back to normal. Where it lacked a permission or data source it said so. He notes results varied between runs, and skills made them more consistent without making them deterministic.

These were lab environments with planted faults, not live incidents. The agent can only find what it can see: missing CloudTrail, Kubernetes or repository access leaves gaps, and AWS's own documentation says findings need a person to check them before anything changes production. AWS DevOps Agent is not in our agent list, so this is tagged custom.