Run a Kaggle competition with a team of agents and a read-only auditor

Agarwal, who validates financial crime models at a bank, entered Kaggle's Playground Series S6E10 in October 2026, a binary classification task on airline passenger satisfaction scored by ROC-AUC. Instead of one big prompt he wrote ten small Claude Code agent definitions. A manager agent that never writes code picked hypotheses from a backlog file and dispatched them. Researcher and ideator agents read the competition forums and public notebooks. A feature engineer and a modeler built features, ran five-fold cross-validation and logged each run to a CSV ledger. An auditor that could only read files and run git diff returned Pass or Fail, and an ops agent committed and submitted only after a Pass. A change was kept only if it raised out-of-fold AUC by 0.0001 and won in at least four of five folds. Agarwal acted as reviewer.

The baseline LightGBM scored 0.95886 in cross-validation. The biggest gain came from treating flight distance as a route id, an idea the researcher agent found in other competitors' public work. Later gains came from adding pretrained tabular models and a stack of publicly shared out-of-fold predictions, ending at 0.96226 CV and 0.96182 on the public board. Many plausible ideas, including a run of seven autonomous experiments, failed the keep rule. The audit also showed his own validation was optimistic: early stopping on the scored fold explained about half the gap between CV and the leaderboard. Most of his time went into checks, not feature code. The final rank was left blank in the post.

The fixed keep rule and a checker that cannot edit anything did more for the result than adding model types. Agarwal turned down a public blending recipe that scored higher on the leaderboard because it lost under his own validation and had been tuned to that leaderboard. He says he should have set up honest early stopping and nested stacking from day one, and started from public notebooks sooner.