What they did
u/bobaburger wanted the fastest decode they could get from Qwen3.8 Flash Next in llama.cpp on a Ryzen 7 7700X, an RTX 5060 Ti 16GB and 32 GB of RAM, with a 128k context and the model spilling to disk. Instead of tuning by hand, they set up a loop driven by a paid cloud model: 12 test prompts across coding, maths, Q&A and general knowledge; a small program that compares output byte for byte against the stock build; and metrics for decode speed, cache hit rate and disk reads. A change was kept and logged only if it beat the last run and the outputs still matched. The agent tried flags, KV cache quantization, batch sizes and expert cache sizes, then moved on to editing llama.cpp source.
How it went
After almost 20 hours, IQ1_S rose from 24 to 40.6 tok/s mean decode and IQ4_XS from 8.4 to 20.1. Many ideas were tried and dropped: loading the whole model to heap crashed the machine on IQ4_XS, n-gram speculative decoding and lookahead prefetch hurt or did nothing, and a requantization that gained speed lost quality. The code changes are public on the poster's GitHub fork. On models that fit in VRAM plus RAM, the patch ran 2 to 5% slower than stock.
Worth knowing
The poster says they do not fully understand the code the agent wrote. The gains are for MoE models that spill to SSD. A commenter suggested keeping a held-out prompt set outside the loop, since 12 prompts drove every decision. The model and harness used were not named.