Tune llama.cpp decode speed with an overnight autoresearch loop

A LocalLLaMA user left a cloud model tuning llama.cpp for about 20 hours on an RTX 5060 Ti. Qwen3.8 Flash Next IQ1_S went from 24 to 40.6 tok/s and IQ4_XS from 8.4 to 20.1.

Done withCustom agent· Nov 19, 2026

What they did

u/bobaburger wanted the fastest decode they could get from Qwen3.8 Flash Next in llama.cpp on a Ryzen 7 7700X, an RTX 5060 Ti 16GB and 32 GB of RAM, with a 128k context and the model spilling to disk. Instead of tuning by hand, they set up a loop driven by a paid cloud model: 12 test prompts across coding, maths, Q&A and general knowledge; a small program that compares output byte for byte against the stock build; and metrics for decode speed, cache hit rate and disk reads. A change was kept and logged only if it beat the last run and the outputs still matched. The agent tried flags, KV cache quantization, batch sizes and expert cache sizes, then moved on to editing llama.cpp source.

How it went

After almost 20 hours, IQ1_S rose from 24 to 40.6 tok/s mean decode and IQ4_XS from 8.4 to 20.1. Many ideas were tried and dropped: loading the whole model to heap crashed the machine on IQ4_XS, n-gram speculative decoding and lookahead prefetch hurt or did nothing, and a requantization that gained speed lost quality. The code changes are public on the poster's GitHub fork. On models that fit in VRAM plus RAM, the patch ran 2 to 5% slower than stock.

Worth knowing

The poster says they do not fully understand the code the agent wrote. The gains are for MoE models that spill to SSD. A commenter suggested keeping a held-out prompt set outside the loop, since 12 prompts drove every decision. The model and harness used were not named.

Try it

With Custom agent· 7 parts to fill in

Make [llama.cpp or my inference engine] decode [model] faster on [my hardware] without changing its output. Set up a loop: run [my fixed test prompts], compare the output byte for byte with the stock build, and record decode speed, cache hit rate and disk reads. Try one change at a time, flags first, then source changes. Keep a change only if it is faster and the outputs still match, and log every attempt with its numbers and why it was kept or dropped. Do not touch anything outside [the repo and a scratch directory]. Stop after [number] hours or when [number] attempts in a row give no gain, and send me the log.

Square brackets are the parts to fill in. The prompt is ours; the story is theirs.

Where they said it

Related