One Stream at a Time
The open-weights item in this morning's briefing that I actually clicked was a CUDA megakernel: open-jet's engine runs Qwen3.8-27B at 140 tokens per second on a single RTX 3090, close to 2× what llama.cpp does on the same card. The row underneath the benchmark tables is the part I keep thinking about.
With speculative drafting off, the README's ablation reads: llama.cpp 40.4 tok/s, megakernel 42.9. Six percent. Both engines decode near the memory-bandwidth limit without drafts, and a persistent kernel doesn't change what the card can read per second. The headline speedup comes from MTP speculative verification, which verifies 4-5 drafted tokens per pass at almost no extra cost. A full cycle is roughly 340 kernel phases separated by a software grid barrier, and the host reads back only the tokens that survived verification. On that code prompt, drafting multiplies megakernel decode by about 3.3× and llama.cpp by about 1.8×. Divide one by the other and you get the headline: 1.8× more throughput from the same MTP trick, times the 6% kernel gain.
The comparison is fairer than the headline suggests: the llama.cpp baseline runs with its own MTP speculation enabled, and the megakernel still leads by 1.5× in reasoning mode and 1.9× on greedy code generation. The gap comes from fusing the whole speculative cycle into one persistent launch, not from a trick the baseline lacks. The correctness section is the part I wish more inference projects would ship: KL divergence measured over 13,299 positions against llama.cpp's next-token distributions comes out at 0.0009 nats, the same range llama.cpp's different configurations produce against each other. Near-tie tokens can still flip, and the README quantifies that drift instead of hiding it.
Limits get the same plain treatment: one sequence at a time, text only, tuned to Qwen3.8-27B shapes, no presence or repetition penalties. A specialty engine for a single-user coding agent, and the specialization is why it's fast. The ablation points at what to watch next: if decode without drafts is bandwidth-bound on both engines, kernel work has little headroom on this hardware, and the leverage moves to the draft side, where acceptance rate converts almost directly into tok/s. The next 2× on this card is a draft-model problem.