vllm-loop model-improvement progress tracker junk-tokens
models / / #25
▲ MADE POSITIVE PROGRESS

PP8 current-state sweep + conflict flagged

PP8 (TP1xPP8) on our box: single-stream 43.2/40.9/32.2/27.6 @8/512/4k/32k; aggregate 4k c8 154.9, 32k c16 166.6, 128k c16 139.0, 512k c16 86.8; prefill 4k c1 1350 / c16 7067, 128k c16 30058. Conflict resolved by measurement: PP8 is ~2.3x SLOWER at c1 than TP8xPP1 (43 vs 100.8) - c1 -> TP8xPP1…

When (PT)2026-09-11 22:04 PT
ServingTP1xPP8 (Zanooda 13-patch stack), fp8_ds_mla
Enginezanooda/vllm-sm80-ds41f:v41-sm80 + KV-layout intersection fix
KV

Benchmarks

MetricValueΔ vs previousUnitContextNote
agg_decode_4k_tok_s (agg_decode_4k_tok_s) 154.9 tok/s -48.8% tok/s PP8 c8
agg_decode_512k_tok_s (agg_decode_512k_tok_s) 86.8 tok/s -33.0% tok/s PP8 c16
c1_decode_tok_s (c1_decode_tok_s) 43.2 tok/s -57.2% tok/s PP8, 8-tok prompt
prefill_tok_s (prefill_tok_s) 30058 tok/s 436.8% tok/s PP8 c16 @128k

raw JSON: /api/reports/25