Qwen3.8-Flash-Next @ NVFP4 · RTX 5090 · mtp3-host-experts

warning

External engine. Not built with crit, and its source has not been reviewed by crit-hub. Read it before you build or run it.

Contributed by giveen · giveen/ninfer-ext · source at 7b586bde

140.9 tok/s
Decode: –
Prefill: –
TTFT: 3.518
Perplexity (absolute): –
Mean KL divergence: 262,144 tok
Context

NInfer, a from-scratch C++23/CUDA inference engine (an extended fork of upstream NInfer), on one RTX 5090 (32 GB) with Ubuntu 26.10, GCC 15.3 and CUDA 13.3. Qwen3.8-Flash-Next is a ~180B-parameter MoE whose 512 routed experts per layer and 51B n-gram embedding table cannot fit in 32 GB. This engine keeps the routed experts NVFP4 in pinned Host memory and fetches misses into an LRU device expert cache inside the decode CUDA Graph; the n-gram table stays off the device, mapped through the page cache or streamed from NVMe. MTP drafts three tokens per round. The source is not in this folder; it is giveen/ninfer-ext at the pinned commit above.

Model

nvidia/Qwen3.8-Flash-Next-NVFP4 at revision fc694b54fb0174e0913e6adf86691ef85a4ead47 (Qwen4ExpForCausalLM: 48 layers, 36 gated DeltaNet and 12 Qwen Sparse Attention, 4-stream hyper-connections, 512 NVFP4 experts top-10 plus a shared expert, a per-layer n-gram table, an MTP block, and a vision tower). Routed experts keep their stored NVFP4 codes and scales (the MTP bank is re-encoded from block FP8); the n-gram table becomes one row-scaled FP8 matrix; attention, GDN, hyper-connection and PLE projections are Q8; token embedding Q8 and the output head Q6. card.lock.json lists every checkpoint file with the size and SHA-256 the conversion recorded; python -B tools/verify_model.py from a crit-hub checkout checks a download against it.

The converted artifact (about 119 GB, published as parts) and its SHA256SUMS are at jabbatheduck/ninfer-ext-models/qwen3.8-flash-next; qwen3_8_flash_next_nvfp4.ninfer is bed67cf5343dfc697b8ddd3f29052e5f8bda1a22eac9985f5c4c0ac58e064d0f. It is not redistributed here.

Design

Part	How
Execution	One Program per Engine on one GPU; one compact decode batch per round in one CUDA graph; concurrency 1–8 fixed at startup; no preemption
Routed experts	NVFP4 in pinned Host memory, resident per layer as one bank; each MoE layer resolves top-10 against an LRU device cache inside the graph and copies misses over PCIe; `--expert-cache auto` gives the cache the memory left after the KV floor
Optional host split	`--host-experts N` computes `--host-expert-share` of a narrow call's misses on N host threads from the pinned bank while the rest cross PCIe (off by default; Q8_0 activations, so output then depends on cache contents)
Prefill	Chunks of 103+ tokens stream a whole expert layer through double-buffered staging banks, copying cached experts device-to-device; staged experts run the W4A4 tensor-core route when the artifact stores NVFP4 input divisors (A16 branch otherwise); idle steps widen to 16,384 tokens using the cache's top slots
N-gram table	File-mapped through the page cache (`mapped`, default when host memory allows) or batched direct I/O from NVMe (`stream`)
Speculation	MTP, 3 draft tokens per round, adaptive length, full proposal head; `--lm-head-draft` selects the optimized head
KV cache	Qwen Sparse Attention with `bf16`, `int8`, `fp8`, `nvfp4` and `k8v4` profiles; `--kv-capacity` explicit at full context avoids reserving more than one request needs
Serving	OpenAI- and Anthropic-compatible HTTP APIs, SSE streaming, grammar constraints, tool calls

Measured

Hardware for every row: one RTX 5090 (32 GB, sm_120a), CUDA 13.3, Ubuntu, --expert-cache auto.

Leaderboard run

LocalMaxxing records 140.9 tok/s output for this engine and encoding on the same GPU. The run page carries the exact test settings and depth.

Project measurements (published with this artifact)

From the artifact card (FP8 KV, --spec mtp K=3, development RTX 5090):

Metric	Value
Causal perplexity (`ninfer-ppl-1m-v1`, quick, FP8 KV)	3.518
Decode, one request, MTP K=3	81 tok/s
Prefill, 262,144-token budget, `--ngram-residency stream`	1,090 / 3,666 / 3,498 / 2,908 / 2,218 tok/s at 1.5k / 8k / 64k / 128k / 256k tokens
Prefill with the table mapped, 8k prompt	about 6,250 tok/s (about 1.7x the streamed figure)
Peak host RSS with `--ngram-residency stream`	about 65 GiB

Decode saturation (reasoning load)

From the repository's master-chart campaign (docs/flash-next.md): one RTX 5090, BF16 KV, a 300-token AIME prompt and 1,024-token generations per request, aggregate decode over complete intervals whose decode batch equals C.

C	plain	MTP K=3	gain
1	69.7	93.4	1.34x
2	116.1	231.3	1.99x
4	169.2	194.7	1.15x
8	252.5	265.5	1.05x

Prefill over the same protocol: 4,861 / 5,232 / 4,468 / 3,319 tok/s at 7,680 / 64,512 / 130,048 / 260,096 prompt tokens; it is the only model in that campaign that runs the last prompt.

Serving concurrency (512-token essays)

From the same repository at fork commit fc305acd, ninfer-serve, aggregate decode tok/s, mean of two runs: no speculation 98.5 / 141.6 / 180.6 / 185.3 at C=1 / 2 / 4 / 8; --spec mtp 82.2 / 136.8 / 165.0 / 182.1. A 241,003-token prompt was answered correctly with FP8 KV and an explicit 262,144-token capacity (3 min 59 s to first token; prefill about 1,000 tok/s averaged).

Checks

Arithmetic: the Qwen4Exp mathematics and state semantics are checked against FP64 oracles in tests/models/qwen4_exp/ upstream; this entry does not re-run them.
Artifact identity: the published artifact carries SHA256SUMS; card.lock.json pins the source checkpoint files by revision and SHA-256.
Benchmarks: the tables above are the project's published runs and the leaderboard run; each row names its protocol and, where the fork's numbers come from an earlier commit, that commit.

Limits

One RTX 5090 and about 65 GiB of host RAM for the pinned experts with --ngram-residency stream; the 51 GB n-gram table is never fully resident in that mode, and about 127 GB of disk is needed for the artifact.
Expert misses cross PCIe, so decode throughput depends on the host and link, not only the GPU.
--host-experts changes rounding (Q8_0 activations) and is off by default.
MTP wins on long reasoning generations and loses on short essays; the mode that wins is a property of the load.
--kv-capacity auto reserves more KV than one full-length request needs and takes it from the expert cache; use an explicit capacity for concurrency.
Vision is part of the artifact but not covered by the numbers above.

Build and chat

Prerequisites: CMake 3.28+, Ninja, a C++23 host compiler, CUDA 13.x (sm_120a), FFmpeg development libraries, libcurl 7.85+, and Python 3.11 with numpy/torch for conversion.

git clone https://github.com/giveen/ninfer-ext.git
git -C ninfer-ext checkout 7b586bde50747eac26d198530aa04d83216947a9
cd ninfer-ext
cmake --preset release
cmake --build --preset release
# binaries: build/apps/ninfer, build/apps/ninfer-serve, build/apps/ninfer-perplexity

Check the checkpoint against card.lock.json from a crit-hub checkout: python -B tools/verify_model.py <this engine folder> /path/to/checkpoint.

Convert the checkpoint from the cloned tree:

python3 -m tools.convert \
  --model /path/to/Qwen3.8-Flash-Next-NVFP4 \
  --recipe qwen3_8_flash_next_nvfp4 \
  --components text,vision,mtp \
  --resource chat_template.jinja=tools/chat_templates/qwen.jinja \
  --name qwen3.8-flash-next \
  --out models/qwen3.8-flash-next/qwen3_8_flash_next_nvfp4.ninfer

Serve:

./build/apps/ninfer-serve qwen3_8_flash_next_nvfp4.ninfer \
  --model-id qwen3.8-flash-next --max-concurrency 2 \
  --max-context 229376 --kv-capacity 458752 --kv-dtype fp8 \
  --expert-cache auto --ngram-residency auto --spec mtp --draft-tokens 3

The server listens on http://127.0.0.1:8080 with the OpenAI-compatible /v1/chat/completions and Anthropic-compatible /v1/messages APIs. One-shot terminal chat is the same ninfer command as any other NInfer artifact.

Credits

NInfer is licensed under the Apache License 2.0; the license text is at the pinned commit, LICENSE. NInfer is a fork of Neroued/ninfer, and the components it links (CUDA, FFmpeg, libcurl, xgrammar, llama-jinja) and the checkpoint weights keep their own licenses and terms, recorded in that repository.

crit-hub › rtx5090 › Qwen/Qwen3.8-Flash-Next › nvidia/Qwen3.8-Flash-Next-NVFP4 › mtp3-host-experts