crit-hub › rtx5090 › Qwen/Qwen3.8-27B › QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 › dflash2-int8kv

Qwen3.8-27B @ NVFP4 · RTX 5090 · dflash2-int8kv

NInfer, a from-scratch C++23/CUDA inference engine (an extended fork of upstream NInfer), on one RTX 5090 (32 GB) with Ubuntu 26.10, GCC 15.3 and CUDA 13.3. This entry runs the Qwen3.8-27B QUASAR QAT checkpoint converted to NInfer's v3 .ninfer artifact, with DFlash2 block drafting and an INT8 KV cache, which is what lets a 196,032-token context fit next to the draft companion. The source is not in this folder; it is giveen/ninfer-ext at the pinned commit above.

Model

QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 at revision 4a22293aa6d1026844bc93a8727f1837e48acd89: 64 layers (48 gated DeltaNet and 16 full-attention), hidden 5,120, vocab 248,320, native context 262,144, with a bundled BF16 MTP layer and vision tower. All 496 Text linears are stored NVFP4 and imported as stored; the token embedding and output head are re-encoded to FP8 E4M3 per row; the GDN a/b projections are stored NVFP4 but have no NVFP4 route, so the converter decodes them to their BF16 values. card.lock.json lists every checkpoint file with the size and SHA-256 the conversion recorded; python -B tools/verify_model.py from a crit-hub checkout checks a download against it.

Drafter: incoai/Qwen3.8-27B-DFlash2 at revision 015e795645c74b1a0eeef3b570031fb62e769bc5 (block diffusion, block size 8, five target taps, selector rank 256/top-k 16), converted into the same artifact and proposed through the target's output head (--lm-head-draft). Download it separately and pass it as the dflash2 source below.

The converted artifact is published at jabbatheduck/ninfer-ext-models/qwen3.8-27b-quasar-nvfp4 with a SHA256SUMS next to it; qwen3_8_27b_quasar_nvfp4.ninfer is bfd7f9e100fc26399096f920c937a168f51b4a2af1ba1d2cd7df57242456787a. It is not redistributed here.

Design

Measured

Machine: RTX 5090 32 GB, Ubuntu 26.10, GCC 15.3, CUDA 13.3, CMake 4.3, Ninja. All runs below are --max-concurrency 1, concise reasoning disabled, process pinned with taskset.

Leaderboard run

LocalMaxxing records 510.7 tok/s output for this engine and encoding on the same GPU. The run page carries the exact test settings and depth.

Context capacity and memory

Startup resolves the KV page count and refuses a plan that does not fit next to the weights and the 1 GiB automatic headroom. Measured with the artifact above, DFlash2 K=9 and --lm-head-draft:

The BF16 ceiling at K=9 is about 157k tokens; the draft companion holds roughly 2.6 GB of device memory outside the KV reservation (at 200,704 tokens the same artifact without --spec starts with BF16 KV, while with DFlash2 it does not). Free memory is read at load time, so anything else holding device memory when a server starts moves the ceiling by up to about 0.7 GB.

Quality (KV representation)

ninfer-perplexity on the repository's ninfer-ppl-1m-v1 corpus, --quick, --context 16384 --stride 8192, 261,167 scored tokens, one run per row, same artifact:

The differences are within run-to-run variation at this window size; only nvfp4 and k8v4 show a consistent cost. No KLD-budget campaign was run for this checkpoint.

Scope of these numbers

The leaderboard row is LocalMaxxing's standardized test. The capacity, memory and perplexity tables are this entry's own measurements with the commands below. The commands also print prefill speed and decode speed, and the server logs token counts every 5 s, so a local run can reproduce throughput on a specific machine.

Checks

• Capacity and memory: the table above is the engine's own startup resolution (cudaMemGetInfo after weights, page count, runtime reservation) printed on every start; values were read from those logs.
• Quality: perplexity over 261,167 tokens per KV representation, against a fixed corpus and fixed window protocol.
• Speculation: startup, drafting and verify were exercised by the capacity runs (each started, listened, and was stopped); speculative output is not bitwise equal to plain greedy decoding, by design.
• Arithmetic: the op oracles that qualify every numerical route live upstream (tests/ops/, docs/maintainer/op-development.md) and are not re-run by this entry.

Limits

• Text only; the artifact carries the vision tower but this entry does not exercise it.
• Needs a 32 GB-class GPU. With DFlash2 K=9 the BF16 KV ceiling is about 157k tokens; 196,032 tokens requires --kv-dtype int8 (or nvfp4/k8v4/fp8).
• Single GPU, one resident model, concurrency 1–8 fixed at startup, no request preemption.
• Speculative decoding changes rounding versus plain decode and is not bit-exact.
• Long-context quality was measured with 16,384-token windows; quantization error grows with the amount of history the KV cache holds.
• The measured ceiling depends on free device memory at load; the startup check is strict, so a desktop holding memory can reject a capacity that started earlier.

Build and chat

Prerequisites: CMake 3.28+, Ninja, a C++23 host compiler, CUDA 13.x (sm_120a), FFmpeg development libraries (libavformat, libavcodec, libavutil, libswscale), libcurl 7.85+, and Python 3.11 with numpy, torch, transformers for conversion.

git clone https://github.com/giveen/ninfer-ext.git
git -C ninfer-ext checkout 7b586bde50747eac26d198530aa04d83216947a9
cd ninfer-ext
cmake --preset release
cmake --build --preset release
# binaries: build/apps/ninfer, build/apps/ninfer-serve, build/apps/ninfer-perplexity

Check the checkpoint against card.lock.json from a crit-hub checkout: python -B tools/verify_model.py <this engine folder> /path/to/checkpoint.

Convert the checkpoint from the cloned tree:

python3 -m tools.convert \
  --model /path/to/Qwen3.8-27B-QUASAR-NVFP4 \
  --recipe qwen3_8_27b_quasar_nvfp4 \
  --source quantized=/path/to/Qwen3.8-27B-QUASAR-NVFP4 \
  --source dflash2=/path/to/Qwen3.8-27B-DFlash2 \
  --components text,vision,mtp,dflash2 \
  --resource chat_template.jinja=tools/chat_templates/qwen.jinja \
  --proposal \
  --name qwen3.8-27b-quasar \
  --out models/qwen3_8_27b_quasar_nvfp4.ninfer

--proposal reads tools/freq_corpus/fixtures/ranking/ranking.train.counts.i64 from the clone.

Serve and chat:

./build/apps/ninfer-serve models/qwen3_8_27b_quasar_nvfp4.ninfer \
  --model-id qwen3.8-27b --max-concurrency 1 --max-context 196000 --kv-capacity auto \
  --kv-dtype int8 --spec dflash2 --draft-tokens 9 --lm-head-draft

The server listens on http://127.0.0.1:8080 with the OpenAI-compatible /v1/chat/completions and Anthropic-compatible /v1/messages APIs. One-shot terminal chat:

./build/apps/ninfer models/qwen3_8_27b_quasar_nvfp4.ninfer \
  --prompt "Explain why data races are hard to debug." --max-context 196000 \
  --kv-capacity auto --kv-dtype int8 --spec dflash2 --draft-tokens 9 --lm-head-draft \
  --max-new 256 --greedy --no-thinking

Credits

NInfer is licensed under the Apache License 2.0; the license text is at the pinned commit, LICENSE. NInfer is a fork of Neroued/ninfer, and the components it links (CUDA, FFmpeg, libcurl, xgrammar, llama-jinja) and the checkpoint and drafter weights keep their own licenses and terms, recorded in that repository.