The DGX Spark holds 128 GB of memory shared by its CPU and GPU but reads it at only 273 GB/s, and for single-user decoding that bandwidth, far more than the GB10’s arithmetic, decides how fast a model runs. Each generated token requires the GPU to read every weight in the model once, so a dense model’s decode rate cannot exceed the bandwidth divided by the bytes of its weights. For Qwen3.8-27B in the 4-bit NVFP4 checkpoint that Unsloth publishes, the bytes actually read per token come to 19.1 GB and the ceiling to 14.3 tokens per second; measured on one Spark, plain decoding reaches 11.4, which is 80% of the ceiling. Under the Chisel recipe the median request decodes at 67.8 tokens per second and the average over every generated token is 54.3, between five and six times faster than plain decoding and far above the ceiling that bounds it.

Throughput is the number of tokens each decode step produces divided by the time the step takes, and on this machine the step time is the bytes read divided by the bandwidth achieved. Quantization reduces the bytes, and speculative decoding raises the tokens per step because one read of the weights can verify several drafted tokens. The recipe comes from Chisel, an open-source project by the GitHub developer yuanlin2004 that automates building and tuning the inference stack for a given model on a given machine. Its DGX Spark recipes are each published with a measured single-user decode speed and task scores against the model’s BF16 weights, and the fastest of them for Qwen3.8-27B, combining Unsloth’s weights, a quantized DFlash2 drafter and a small vLLM plugin, is the one measured here. The post separates that recipe’s gain into six stages, each changing one thing and each measured with the same benchmark.

The median-request rate is Chisel’s headline metric and the one its published figures use; the token-weighted rate divides all generated tokens by all decode time and is lower because chat and long-story prompts, which decode more slowly, contribute as many tokens as code. Tokens per step divided by the step time gives the token-weighted rate.

Decode speed is bytes divided by bandwidth

Qwen3.8-27B is a stack of 64 decoder layers between an embedding table and an output layer. Each layer has a token-mixing block followed by a feed-forward block (the MLP). In 48 of the layers the token-mixing block is Gated DeltaNet, a form of linear attention, and in every fourth layer it is full attention. Every one of these blocks is a handful of weight matrices: the MLP’s gate and up projections are each $17{,}408 \times 5{,}120$ and its down projection is $5{,}120 \times 17{,}408$, while the largest attention projection is $12{,}288 \times 5{,}120$. Decoding one token for one user passes that token’s hidden state, a vector of 5,120 numbers, through every matrix once. A matrix with $d_{\text{out}}$ rows and $d_{\text{in}}$ columns turns a vector of length $d_{\text{in}}$ into one of length $d_{\text{out}}$, which takes one multiplication and one addition per weight, $2 d_{\text{out}} d_{\text{in}}$ floating-point operations, while the whole matrix, $d_{\text{out}} d_{\text{in}} b / 8$ bytes at $b$ bits per weight, has to be read from memory. That is an arithmetic intensity of $16/b$ operations per byte, 4 for NVFP4 and 2 for FP8, and the GB10 can perform hundreds of times more arithmetic per byte than that before compute becomes the limit, so the step time is set by the bytes alone. The MLPs hold 59% of the layers’ bytes, the Gated DeltaNet blocks 31% and the full-attention blocks 9%. The figure’s flow view follows one token through a single layer in the order the GPU runs it, and it shows that the operation that mixes information across tokens, the Gated DeltaNet state update or the attention over the cache, reads no weights at all, so every weight byte of a decode step is spent in the projections around that operation and in the MLP.

If $M$ is the number of bytes read per step and $W$ the memory bandwidth, the decode rate obeys

and the question of decode speed becomes a question of what $M$ is. The roofline argument behind this bound is developed in an earlier post on GPU monitoring for inference and, for mixture-of-experts models, in Part 1 of the DeepSeek series; here it is applied to a dense model on a much narrower memory bus.

The bytes read per token are smaller than the checkpoint on disk, and they were counted from the tensor headers of the Unsloth checkpoint rather than estimated from the parameter count. The 64 decoder layers hold 17.83 GB and the output layer, a $248{,}320 \times 5{,}120$ matrix that turns the final hidden state into a score for every vocabulary token, holds 1.27 GB. The embedding table (2.54 GB) contributes only one row per token, and neither the vision encoder (0.92 GB) nor the multi-token-prediction head (0.85 GB) runs during plain text decoding. The total read per step is therefore $M = 19.10$ GB, and at $W = 273$ GB/s the bound is 70.0 ms per token, or 14.3 tokens per second.

The measured step takes 88.0 ms, so plain decoding achieves 217 GB/s, 80% of the bandwidth on the datasheet. Chisel’s own measurement of the FP8 checkpoint on another Spark gives the same picture from a different starting size: 27.9 GB per token, a ceiling of 9.8 tokens per second and a measured 8.0, which is 82%. The 20% gap is treated below as a property of the machine. In BF16 the same model would read about 53 GB per token and could not exceed 5 tokens per second, which is why every practical recipe for this model on this machine starts from a quantized checkpoint.

Fewer bytes per weight: NVFP4

NVFP4 stores each weight as a 4-bit floating-point number in the E2M1 layout, with one sign bit, two exponent bits and one mantissa bit, which can represent only the values $0, \pm 0.5, \pm 1, \pm 1.5, \pm 2, \pm 3, \pm 4$ and $\pm 6$. A grid that coarse is usable only because every block of 16 consecutive weights carries its own scale. The scale is chosen so that the largest magnitude in the block maps to 6, it is stored as an 8-bit E4M3 floating-point number, and a single 32-bit scale per tensor brings the block scales into the range E4M3 can represent. A weight is reconstructed as

and the storage cost is $4 + 8/16 = 4.5$ bits per weight, with the per-tensor scale negligible. Because a block holds only 16 weights and its scale carries three mantissa bits, the largest weight in each block lands close to 6 and the remaining fifteen spread across the grid below it, and an outlier coarsens the grid only for the fifteen weights that share its block.

The Unsloth checkpoint is not uniformly 4-bit, and its quantization configuration states the exceptions. The feed-forward blocks of layers 0 to 55 are NVFP4; the attention projections, the linear-attention projections, the output layer and the feed-forward blocks of the last eight layers are FP8. Keeping the attention projections and the last layers at higher precision protects the parts of the network most sensitive to rounding, and it explains why the 64 layers occupy 17.83 GB, about 5.6 bits per weight on average, instead of the roughly 14 GB a pure 4.5-bit layout would need. One community recipe for SGLang describes this checkpoint as FP8 mislabelled as NVFP4 and warns that it produces incoherent text. The tensor headers contradict the first claim, since 168 of the tensors are packed 4-bit weights, and the second does not reproduce in vLLM: Chisel scored the same weights against the BF16 original on GSM8K, IFEval, HumanEval and LongBench NarrativeQA and found 99.3% to 101.2% of the BF16 score on every task.

Quantization lowers $M$ by a fixed factor, and the next step down, to 3 or 2 bits, costs accuracy. The remaining factor of almost four to five between the 14.3 tokens per second ceiling and the 54.3 to 67.8 measured has to come from producing more than one token for each read of the weights.

More tokens per weight read: speculative decoding

Speculative decoding exploits the fact that a step verifying several tokens costs almost the same as a step producing one. A cheap drafter proposes $K$ tokens, the target model runs one forward pass over all $K+1$ positions, and the target keeps the longest prefix of the draft that matches its own choices, plus one token of its own at the first disagreement. With greedy decoding a draft token is kept only if it equals the target’s top choice, so the output is identical to plain decoding and only the number of tokens per step changes. The verify pass multiplies each weight matrix by $K+1$ vectors instead of one, which raises arithmetic intensity by that factor while the bytes stay fixed, and at $K+1 = 11$ an NVFP4 layer is still far below the GB10’s ridge point. The mechanism, including the rejection-sampling rule that preserves the target distribution under sampling, is derived in an earlier post on speculative decoding.

If each draft position is accepted with probability $\alpha$, independently of the others, the number of tokens a step produces is a truncated geometric series, as derived by Leviathan, Kalman and Matias:

and the speedup over plain decoding is $E \cdot T_1 / T_{\text{step}}$, where $T_1$ is the plain step time and $T_{\text{step}}$ the time of a draft-and-verify step. A low $\alpha$ caps $E$ near $1/(1-\alpha)$ however many tokens are drafted, and an expensive drafter raises $T_{\text{step}}$ until the extra tokens no longer pay for it. Independence is a simplification, but it fits the measurements here closely: solving the formula for $\alpha$ at each stage gives 0.873 for MTP and 0.845, 0.841 and 0.839 for the three DFlash2 stages, and the value fitted at 7 draft tokens predicts 5.45 tokens per step at 10 against a measured 5.35.

MTP: the drafter inside the checkpoint

Qwen3.8-27B ships with a multi-token-prediction head, one extra transformer layer trained to predict the token after next from the final hidden state. With three draft tokens it raises the decode rate from 11.4 to 29.5 tokens per second, accepting 2.30 of its 3 drafts per step on average, at a cost the byte count makes visible.

The MTP head drafts one token per pass, so three drafts require three sequential passes, and each pass ends by projecting its hidden state through the output layer it shares with the target model. Every draft therefore reads the 0.85 GB layer and the 1.27 GB output layer, 2.12 GB in all, and a step with three drafts reads $19.10 + 3 \times 2.12 = 25.5$ GB. At the 217 GB/s that single-token passes achieve, that predicts a step of 117 ms, against 119.7 ms in the benchmark and 117.8 ms in an Nsight Systems profile, which records the three draft output-layer passes at 5.5 ms each, 16.7 ms per step in all. Three fifths of the draft cost is spent on the output layer, which has to score all 248,320 vocabulary entries only to pick the single most likely one. The cost also grows linearly with $K$ while the accepted tokens grow geometrically more slowly, so longer MTP drafts stop paying: an independent FP8 ablation on another Spark measured 17.7 tokens per second at 3 drafts, 18.6 at 8 and 13.4 at 15.

DFlash2: drafting a block in one pass

DFlash2 removes the sequential loop by drafting all $K$ tokens in a single forward pass. Its drafter is a five-layer block-diffusion model: the positions to be drafted enter as mask tokens, and the drafter predicts all of them at once, conditioned on hidden states taken from five layers of the target model (layers 5, 19, 33, 47 and 61 in the drafter’s configuration). Reusing the target’s representations of the context is what keeps a five-layer network accurate. A single pass over a block of masked positions would normally produce independent guesses that need not form a coherent sequence, so DFlash2 keeps the 16 most likely candidates at every position and runs a small selector that traces one consistent path through them. Its backbone also uses two-tap dynamic convolutions, which the authors introduce to stop draft quality decaying towards the end of the block.

The measurements show the two properties that make this design pay. The drafter’s cost does not depend on $K$, because one pass drafts the whole block, so lengthening the draft from 7 to 10 tokens added accepted tokens without adding draft passes: tokens per step rose from 4.78 to 5.35 while the step lengthened by only 2.4 ms, from 112.1 to 114.5 ms, for verifying three more positions, and the median request rose from 49.1 to 56.5 tokens per second. The acceptance probability per position also stayed constant at about 0.84, so the gain from a longer draft is exactly what the geometric formula predicts and nothing in the drafter degraded at the far end of the block. The stock drafter is not small, however. It occupies 3.85 GB in BF16, a fifth of the target’s own bytes, and that is where Chisel’s changes start.

A profile of the step shows where its 116 ms go. The 64 target layers verify eleven positions in 87.7 ms, only 6.2 ms more than the 81.5 ms they take for a single token, which is the property speculative decoding relies on, measured directly. The target’s output layer takes 5.6 ms, the drafter’s five layers take 16.7 ms, and the drafter then reads the full 1.27 GB output layer a second time, another 5.6 ms, to score its candidates. Every phase runs at 220 to 240 GB/s, so the byte account that predicted the plain step also predicts each part of this one.

Shaving the step: Chisel’s three changes

The last two stages leave acceptance unchanged, at 5.35, 5.32 and 5.32 tokens per step, and make each step cheaper, from 114.5 to 105.0 to 97.9 ms. All three changes are lossless with respect to the served tokens, because each alters only the drafter or the bookkeeping around verification, never the target’s own computation of its next token.

The first change is the drafter itself. Chisel quantizes the DFlash2 drafter’s linear layers to NVFP4, which shrinks it from 3.85 GB to 1.36 GB, and the step time falls by 9.5 ms with no measurable change in acceptance; in the profile the drafter’s layers drop from 16.7 to 5.7 ms. The second is a reduced vocabulary for the draft head. A draft token needs only to be good enough to be accepted, and the tokens generated in English chat, code and structured output come from a small fraction of Qwen’s 248,320-entry vocabulary, so Chisel gives the drafter its own output layer containing only the 100,032 most-used rows, stored in FP8, which is 0.51 GB instead of 1.27, and the profiled draft head falls from 5.6 to 2.3 ms. When the correct next token lies outside that set, the draft at that position is wrong and the target supplies the token itself, so the restriction can reduce acceptance but never change the output, and the verify pass still scores the full vocabulary.

The third change follows from the architecture of Qwen3.8. Only 16 of its 64 layers use full attention; the other 48 are Gated DeltaNet layers, a form of linear attention that replaces the growing key-value cache with a fixed-size recurrent state per head, updated at every token $t$ as

where $a_t$ is a learned decay gate, $\beta_t$ a learned write strength, and $k_t$ and $v_t$ the token’s key and value vectors. Each of the 48 layers holds 48 such states of $128 \times 128$ values, which in BF16 is 75.5 MB per snapshot across the model. A verify pass over $K+1$ positions does not know in advance how many drafts will be accepted, so stock vLLM stores the state after every position, eleven snapshots or about 0.83 GB of writes per step at $K = 10$, and keeps only one. Chisel’s plugin stores the small per-token inputs instead, and once acceptance is known it replays the recurrence up to the accepted position and writes that state once, producing a state bit-identical to vLLM’s. In the profile the target’s layers take 2.9 ms less with replay enabled and the replay itself takes 0.7 ms. The reduced vocabulary and the state replay together take a further 7.1 ms off the step, which brings the full recipe to 97.9 ms per step, 54.3 tokens per second weighted by token and 67.8 for the median request.

What the six stages measure

All six stages ran on one Spark on the same day with the same vLLM build (vLLM 0.31.1rc1, pinned by digest), the same weights and the same server flags. The benchmark is Chisel’s own: 25 prompts covering chat, code, schema-constrained JSON and one story with about 34,000 tokens of context, greedy decoding, 256 output tokens, one request at a time, three repeats that agree to within 0.4%. Our full-recipe median of 67.8 tokens per second is 4% above Chisel’s published 65.1 on the same metric, and the likely reason is the GPU clock, which Chisel capped at 2,000 MHz and which averaged about 2,490 MHz during our runs.

The headline rate hides a twofold spread across prompt types, because acceptance depends on how predictable the output is. Under the full recipe, code decodes at 13.7 ms per token and schema-constrained JSON at 13.2 ms, about 73 and 76 tokens per second, while open-ended chat takes 26.2 ms and the long story 26.9 ms, about 38 tokens per second. Code and JSON carry long stretches of text that are nearly determined by what precedes them, such as closing brackets, repeated identifiers and field names, and the drafter gets those stretches right far more often, producing about 6.7 tokens per step on code and 7.3 on JSON against 4.0 on chat. The median time to the first token is 241 ms.

What these numbers do not cover

Every measurement in this post is for one user. Speculative decoding converts spare arithmetic into extra tokens, and a batched server has less spare arithmetic to convert, so the advantage of the drafters shrinks as concurrency rises; Chisel’s two-user measurement at 16,000 tokens of context gives 58.0 tokens per second in total, and an independent FP8 study reports that at 16 concurrent users a 4-bit checkpoint with its built-in MTP head overtakes DFlash2. Decoding was greedy, and with sampling the acceptance rule changes from an equality test to rejection sampling, which typically accepts fewer drafts. The quality figures are Chisel’s measurements on these weights, which we have not repeated.

References

1. NVIDIA (2025). DGX Spark hardware specifications. GB10 Grace Blackwell superchip, 128 GB unified LPDDR5x memory, 273 GB/s memory bandwidth.
2. NVIDIA (2025). Introducing NVFP4 for efficient and accurate low-precision inference. The E2M1 element format, 16-value blocks with E4M3 scales and the per-tensor FP32 scale.
3. Leviathan, Y., Kalman, M. and Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. The draft-verify scheme and the expected tokens per step.
4. Yang, S., Kautz, J. and Hatamizadeh, A. (2024). Gated Delta Networks: Improving Mamba2 with Delta Rule. The Gated DeltaNet recurrence used in Qwen3.8’s linear-attention layers.
5. Chen, J., Liang, Y. and Liu, Z. (2026). DFlash: Block Diffusion for Flash Speculative Decoding, and the DFlash2 drafter for Qwen3.8-27B.
6. yuanlin2004 (2026). Chisel and chisel-spark-recipes. The recipe, the NVFP4 drafter, the chisel-passes plugin, the benchmark and the BF16 quality comparison.
7. Unsloth (2026). Qwen3.8-27B-NVFP4. The mixed NVFP4 and FP8 checkpoint used for every stage.
8. 0xBakeer (2026). Qwen3.8-27B-FP8 on a single DGX Spark. The independent FP8 ablation of MTP draft lengths and concurrency.