Bud Decision Runtime is an inference server for decision models: models that read a piece of text, such as an email, a ticket or an agent's trace, and answer set questions about it (pick an option, give a score, yes or no). It runs four model families, Laya, Julia, GLiNER2.5-Decide and Kev, behind one HTTP API, on NVIDIA GPUs and on CPUs, and configures itself for the hardware it finds.
This post measures it against the engines you could serve these models with today: the publisher's own server and its hand-fused fast path, PyTorch's own compiler, and vLLM for the Kev models, which are built on Qwen. Same machines, same requests, same client. The runtime answers one Laya question in 7.4 ms over HTTP on an RTX 3050 Laptop GPU, is faster in most tests, and keeps its speed as clients are added. On Laya, its answers matched the publisher's original model on 99.8% of 1,800 labelled examples.
Measured 10 to 16 November 2026 on an RTX 3050 Laptop, a GTX 1650, a Xeon 6767P, a Core i7-11370H and an Apple M5, against vLLM 0.32. Every figure is the engine at its defaults; nothing was tuned by hand.
Request sizes used throughout: short is a 30-word state with 1 question (about 50 tokens); typical is 60 words and 3 questions (about 240 tokens); long is 300 words and 8 questions (about 2,600 tokens). States from 1K tokens up to each model's full context have their own section.
What a decision runtime does
The text a decision model reads is called its state, and each question has a fixed answer type: pick one of these options, score this from 1 to 5, yes or no. Laya and Julia are ModernBERT encoders with an option-marker head, GLiNER2.5-Decide is a DeBERTa model, and Kev is a family of LoRA adapters with a pointer head on Qwen bases, from 0.5B to 9B parameters (Qwen2.5, Qwen3 and the hybrid Qwen3.5), that can keep a state's computation and reuse it for later questions.
The runtime serves all four behind one HTTP API. A Rust front end tokenises requests and packs their rows into passes; a device loop runs each pass on a GPU or CPU. Most of the speed comes from how passes are planned and how each pass is computed.
How it gets there
- It decides its own configuration on the hardware it finds. At start the engine probes the device: which kernels pass a self-test there, which 16-bit type multiplies fastest (timed, not read from a table: a GTX 1650 reports a T4's compute capability but multiplies float16 at 0.16× float32), and which eight-bit engine is correct on this CPU. Nothing is tuned by hand: every number in this post is the engine at its defaults.
- It sizes passes from measurement. At load it times passes of growing size and picks the shortest within 5% of the best throughput, and a larger one while a backlog waits. On a CPU a large model reads all its weights once per pass, so there the backlog pass may run well past the interactive target when the curve shows it pays: Kev-8B's went from 112 tokens to 1,024. Every pass size it will plan is captured as a CUDA graph before it reports ready.
- Rows are packed, not padded. Requests of different lengths share a pass with no padding between them. Attention runs inside each row, so a row's answer is the same bits whatever shares its pass.
- Multiplies are pinned and correctly rounded. One cuBLASLt algorithm defines each weight's bits; per band of pass sizes the engine uses the fastest algorithm that gives exactly those bits, timed on the card at load.
- On a CPU, the small steps are compiled C++. A decoder layer's per-token steps (norms, rotations, SwiGLU), attention for short rows and Qwen3.5's delta rule run in C++ operators built on the host for its own instruction set at first start. As chains of PyTorch operators they cost more in dispatch than in arithmetic: a short Kev-0.6B request went from 59 to 20 ms a pass, and a Kev-0.8B pass over a 2,300-token document from 1,362 to 455 ms.
- The CPU encoder is compiled. Laya's and Julia's encoder runs as one graph compiled by PyTorch's inductor with frozen, packed weights and attention inside the graph; one graph serves every pass size.
- Queues are bounded by time and shared fairly. A request its deadline cannot survive is refused at once with a 503 and
retry-after, each tenant queues apart, and a row asked again within a minute is answered from memory.
How we measured
Each engine served Laya on the same machine, driven by the same load generator: a fixed number of clients, each sending its next request as soon as the last is answered, every state new. Accuracy comes from one benchmark with six frozen labelled sets of 300 examples each (topic, emotion, sentiment, entailment, prompt injection, long reviews), scored against the publisher's own model in float32.
The publisher's engines ran in 16 bits as configured (autocast on the GPU, LAYA_CPU_AMP=bf16 on the Xeon). The runtime ran at its defaults, which chose bfloat16 on both.
Scaling with concurrent clients
A single request says little about a server. What matters is how throughput grows as clients are added, and what each of them waits. On typical requests:
- The publisher's server does not scale. laya-serve answers one request at a time. Its throughput is flat from one client to 48 (17.2 requests a second on the GPU, 11.5 on the Xeon) and its latency grows in step with the queue, to 2.8 seconds at 48 clients.
- The runtime's throughput rises with clients, and its latency is lower at every load. On the GPU it reaches 44 requests a second at four clients and 46 at 48, with a median wait of 75 ms at four clients against 232 ms for laya-serve. On the Xeon it goes from 20.8 at one client to 39.7 at 16.
| Request | Clients | laya-serve | Runtime | Runtime vs laya-serve |
|---|---|---|---|---|
| short | 1 | 36.6/s · 27 ms | 93.3/s · 10.7 ms | 2.5× |
| typical | 1 | 17.2/s · 58 ms | 36.2/s · 28 ms | 2.1× |
| typical | 4 | 17.2/s · 232 ms | 44.4/s · 75 ms | 2.6× |
| typical | 16 | 17.2/s · 930 ms | 46.3/s · 289 ms | 2.7× |
| typical | 48 | 17.2/s · 2.8 s | 46.2/s · 1.0 s | 2.7× |
| long | 1 | 3.2/s · 314 ms | 4.9/s · 206 ms | 1.5× |
| long | 4 | 3.2/s · 1.25 s | 5.0/s · 743 ms | 1.6× |
| long | 16 | 3.2/s · 5.0 s | 5.0/s · 2.5 s † | 1.6× |
| long | 48 | 3.2/s · 15.1 s | 5.0/s · 2.6 s † | 1.6× |
Table 1 — RTX 3050 Laptop: requests answered per second and the median latency of answered requests. † The runtime refused the rest at once with a 503 (see "Past saturation" below). laya-serve was measured on 10 November 2026, the runtime on 12 November.
| Request | Clients | laya-serve | Runtime | Runtime vs laya-serve |
|---|---|---|---|---|
| short | 1 | 14.2/s · 67 ms | 43.5/s · 21 ms | 3.1× |
| typical | 1 | 9.6/s · 101 ms | 20.8/s · 47 ms | 2.2× |
| typical | 4 | 11.5/s · 343 ms | 27.5/s · 141 ms | 2.4× |
| typical | 16 | 11.5/s · 1.37 s | 39.7/s · 388 ms | 3.5× |
| long | 1 | 2.9/s · 338 ms | 4.3/s · 227 ms | 1.5× |
| long | 4 | 3.0/s · 1.31 s | 4.7/s · 813 ms | 1.6× |
Table 2 — Xeon 6767P, 16 threads, same measures. laya-serve's long-request runs stopped at four clients, so the runtime's did too.
Past saturation
When more work arrives than a device can handle, a server has to either queue it or refuse some of it. laya-serve queues everything, so with 48 clients sending long requests a request waits 15 seconds. The runtime holds the work it accepts to what it can finish within its queue limit (two seconds of interactive work by default) and refuses the rest at once with a 503 and retry-after, which a client or a gateway can send to another replica.
It answers 1.6 times as many requests a second as laya-serve (5.0 against 3.2), and the ones it accepts wait 2.6 seconds instead of 15. In these runs the clients resend immediately after a refusal, so the refusal counts measure the load generator, not lost work.
In the decision benchmark's overload test, 256 clients sending typical requests, the runtime still answered 57.8 requests a second on the RTX 3050 (median 2.2 s) and 55.8 on the Xeon (2.7 s): throughput holds far past the point where every client could be served at once.
Against vLLM, on the Kev models
Kev's models are LoRA adapters on Qwen base models, which vLLM already serves. If a general engine serves them as fast, there is no reason to use a separate runtime for them. So we put each Kev model's base behind vLLM 0.32 on the same machine and sent the same requests.
vLLM takes prompts, not decision requests. Kev's model answers each question looking only at the state and at that question, so to get Kev's answers a client sends vLLM one prompt per question, the state followed by that question, all at once, and vLLM's prefix cache shares what it can of the state between them. The runtime takes the request as it is: it computes the state once, and each question attends to the state and to itself only.
vLLM has no pointer head, so our wrapper reads one next token per prompt instead, slightly less work than Kev's readout; only its speed counts. Both engines ran in bfloat16. On the Xeon each got the same 16 cores of one memory node, with the load generator on others. Each point is 30 seconds of closed-loop load at that many clients, every request new, measured 15 and 16 November 2026.
- The runtime is ahead at every load, on both machines. It answers 1.75 to 7 times as many requests a second as vLLM with one client, and 1.26 to 6.9 times as many with 4 to 64. A typical Kev-0.6B request takes 11.8 ms on the RTX 3050 against 29 ms, and 27 ms on the Xeon against 70.
- The gap is widest on Qwen3.5. vLLM's prefix cache shares a Qwen3.5 state only in whole blocks of 544 tokens on the GPU and 640 on the CPU, longer than these states, so it computes the state again for every question. On Kev-0.8B's long requests the runtime serves 7 times as many with one client on the Xeon, and 5.6 times as many at 16 clients on the RTX 3050.
- It is closest under heavy load on the dense models. There both engines spend each pass on the same multiplies, and vLLM shares most of a dense model's state between questions: 1.26 times on Kev-0.6B's long requests at 64 clients on the RTX 3050, and 1.51 times on Kev-8B's at 64 clients on the Xeon.
| Machine · model | Clients | Runtime | vLLM | Runtime vs vLLM |
|---|---|---|---|---|
| RTX 3050 Laptop Kev-0.6B (Qwen3) | 1 | 84.1/s · 12 ms | 34.0/s · 29 ms | 2.47× |
| 4 | 88.4/s · 45 ms | 52.9/s · 76 ms | 1.67× | |
| 16 | 96.4/s · 166 ms | 74.6/s · 214 ms | 1.29× | |
| 64 | 102.6/s · 661 ms | 81.3/s · 787 ms | 1.26× | |
| RTX 3050 Laptop Kev-0.8B (Qwen3.5) | 1 | 58.2/s · 17 ms | 18.7/s · 58 ms | 3.11× |
| 4 | 65.1/s · 61 ms | 27.4/s · 146 ms | 2.38× | |
| 16 | 74.3/s · 215 ms | 34.2/s · 466 ms | 2.17× | |
| 64 | 78.9/s · 860 ms | 34.0/s · 1.7 s | 2.32× | |
| Xeon 6767P, 16 cores Kev-0.6B (Qwen3) | 1 | 35.2/s · 27 ms | 13.1/s · 70 ms | 2.69× |
| 4 | 44.7/s · 87 ms | 21.1/s · 186 ms | 2.12× | |
| 16 | 78.5/s · 203 ms | 30.7/s · 519 ms | 2.55× | |
| 64 | 88.8/s · 779 ms | 33.7/s · 1.9 s | 2.63× | |
| Xeon 6767P, 16 cores Kev-0.8B (Qwen3.5) | 1 | 25.9/s · 37 ms | 8.62/s · 106 ms | 3.01× |
| 4 | 38.6/s · 102 ms | 13.4/s · 291 ms | 2.87× | |
| 16 | 50.5/s · 311 ms | 16.7/s · 937 ms | 3.03× | |
| 64 | 52.2/s · 1.2 s | 20.9/s · 3.0 s | 2.50× | |
| Xeon 6767P, 16 cores Kev-8B (Qwen3) | 1 | 3.50/s · 283 ms | 1.42/s · 657 ms | 2.46× |
| 4 | 5.00/s · 791 ms | 2.10/s · 1.9 s | 2.38× | |
| 16 | 7.75/s · 2.0 s | 3.66/s · 4.3 s | 2.12× | |
| 64 | 7.78/s · 8.2 s | 4.12/s · 15.2 s | 1.89× |
Table 3 — typical requests: requests answered per second and the median latency of answered requests, bfloat16 for both engines. The ratio is the runtime's requests per second over vLLM's. No request failed in any run.
| Machine · model | Clients | Runtime | vLLM | Runtime vs vLLM |
|---|---|---|---|---|
| RTX 3050 Laptop Kev-0.6B (Qwen3) | 1 | 21.7/s · 46 ms | 12.4/s · 82 ms | 1.75× |
| 4 | 23.3/s · 172 ms | 17.4/s · 230 ms | 1.34× | |
| 16 | 25.3/s · 632 ms | 19.9/s · 802 ms | 1.27× | |
| 64 | 25.3/s · 2.5 s | 20.1/s · 3.2 s | 1.26× | |
| RTX 3050 Laptop Kev-0.8B (Qwen3.5) | 1 | 17.4/s · 57 ms | 3.16/s · 318 ms | 5.51× |
| 4 | 18.3/s · 218 ms | 3.57/s · 1.1 s | 5.13× | |
| 16 | 20.0/s · 801 ms | 3.56/s · 4.6 s | 5.61× | |
| 64 | 19.9/s · 3.2 s | 3.56/s · 17.7 s | 5.60× | |
| Xeon 6767P, 16 cores Kev-0.6B (Qwen3) | 1 | 11.4/s · 85 ms | 6.19/s · 146 ms | 1.85× |
| 4 | 15.2/s · 256 ms | 8.52/s · 460 ms | 1.79× | |
| 16 | 18.5/s · 850 ms | 9.86/s · 1.6 s | 1.87× | |
| 64 | 18.3/s · 3.4 s | 9.96/s · 6.4 s | 1.84× | |
| Xeon 6767P, 16 cores Kev-0.8B (Qwen3.5) | 1 | 11.4/s · 85 ms | 1.63/s · 592 ms | 7.00× |
| 4 | 12.9/s · 302 ms | 1.88/s · 2.1 s | 6.87× | |
| 16 | 13.5/s · 1.2 s | 2.02/s · 7.8 s | 6.67× | |
| 64 | 13.5/s · 4.7 s | 2.03/s · 30.1 s | 6.65× | |
| Xeon 6767P, 16 cores Kev-8B (Qwen3) | 1 | 1.58/s · 623 ms | 0.70/s · 1.3 s | 2.26× |
| 4 | 1.91/s · 2.1 s | 1.02/s · 3.8 s | 1.87× | |
| 16 | 1.89/s · 8.4 s | 1.24/s · 12.6 s | 1.52× | |
| 64 | 1.90/s · 32.1 s | 1.26/s · 44.0 s | 1.51× |
Table 4 — long requests (300 words, 8 questions), same measures as Table 3.
All six Kev models on a CPU
Kev's family runs from 0.5B to 9B parameters on three kinds of Qwen base. On the 16-core Xeon we measured all six; Kev-4B, 8B and 9B do not fit the 4 GB laptop GPU. The runtime is 1.26 to 3.4 times as fast on a new request, 1.9 to 17.6 times as fast on follow-up questions to a document it has seen, and serves 1.46 to 6.1 times as many requests at 16 clients.
| Model (base) | One typical request | 2,300-token document, 5 questions | 5 more questions on it | 16 clients, typical | 16 clients, long |
|---|---|---|---|---|---|
| Kev-0.5B (Qwen2.5) | 31 / 59 ms | 229 / 369 ms | 46 / 88 ms | 91.7 / 40.9 | 26.2 / 11.4 |
| Kev-0.6B (Qwen3) | 29 / 72 ms | 333 / 468 ms | 54 / 122 ms | 74.5 / 29.5 | 17.5 / 10.0 |
| Kev-0.8B (Qwen3.5) | 39 / 99 ms | 0.42 / 1.42 s | 38 / 641 ms | 50.4 / 17.1 | 12.5 / 2.04 |
| Kev-4B (Qwen3.5) | 151 / 410 ms | 1.97 / 5.10 s | 0.15 / 2.57 s | 11.4 / 5.14 | 2.85 / 0.54 |
| Kev-8B (Qwen3) | 288 / 656 ms | 2.34 / 2.94 s | 277 / 707 ms | 7.39 / 3.55 | 1.77 / 1.21 |
| Kev-9B (Qwen3.5) | 297 / 715 ms | 2.66 / 8.72 s | 0.26 / 4.31 s | 6.82 / 3.15 | 1.69 / 0.32 |
Table 5 — runtime / vLLM, bfloat16, Xeon 6767P with 16 cores, 15 and 16 November 2026. Latency is the median of one client at a time; the last two columns are requests answered per second. The runtime's memory of answers was off, so nothing here was answered from memory. "5 more questions" asks new questions about a document the engine has seen: the runtime keeps the document's state at any length, while vLLM's prefix cache works only in whole blocks, 640 tokens long on Qwen3.5 on a CPU.
On the GPU, the same comparison
| Model (base) | One typical request | 2,300-token document, 5 questions | 5 more questions on it | 16 clients, typical | 16 clients, long | 256 clients, typical |
|---|---|---|---|---|---|---|
| Kev-0.5B (Qwen2.5) | 10.7 / 24.7 ms | 177 / 212 ms | 15.6 / 42.6 ms | 120.2 / 88.2 | 30.9 / 25.7 | 125.9 / 97.8 |
| Kev-0.6B (Qwen3) | 11.9 / 30.4 ms | 250 / 284 ms | 19.4 / 56 ms | 96.6 / 73.6 | 25.1 / 20.3 | 101.2 / 81.4 |
| Kev-0.8B (Qwen3.5) | 17.1 / 58.4 ms | 252 / 340 ms | 19 / 103 ms | 74.8 / 34.2 | 20.1 / 3.6 | 78.4 / 34.1 |
Table 6 — runtime / vLLM, bfloat16, RTX 3050 Laptop 4 GB, 13 and 14 November 2026, the runtime's memory of answers off.
The one load vLLM wins
Identical requests sent again with the runtime's memory of answers turned off, on the small GPU. vLLM's prefix cache then also holds a repeated prompt's question tokens, while the runtime's state cache holds the state only: on Kev-0.6B at 64 clients over 8 repeated requests, vLLM answers 132.6 requests a second against the runtime's 108.5. On the Xeon the runtime wins this load too, by 3 times. With the memory on, which is the default, the runtime answers repeated requests about 30 to 1,800 times as fast as vLLM.
Longer states, up to each model's full context
The requests above carry states of 50 to 2,600 tokens. Decision models also read whole contracts, long tickets and agent traces, so we sent each engine one new state at a time, from 1K tokens (K = 1,024) up to the model's full context: 32K for the Qwen2.5 and Qwen3 models and 256K (262,144 tokens) for Qwen3.5. Each request asks five questions about its state, and five other questions on the same state follow it.
The states are the benchmark's long reviews, end to end, numbered so that no two are alike. Lengths stopped growing once a new request took more than two minutes; one request at the full context then showed whether the engine could serve it at all. One client at a time, bfloat16, on the same GPU and the same 16 Xeon cores as above.
- Follow-up questions are faster at every length. The runtime keeps a state's computation at any length, so five more questions cost little. On the RTX 3050 it answers them 2.0 to 2.8 times as fast as vLLM on the dense models and 4.8 to 18 times on Kev-0.8B. On the Xeon it is 6.4 to 27 times as fast on the three Qwen3.5 models and up to 3.5 times on the dense ones, level only on a 4K Kev-0.5B state (62 ms against 63). At 32K on Kev-0.6B on the Xeon the questions take 247 ms against vLLM's 501.
- New Qwen3.5 states are faster on the Xeon at every length, and at the longest on the GPU. On the Xeon a new Kev-0.8B state is 4.5 times as fast at 1K and 1.69 times at the full 256K (195 s against 329); Kev-4B 4.4 times at 1K and 1.61 times at 256K (617 s against 996); Kev-9B 3.9 and 1.66 times (748 s against 1,245). On the RTX 3050 Kev-0.8B is 2.8 times as fast at 1K, level at 8K, behind by 6 to 10% at 16K and 32K, and 1.74 times as fast at 64K (19.7 s against 34.2).
- On long dense states, vLLM computes a new state faster. The runtime leads on Qwen2.5 and Qwen3 states up to 2K tokens on the GPU and 4K to 8K on the Xeon. Beyond that vLLM leads, by up to 1.15 times on the GPU (Kev-0.6B at 8K: 1.44 s against 1.66) and 1.11 times on the Xeon (Kev-0.5B at 32K: 10.4 s against 11.6; Kev-8B: 67.9 s against 70.6). At these lengths attention is most of the work, and vLLM's attention kernels are faster than the ones the runtime uses there; on the CPU that is PyTorch's flash attention.
- Both engines reach the full context on the Xeon. The runtime computes a state longer than 8,192 tokens in segments (4,096 on a GPU of 8 GB or less), each continuing what the earlier ones kept, so its state cache sets the longest state, not the memory of one pass. vLLM computes a long prompt a few thousand tokens at a time. On the 4 GB GPU, Kev-0.8B reached 64K on both engines, Kev-0.5B its full 32K on both, and Kev-0.6B 10K on the runtime and 8K on vLLM.
| Machine · model | Full context | Longest served, runtime / vLLM | A new state that long, runtime / vLLM | Where it stopped |
|---|---|---|---|---|
| RTX 3050 Laptop Kev-0.5B (Qwen2.5) | 32K | 32K / 32K | 6.6 / 6.4 s | Full context on both |
| RTX 3050 Laptop Kev-0.6B (Qwen3) | 32K | 10K / 8K | 2.3 / 1.4 s | Runtime: the state cache the card holds (1,200 MB, 112 KB a token). vLLM: 16K refused, longer than the 12,288-token max-model-len the card holds |
| RTX 3050 Laptop Kev-0.8B (Qwen3.5) | 256K | 64K / 64K | 19.7 / 34.2 s | Runtime: the state cache the card holds (880 MB). vLLM: 65,536 refused, the most max-model-len the card holds |
| Xeon 6767P Kev-0.5B (Qwen2.5) | 32K | 32K / 32K | 11.6 / 10.4 s | Full context on both |
| Xeon 6767P Kev-0.6B (Qwen3) | 32K | 32K / 32K | 17.6 / 16.2 s | Full context on both |
| Xeon 6767P Kev-0.8B (Qwen3.5) | 256K | 256K / 256K | 195 / 329 s | Full context on both |
| Xeon 6767P Kev-4B (Qwen3.5) | 256K | 256K / 256K | 617 / 996 s | Full context on both |
| Xeon 6767P Kev-8B (Qwen3) | 32K | 32K / 32K | 70.6 / 67.9 s | Full context on both |
| Xeon 6767P Kev-9B (Qwen3.5) | 256K | 256K / 256K | 748 / 1,245 s | Full context on both |
Table 7 — how far each engine went, and the time for one new state of that length. On the 4 GB GPU memory sets the limit: vLLM got the longest max-model-len it would accept on the card, and the runtime a state cache within the half of free memory it allows. On the Xeon each run could use 64 GB.
| Machine · model | 1K | 4K | 8K | 16K | 32K | 64K | 128K | 256K |
|---|---|---|---|---|---|---|---|---|
| RTX 3050 Laptop Kev-0.5B (Qwen2.5) | 87 ms 115 ms | 396 ms 370 ms | 914 ms 853 ms | 2.4 s 2.2 s | 6.6 s 6.4 s | |||
| RTX 3050 Laptop Kev-0.6B (Qwen3) | 115 ms 142 ms | 602 ms 553 ms | 1.7 s 1.4 s | |||||
| RTX 3050 Laptop Kev-0.8B (Qwen3.5) | 136 ms 384 ms | 523 ms 722 ms | 1.1 s 1.2 s | 2.7 s 2.6 s | 7.0 s 6.4 s | 19.7 s 34.2 s | ||
| Xeon 6767P Kev-0.5B (Qwen2.5) | 129 ms 231 ms | 453 ms 713 ms | 1.3 s 1.5 s | 3.8 s 3.5 s | 11.6 s 10.4 s | |||
| Xeon 6767P Kev-0.6B (Qwen3) | 176 ms 315 ms | 695 ms 841 ms | 2.0 s 1.9 s | 5.5 s 5.1 s | 17.6 s 16.2 s | |||
| Xeon 6767P Kev-0.8B (Qwen3.5) | 240 ms 1.1 s | 812 ms 2.1 s | 1.9 s 4.2 s | 4.0 s 7.8 s | 9.3 s 16.0 s | 22.8 s 39.2 s | 63.2 s 106 s | 195 s 329 s |
| Xeon 6767P Kev-4B (Qwen3.5) | 932 ms 4.1 s | 3.6 s 7.8 s | 7.3 s 15.2 s | 15.8 s 28.1 s | 34.3 s 56.6 s | 80.1 s 132 s | 210 s – | 617 s 996 s |
| Xeon 6767P Kev-8B (Qwen3) | 1.4 s 2.0 s | 4.5 s 5.5 s | 11.6 s 11.7 s | 27.6 s 26.5 s | 70.6 s 67.9 s | |||
| Xeon 6767P Kev-9B (Qwen3.5) | 1.8 s 7.1 s | 5.2 s 12.9 s | 10.4 s 24.5 s | 23.0 s 44.6 s | 48.6 s 88.4 s | 105 s 194 s | 263 s – | 748 s 1,245 s |
Table 8 — a new state with five questions, one client: the runtime on top, vLLM below in grey. Each figure is the faster of two requests at that length (one beyond 32K); the first request at a new length can carry costs a running server pays once.
| Machine · model | 1K | 4K | 8K | 16K | 32K | 64K | 128K | 256K |
|---|---|---|---|---|---|---|---|---|
| RTX 3050 Laptop Kev-0.5B (Qwen2.5) | 13 ms 36 ms | 20 ms 50 ms | 30 ms 69 ms | 50 ms 114 ms | 94 ms 207 ms | |||
| RTX 3050 Laptop Kev-0.6B (Qwen3) | 15 ms 30 ms | 27 ms 75 ms | 41 ms 96 ms | |||||
| RTX 3050 Laptop Kev-0.8B (Qwen3.5) | 18 ms 314 ms | 24 ms 283 ms | 26 ms 126 ms | 37 ms 191 ms | 56 ms 416 ms | 97 ms 529 ms | ||
| Xeon 6767P Kev-0.5B (Qwen2.5) | 47 ms 81 ms | 62 ms 63 ms | 67 ms 107 ms | 85 ms 174 ms | 130 ms 309 ms | |||
| Xeon 6767P Kev-0.6B (Qwen3) | 57 ms 103 ms | 60 ms 134 ms | 84 ms 186 ms | 136 ms 298 ms | 247 ms 501 ms | |||
| Xeon 6767P Kev-0.8B (Qwen3.5) | 52 ms 734 ms | 43 ms 655 ms | 46 ms 1.1 s | 52 ms 1.2 s | 71 ms 999 ms | 188 ms 2.1 s | 331 ms 3.7 s | 609 ms 3.9 s |
| Xeon 6767P Kev-4B (Qwen3.5) | 169 ms 3.2 s | 156 ms 2.6 s | 161 ms 4.4 s | 187 ms 4.2 s | 241 ms 2.8 s | 448 ms 5.4 s | 855 ms – | |
| Xeon 6767P Kev-8B (Qwen3) | 288 ms 1.0 s | 283 ms 788 ms | 331 ms 988 ms | 450 ms 881 ms | 601 ms 1.2 s | |||
| Xeon 6767P Kev-9B (Qwen3.5) | 287 ms 5.6 s | 270 ms 4.3 s | 269 ms 6.4 s | 311 ms 6.2 s | 332 ms 4.4 s | 635 ms 6.5 s | 824 ms – |
Table 9 — five more questions on that state: the runtime on top, vLLM below in grey. 32K is 32,512 tokens on the Qwen2.5 and Qwen3 models, which leaves room for the questions in their 32,768; 256K is 261,888 tokens; vLLM's 64K on the GPU is 65,280. A blank cell is a length that engine did not reach or was not asked; on Kev-4B and Kev-9B vLLM went from 64K straight to the full context, because a new request at 64K had taken more than two minutes.
Against Laya's own fast path, on the same GPU
Laya's publisher ships a fast path for CUDA (laya[fast]): TileLang kernels that fuse the multiplies with the activation, norms and rotary positions, run with CUDA graphs. Its published 4.6 ms for one question was measured on an RTX 4070 Ti SUPER. We installed it on our RTX 3050 Laptop and timed both in the same process, on the same requests. The runtime's time covers everything its engine does per request apart from HTTP: tokenising in Rust, the copy to the GPU, the captured graph and the copy back.
| Request | Runtime | Laya stock, bf16 | Laya fast, bf16 | Runtime vs fast path |
|---|---|---|---|---|
| 1 question, short state | 6.8 ms | 23.9 ms | 9.4 ms | 1.39× |
| 3 questions, short state | 16.7 ms | 40.0 ms | 23.1 ms | 1.39× |
| 8 questions, long state | 290 ms | 446 ms | 335 ms | 1.16× |
| 30 questions, short state | 122 ms | 184 ms | 167 ms | 1.38× |
Table 10 — RTX 3050 Laptop, 4 GB, medians of 40 calls after warm-up. "Short state" is two sentences; "long state" is 28. The fast path in fp16 measured 9.5, 23.3, 341 and 170 ms on the same four requests, within 2% of bf16. It serves one request at a time; the concurrency comparison above uses the publisher's server.
Against PyTorch's compiler, on a CPU
On a CPU the strongest general alternative is PyTorch itself: Hugging Face's ModernBERT-large compiled by inductor with its weights frozen, which is what Intel recommends for CPU inference. We timed Laya's encoder each way on the same threads, in turn, over five rounds, on rows of one length and on a pass of real requests of mixed lengths.
Packing is where the runtime wins. On passes of rows that are all one length, which traffic rarely sends, there is no padding to remove, and the stock compiled model keeps a 5 to 10% lead on the larger ones.
| Pass | PyTorch eager | PyTorch compiled | Runtime |
|---|---|---|---|
| Real requests, mixed lengths (1,276 tokens) | 329.4 ms | 256.4 ms | 161.4 ms |
| 64 tokens, one short row | 49.0 ms | 17.3 ms | 16.6 ms |
| 1,024 tokens, eight rows of one length | 111.7 ms | 88.3 ms | 97.6 ms |
| 3,584 tokens, eight 448-token rows | 309.2 ms | 243.8 ms | 257.0 ms |
Table 11 — one encoder pass on the Xeon 6767P, 16 threads, bfloat16 on AMX, median of five interleaved rounds.
On every machine
The same engine, at its defaults, on each machine and model family we could reach. On each one it chose its own precision and kernels.
| Machine · model, precision chosen | Short | Typical | Long | Answers changed |
|---|---|---|---|---|
| RTX 3050 Laptop, 4 GB Laya, bfloat16 | 7.4 ms | 20.4 ms | 208 ms | 0.2% |
| RTX 3050 Laptop Julia, bfloat16 | 2.2 ms | 4.2 ms | 35 ms | 1.9% |
| RTX 3050 Laptop GLiNER2.5-Decide, bfloat16 | 7.2 ms | 11.1 ms | 48 ms | 0.3% |
| RTX 3050 Laptop Kev, bfloat16 | 7.2 ms | 10.3 ms | 37 ms | 1.0% |
| GTX 1650, 4 GB, no tensor cores Laya, float32 | 44 ms | 129 ms | 1.70 s | 0% |
| Xeon 6767P, 16 threads, AMX Laya, bfloat16 | 24.6 ms | 45.8 ms | 232 ms | 0.2% |
| Xeon 6767P, 16 threads Julia, bfloat16 | 25.6 ms | 35.8 ms | 61 ms | 2.1% |
| Core i7-11370H, 4 threads, no AMX Laya, float32 | 183 ms | 494 ms | 6.3 s | 0% |
| Core i7-11370H, 4 threads Laya, int8 (opt-in) | 66 ms | 206 ms | 3.3 s | 0.8% |
Table 12 — median latency of one caller at a time, from the decision benchmark. "Answers changed" is the share of 1,800 labelled examples whose winning option differs from the publisher's float32 model; Julia's and GLiNER's references are the engine's own float32, which matches their libraries within 0.0001. Laya's throughput on typical requests was 13,421 tokens a second on the RTX 3050 and 12,322 on the Xeon. The engine also runs on an Apple M5, where it chose bfloat16 on the GPU and float32 on the CPU; we did not benchmark it there.
Against published figures
Laya's publisher reports single-question latency on several devices. None of them is a machine we had, so the table below puts each figure on its own hardware rather than claiming a like-for-like win. Our short request (30 words, 1 question, about 51 tokens) is close to the publisher's 72-token case.
| Engine | Device | Source | One question |
|---|---|---|---|
| Laya fast path | RTX 4070 Ti SUPER | published | 4.6 ms |
| Bud Decision Runtime, bfloat16, over HTTP | RTX 3050 Laptop, 4 GB | measured | 7.4 ms |
| Bud Decision Runtime, bfloat16 | Xeon 6767P, 16 threads | measured | 24.6 ms |
| Laya, model card | Tesla T4 | published | 39.5 ms |
| Bud Decision Runtime, float32 | GTX 1650, 4 GB | measured | 44.2 ms |
Bud Decision Runtime, int8 | Core i7-11370H, 4 threads | measured | 66.3 ms |
| Bud Decision Runtime, float32 | Core i7-11370H, 4 threads | measured | 182.8 ms |
| Laya | Ryzen 9 6900HX, 8 threads | published | 329 ms |
| Laya, float32 | EPYC 9R14, 4 cores | published | 580 ms |
Table 13 — median milliseconds for one question on a short state; each row is on its own hardware. Published figures are from Laya's BENCHMARKS.md: the model card's T4 figure, its TileLang fast path on an RTX 4070 Ti SUPER (a card with several times the 3050 Laptop's compute), and CPU figures in float32.
The publisher's 4.6 ms on an RTX 4070 Ti SUPER is lower than our 7.4 ms, on a card with about 3.5 times the memory bandwidth, which is what bounds a single question. Measured on the same card (Table 10), the runtime is 1.39 times as fast as that fast path. On CPUs the gap is wide: a 4-core EPYC 9R14 takes 580 ms in float32, and our 4-thread laptop i7 takes 183 ms in float32 and 66 ms in int8.
On GPUs the best published throughput for a model of this size is text-embeddings-inference on an H100, at about 38% of the card's nominal speed. The runtime turns 74% of the RTX 3050's measured multiply ceiling into Laya tokens. We have not yet run it on an H100.
Accuracy against the original models
A faster engine is only useful if the answers stay the same. On Laya's six labelled sets the runtime in 16 bits scores 80.4% on the RTX 3050 and 80.3% on the Xeon, against 80.4% for the publisher's float32 model, and changes 0.2% of winning options.
We also compared each family against its publisher's own code, on requests none of the benchmark sets contain. In 16 bits the runtime is at least as close to float32 as the publisher's own 16-bit path for Laya, GLiNER2.5-Decide and Kev. Julia is the exception on one measure: its mean change is lower than the library's, but it flips more winners.
| Family | float32, largest difference | 16-bit mean change, runtime / library | Winners changed, runtime / library |
|---|---|---|---|
| Laya | 0.00005 | 0.0018 / 0.0018 | 1.7% / 2.2% |
| Julia | 0.00005 | 0.0090 / 0.0095 | 3.0% / 2.5% |
| GLiNER2.5-Decide | 0.000001 | 0.0004 / 0.0006 | 0.0% / 0.9% |
| Kev | 0.00001 | 0.0026 / 0.0039 | 1.0% / 3.1% |
Table 14 — each family against its publisher's own library; probabilities as a client receives them, on an RTX 3050. Julia is the least confident of the four (65.5% on the benchmark), so near-ties flip. On NVIDIA GPUs the runtime gives a request the same bits whatever shares its pass.
The faster CPU path was held to the same rule. On the Xeon, all six Kev models were checked against Kev's own answers recorded on that machine by Kev's library. In float32 every one is within 0.00006 and no winning option changed. In bfloat16 every one is closer to Kev's float32 answers than Kev's own bfloat16 is, in mean and 99th percentile.
| Kev model, Xeon 6767P | float32, largest difference | bfloat16 mean change, runtime / Kev's own | bfloat16 99th percentile, runtime / Kev's own |
|---|---|---|---|
| Kev-0.5B (Qwen2.5) | 0.000013 | 0.0072 / 0.0097 | 0.043 / 0.053 |
| Kev-0.6B (Qwen3) | 0.000052 | 0.0034 / 0.0055 | 0.027 / 0.031 |
| Kev-0.8B (Qwen3.5) | 0.000009 | 0.0018 / 0.0028 | 0.009 / 0.018 |
| Kev-4B (Qwen3.5) | 0.000001 | 0.0012 / 0.0021 | 0.004 / 0.008 |
| Kev-8B (Qwen3) | 0.000009 | 0.0038 / 0.0082 | 0.021 / 0.057 |
| Kev-9B (Qwen3.5) | 0.000001 | 0.0016 / 0.0028 | 0.007 / 0.011 |
Table 15 — 454 questions for the three smaller models and 244 for the larger ones. One Kev-0.8B question, one that magnifies 16-bit drift, moved by 0.068 in bfloat16 in one run against a bound of 0.055, so on a CPU the engine serves Kev-0.8B in float32 by default, where it fits.
Bring the models, the hardware and the latency target. We will show you the runtime on your own machines, at its defaults.
What surprised us
The 16-bit multiply was not correctly rounded
The algorithm search picked split-K kernels for Laya's attention-output and feed-forward shapes, which add partial sums in bfloat16. Measured against float32, that was 4 to 13.5 units in the last place. Excluding split-K brought every shape to correct rounding (0.5 units) and put the runtime's answers closer to float32 than the publisher's own 16-bit path. It also showed that most algorithms without split-K give identical bits, so the engine can pick the fastest of them per pass size without breaking the rule that a row's answer never depends on its pass.
Compute capability does not say which precision is fast
A GTX 1650 reports the same capability as a T4 but has no tensor cores: it multiplies float16 at 0.16× and bfloat16 at 0.88× the speed of float32. The engine times each type at start instead of trusting the number, and runs that card in float32.
Eight-bit multiplies overflow on older x86 CPUs
On x86 processors without VNNI, PyTorch's oneDNN and fbgemm engines sum pairs of eight-bit products in 16 bits, and the sums saturate: 13% error on a Comet Lake i5, against 1.6% with seven-bit inputs. The engine now checks its own eight-bit path at start, picks the input width that is correct, and refuses int8 where it would not be faster than float32.
More CPU executors do not mean more throughput
Four executors of four threads do 1.3 to 2.6 times the work of one executor of sixteen at the same pass size. At the same pass time, which is what a latency target fixes, they do the same work, a single request takes twice as long, and the idle ones slow the busy one by a quarter. One executor is the default.
The right CPU baseline is PyTorch's compiler, not a published figure
We had aimed for about 1,100 tokens a second per core, derived from Intel's published BERT-large figures. Intel's table gives sentences a second but no sequence length, so no per-token figure follows from it. The comparison that holds is the one above: PyTorch's stock compiled model on the same machine.
An answer cache needs load that does not repeat itself
Our load generator seeded each client the same way on every run, and the benchmark's warm-up sent the same requests as the timed run. With answers remembered, half of one run came from memory. Both now draw new requests every time, except the test that is meant to repeat.
A latency cap cut throughput for a large model on a CPU
The engine caps a pass at 300 ms by default so that a request arriving mid-pass does not wait long. Kev-8B on 16 Xeon cores reads its 16 GB of weights once per pass, so a 300 ms pass held 112 tokens, one request: under 16 clients it served 3.2 short requests a second. Passes of 1,024 tokens take 1.2 s but serve 829 tokens a second instead of 392. The engine now takes such passes while a backlog waits, where the curve it measures shows a fifth more throughput, and served 5.9.
The compiler said the loop was vectorised; it was not
The CPU delta rule needs exp on every channel of every token. GCC's report said the loop was vectorised, but the machine code called the scalar expf once per element: 3 ns an element, and 9 ns for a hand-written polynomial. Written on the compiler's vector types at the machine's own width, the same polynomial takes 1 ns. The report is not the code; read the assembly.
vLLM's prefix cache needs whole blocks on Qwen3.5
On Qwen3.5 a block must also hold a page of the delta rule's state, so vLLM's blocks are 544 tokens on the GPU and 640 on the CPU. A state shorter than one block is never cached, which is most decision requests, so vLLM computes it again for every question; a longer one has its last partial block recomputed for every question. The runtime keeps a state by its exact tokens at any length, which is where its lead of 17 times on follow-up questions to a Qwen3.5 model on the CPU comes from.
Measured 10 to 16 November 2026. Load runs used one load generator against each engine's HTTP endpoint, including vLLM 0.32's concurrency sweeps; accuracy and single-caller latency come from the runtime's decision benchmark, whose per-run result files, including example-by-example probabilities, are saved as baselines. Published figures for Laya are from its BENCHMARKS.md; the H100 embedding figure is from RunPod's text-embeddings-inference benchmark.
