Bud LayerZero
The production-grade kernel orchestration framework for LLM inference — one API across every GPU-optimised kernel library, with automatic, explainable, nanosecond-scale selection per operation, per shape, per device. Open source under Apache 2.0.
Kernel fragmentation, solved once.
Kernel libraries multiply faster than anyone can integrate them, and every serving framework rebuilds the same selection logic — picking differently for identical inputs. LayerZero makes that decision once, makes it visible, and makes it cost effectively nothing: call lz.attention() and the fastest available kernel executes.
lz.explain() returns the candidates, their scores, and what filtered them outBetween your model and the metal.
Your framework calls one API. LayerZero resolves it to the best available kernel on the hardware you're actually running — and every layer of the Bud AI OS above runs on that surface.
Calls in — inference pipelines, custom stacks, and training loops calling ordinary PyTorch: lz.attention() and friends. Serving frameworks and custom stacks alike.
Dispatches to — the fastest available kernel for this operation, on this device, at this shape. CUDA, ROCm, Intel XPU, and CPU, through standardised backend adapters.
Policy, cache, filter, score, select — then dispatch. LayerZero is the selection engine between the call and the kernel: it decides, dispatches through a standardised adapter, and records the outcome for future scoring.
Automatic, explainable, overridable.
Six capability groups — the unified API, the selection engine, the transparency and control on top of it, and the coverage and hardening that make it deployable.
Six steps. Sub-microsecond.
Every call runs the same pipeline — and short-circuits the moment it can. A locked kernel skips everything; a warm cache answers in under 100 nanoseconds.
The selection pipeline — every call
result → flashinfer.decode · next identical call → cache hit
The backend network
Each backend sits behind a standardised adapter. LayerZero routes the operation to whichever one wins — and falls back to torch SDPA whenever none qualifies. All backends are optional; install only the kernel libraries you actually have.
| Backend | Role | Notes |
|---|---|---|
| FlashAttention | IO-aware attention | v2 / v3 / v4 · NVIDIA SM ≥ 80 |
| FlashInfer | Serving-optimised attention | paged KV-cache · GQA |
| xFormers | Memory-efficient attention | sparse patterns |
| Liger Kernel | Fused Triton kernels | norms & beyond — not an attention backend |
| oneDNN · ZenDNN · IPEX | CPU & Intel GPU paths | the hot path isn't GPU-only |
| Torch SDPA | Universal fallback | always present — selection never fails closed |
The policy system — four tiers, worked example
Rules apply in strict priority order: Lock, then Allow, then Deny, then Boost. The YAML file is watched and hot-reloaded, so policy changes reach a live fleet with zero downtime.
# policy.yaml — hot-reloaded version: "1.0" # pin the kernel on A100+ locks: - operation: "attention.causal" backend: "flashinfer" condition: "sm >= 80" # no slow fallbacks in prod denies: - operation: "attention.*" backends: ["torch_sdpa"] condition: "batch > 32" # prefer, don't force boosts: - operation: "attention.*" backend: "flashinfer" boost_factor: 1.5
Same control from code: lz.lock(), lz.prefer(), lz.disabled() — and lz.which() to see what would run before it runs.
Zero required dependencies. Four dispatch modes.
Python 3.11+ with every backend optional — bring only the kernel libraries you actually have. Choose how much flexibility you pay for, from frozen build-time selection to live hot-reloaded policy.
Install & run
PyTorch native, Apache 2.0. Every backend is optional; graceful degradation to torch SDPA means a minimal install still runs everything.
Hardware
The same call resolves differently on every device — that's the point. FlashAttention on SM ≥ 80, ZenDNN on AMD CPUs, oneDNN and IPEX on Intel.
Observability
Telemetry exporters ship with the framework, and pre-deployment validation plus JIT warmup means the first request isn't the slow one.
Dispatch modes — pay only for the flexibility you use
| Mode | Behaviour | Cost |
|---|---|---|
| Static | Selection resolved at compile time and baked in. | 0ns overhead |
| Dynamic | Runtime selection, cached after the first call. | 100–500ns |
| Hot-reload | Config file is watched; changes apply without a restart. | 0s downtime |
| Config-driven | YAML rules with conditional logic for production fleets. | no code change |
A/B kernels in production. Because policy hot-reloads, you can pin a kernel, roll one back, or compare two on a live fleet — with no restart and no redeploy.
Every headline number, with its basis.
A selection layer is only worth having if it costs nothing to ask. These are the framework's design targets for the selection path, and the engineering that backs them.
The latency figures are design targets for LayerZero's own overhead — the cost of deciding which kernel runs, not the kernel's execution time.
Three things a dispatch table can't give you.
The alternative to LayerZero is what every framework does today: a hand-rolled, hard-coded kernel dispatch table. Here is what changes.
| Capability | Hand-rolled dispatch | Bud LayerZero |
|---|---|---|
| Adding a new kernel library | Custom integration per framework | ✓ One adapter — every caller benefits |
| Consistency across stacks | Each stack picks differently | ✓ One answer for identical inputs |
| Explaining a selection | Read the source, guess | ✓ Candidates, scores, filter reasons |
| Changing kernels in production | Code change + redeploy | ✓ Hot-reloaded YAML policy |
| Failure handling | Ad hoc | ✓ Circuit breakers + graceful fallback |
| Selection overhead | Fixed at whatever was written | ✓ 0ns static · <500ns dynamic |
| Hardware awareness | Per-device branches, hand-maintained | ✓ Device, dtype, shape — automatic |
Anywhere kernel choice moves the needle.
If you serve tokens, the kernel decision is on your hot path — whether you've automated it or not. Six places LayerZero earns its keep.
Serving framework builders
Replace your in-house selection logic with one engine — and stop re-integrating every new kernel library by hand.
Custom inference stacks
Get FlashAttention-class performance without owning dispatch code — call ordinary PyTorch and let selection happen underneath.
Production fleet operators
Pin, roll back, or A/B kernels across a live fleet through hot-reloaded YAML — with circuit breakers catching failures before they become outages.
Kernel & performance engineers
lz.explain() turns every selection into an inspectable report — candidates, scores, and filter reasons instead of archaeology.
Mixed-hardware estates
The same call resolves correctly on CUDA, ROCm, Intel XPU, and CPU — including oneDNN, ZenDNN, and IPEX paths off the GPU.
Real-time & quantised workloads
Sampling, quantisation (INT8, FP8, GPTQ, AWQ), tokenizers, and real-time audio via pre-allocated ring buffers — the whole hot path, orchestrated.
Open source, ready to run.
LayerZero is open source under Apache 2.0 — the fastest way to evaluate it is to run it. Or return to the product overview for the story at a glance.
GitHub
LayerZero on GitHub
Apache 2.0 · PyTorch native
Bring your stack, keep your kernels, and let the selection layer disappear into the background. Zero required dependencies — every backend optional.
View the repositoryBack to overview
Bud LayerZero
One API that picks the fastest kernel
Return to the high-level product page — the selection console and the pipeline at a glance.
Back to the product pagePut your data on it.
The fastest way to see what an integrated AI operating system does for your enterprise is a proof-of-concept on your infrastructure, with your data.