GPU sharing in multi-tenant cloud environments requires efficient resource isolation without sacrificing performance. FCSP — Fixed Capacity Spatial Partition — is a user-space GPU virtualization framework that achieves sub-microsecond memory enforcement and deterministic compute throttling through lock-free data structures and hierarchical token bucket rate limiting.
Unlike existing solutions that rely on semaphore-based synchronisation, FCSP employs C11 atomics with cache-line-aligned structures to eliminate contention bottlenecks. Evaluated with the GPU-Virt-Bench suite, it achieves 1000× faster context creation, 3600× faster memory limit enforcement, and 3× better multi-tenant isolation than HAMi-core, the current state-of-the-art open-source GPU sharing solution.
The GPU sharing problem
The proliferation of GPU-accelerated machine learning has created unprecedented demand for efficient GPU resource sharing. Modern data centres deploy thousands of high-end accelerators to serve diverse workloads — LLM inference, training, batch processing. The cost of those accelerators, roughly $10,000 to $40,000 per unit, makes multi-tenancy the only way to justify the capital.
Hardware-based approaches
- Multi-Instance GPU (MIG). Splits a GPU into as many as seven isolated instances with dedicated memory and compute, providing strong hardware-level isolation — but requiring a full GPU reset to reconfigure.
- vGPU. Hypervisor-based virtualization for virtual machines, requiring enterprise licensing and adding significant overhead for containerised workloads.
- SR-IOV. Enables PCI passthrough for VMs, but is limited to virtual machine environments and incompatible with containers.
Software-based approaches
- Time-slicing. The default multi-process service provides no memory isolation, and its fairness depends on application behaviour.
- HAMi-core. The industry-standard open-source approach:
LD_PRELOAD-based interception with semaphore-coordinated shared memory. - KubeShare. Kubernetes-native GPU sharing using a similar interception approach.
Where the existing software methods break down
Our analysis of HAMi-core reveals four fundamental architectural limitations.
The contention bottleneck
HAMi-core uses a single POSIX semaphore to protect the shared memory region. Under multi-tenant load this creates severe contention. On the OH-006 lock contention benchmark with four concurrent processes, P99 latency reached 94.1 ms against a mean of 1.9 ms.
A 94ms P99 is catastrophic for real-time inference with a sub-100ms SLA. The mean looks fine — 1.9ms — which is exactly what makes this failure mode hard to catch before production.
O(N) process scanning
Every memory allocation triggers a linear scan of all process slots to calculate aggregate usage, under a global semaphore lock. With 1,024 process slots, that scan dominates allocation latency — the cost of an allocation becomes a function of how many tenants exist, not of the allocation itself.
Feedback-driven rate limiting
HAMi-core's compute throttling relies on NVML polling to adjust token refill rates, on a 120ms polling interval. That feedback loop introduces up to 120ms of latency between a limit violation and its enforcement, allowing temporary oversubscription in the window.
Context creation overhead
On the OH-004 context creation benchmark, HAMi-core costs 84,000µs against native CUDA's 82,000µs. Negligible for long-running processes — prohibitive for serverless and auto-scaling deployments where cold starts must complete inside 500ms.
What FCSP does differently
- Lock-free shared memory architecture. An inter-process coordination mechanism using C11 atomics that eliminates the P99 latency spikes seen in semaphore-based approaches.
- Hierarchical per-stream rate limiting. A two-tier token bucket that provides per-stream compute isolation while maintaining device-level fairness guarantees.
- Crash-resilient process management. A heartbeat-based reaper pattern that automatically recovers resources from crashed processes without administrator intervention.
- Stream-aware throttling with NCCL bypass. Workload classification that preserves collective communication performance while enforcing compute limits on regular kernels.
Those follow from five design goals: memory limit checks completing in under 1µs so allocation-heavy workloads like KV cache management are unaffected; lock-free hot paths giving O(1) latency regardless of tenant count; deterministic compute limiting using a predictable mathematical model rather than feedback control; crash resilience so process failures neither leak resources nor corrupt shared state; and specific handling for attention patterns, KV cache allocation, and NCCL collectives.
Architecture
FCSP is a user-space interposition layer between ML applications and the CUDA driver. No kernel modules, no driver modifications, no hardware partitioning. It enforces isolation by intercepting GPU API calls, applying policy, and forwarding allowed operations to the native stack.
GPU usage flows as: application → CUDA runtime → FCSP → CUDA driver → GPU. Injection is via LD_PRELOAD, so relevant CUDA and NVML entry points are hooked transparently without application changes.
The four modules
The Memory Tracker provides fast, deterministic memory accounting across all tenant processes on a node. It intercepts allocation and free operations and maintains per-process usage, per-GPU global usage, and allocation metadata for correct deallocation attribution. Allocations are admitted or rejected using atomic updates rather than global locks.
The Kernel Rate Limiter enforces compute isolation by controlling kernel launch admission. Rather than relying on slow utilisation feedback, it uses a rate-based token bucket model to regulate how fast kernels are issued. This converts best-effort sharing into a predictable mechanism that limits noisy neighbours and stabilises tail latency under dense tenancy.
The Stream Classifier labels stream categories — especially communication-focused streams such as NCCL — so appropriate policies apply. NCCL streams are excluded from compute throttling to avoid distributed synchronisation collapse, where throttling one rank's collectives degrades the entire job.
The Process Manager registers processes into shared accounting slots, maintains liveness via heartbeats, and reclaims resources when a process crashes. This prevents ghost allocations from permanently degrading GPU capacity after a failure.
The lock-free shared memory region
All modules coordinate through a shared memory region mapped into every participating process. It is mmap'd for a consistent node-wide view, mlock'd to avoid paging delays in the enforcement hot path, 4KB page-aligned, cache-line padded to reduce false sharing, and atomics-based to avoid global lock hot spots.
It stores the minimal state enforcement needs: per-GPU memory totals, per-process usage counters, rate limiter state, and heartbeat timestamps with slot ownership signals. Three design decisions matter:
- 64-byte cache-line alignment. Every frequently-accessed atomic field is padded to a full cache line to prevent false sharing on multi-socket systems.
- Separated heartbeat array. Heartbeats live apart from process slots because they are written by a dedicated thread at 1Hz, while slots are updated on every allocation — different write frequencies should not share a cache line.
- Running totals. Per-device counters are maintained atomically, which is what eliminates the O(N) scan entirely.
The allocation fast path
Allocation tracking is O(1) and lock-free: fetch the cached slot from thread-local storage, pre-check against the limit with a relaxed atomic load, allocate optimistically with a fetch-add, double-check to catch the race, then update the per-process counter and record the allocation for later free attribution.
The memory ordering is deliberate. The relaxed load is only a hint — a stale value is acceptable because the recheck catches it. The fetch-add uses acquire-release so it both observes prior allocations and publishes its own. The recheck resolves the time-of-check-to-time-of-use race between the two. Together these guarantee the sum of allocations never exceeds the device limit, even under concurrent allocation storms.
Thread-local slot caching keeps this cheap: the cached pointer is validated against the current PID to detect a fork, and the fast path is a TLS dereference plus a comparison — around two CPU cycles.
Hierarchical per-stream rate limiting
FCSP runs a two-tier token bucket. The device bucket sizes its capacity from SM count × threads per SM × a token factor, refilling in proportion to the configured SM limit. Each stream bucket takes a share of the device bucket's capacity and refill rate.
Kernel cost is computed from grid and block dimensions, scaled by a dampening factor, with a throttle penalty added for non-NCCL streams proportional to how far below 100% the SM limit sits.
Our benchmarks found that slight under-throttling — charging 90% of true cost — improves multi-stream efficiency, because it creates natural synchronisation barriers between streams.
That result is why the dampening factor is configurable rather than hard-coded at 1.0.
When a stream must wait, FCSP busy-waits with exponential backoff rather than calling sched_yield(), which costs 1–2µs in context switch overhead. Backoff starts at 50ns, uses the PAUSE instruction for power and SMT-scheduling benefit, and doubles up to a 10µs cap.
Stream classification and NCCL bypass
Streams are classified so throttling can match the work. NCCL streams bypass throttling entirely. Attention streams, being memory-bound, get 20% reduced throttling. Memory-copy operations get 50% reduced throttling. Everything else takes the standard path. NCCL streams are detected by hooking communicator initialisation and marking the stream it will use.
Crash recovery
Each FCSP-enabled process runs a lightweight heartbeat thread that publishes a timestamp at 1Hz with release ordering. A single reaper per node checks whether processes are alive, verifies the heartbeat timeout on those that are not, and performs atomic cleanup on timed-out slots.
Cleanup runs in three phases: claim the slot exclusively with a compare-exchange against a sentinel value, subtract the dead process's usage from global totals for every device, then release the slot behind a memory fence. The sentinel is what prevents another process claiming the slot mid-cleanup.
Implementation surface
FCSP hooks 47 CUDA Driver API functions and 12 NVML functions, intercepting dlsym and consulting an internal hook table, with a thread-local guard to prevent infinite recursion. The hooked surface covers memory management, kernel launch, device and context management, stream management, and the NVML calls that report memory and utilisation.
Individual allocations are tracked in a lock-free hash map — 4,096 buckets, size and device packed into a single 64-bit field, an FNV-1a hash on the pointer value, acquire-semantics traversal, and compare-and-swap insertion with tagged pointers to prevent the ABA problem.
To reduce atomic traffic further, allocation deltas accumulate in thread-local state and flush on one of three conditions: 64 accumulated operations, a ±4MB delta on any device, or 10ms elapsed. This cuts global atomic operations by 10–100× for allocation-heavy workloads.
A 30-minute walkthrough on your hardware mix, governance constraints, and top use case.
Conclusion
FCSP demonstrates that high-density GPU sharing can deliver strong multi-tenant isolation without sacrificing latency-sensitive performance. Replacing semaphore-coordinated shared memory with C11-atomic, cache-line-aligned lock-free structures removes the contention hot spots and enables sub-microsecond memory enforcement at scale.
Deterministic hierarchical token-bucket throttling stabilises compute fairness without feedback-loop lag, while stream-aware policies preserve the communication paths distributed workloads depend on. Together these unlock higher tenant density with predictable degradation and better operational resilience.
- A single semaphore is what produces the 94ms P99 — the mean latency hides it completely.
- Running totals in atomic counters remove the O(N) scan, making allocation cost independent of tenant count.
- Rate-based throttling beats feedback control because it has no 120ms enforcement lag.
Comparative figures are measured with the open-source GPU-Virt-Bench suite against HAMi-core; benchmark IDs (OH-004, OH-006) refer to that suite's metric catalogue. Speed-up multiples are ratios of the paired measurements quoted alongside them and will vary with GPU, driver version, tenant count and workload mix — reproduce on your own hardware before designing capacity against them. GPU unit prices are indicative market ranges at the time of writing. The infrastructure-saving figure quoted in the original abstract is a model extrapolated from density gains at 1,000-GPU scale, not an observed customer saving, and is omitted here pending a published methodology.
