A
DEMO · FP8 WEIGHTS
Qwen2.5-72B
FP8 weights and BF16 KV cache. Run 1 optimizes latency. Run 2 optimizes latency, then throughput while preserving the earlier limit.
MODELS
Different models.
Different demands.
Meet the large model in the demo and the two smaller models used in the separate shared-GPU test.
INSIDE THE DEMO
A model for each test.
One physical GPU.
The demo tunes Qwen2.5-72B on one RTX Pro 6000 Blackwell. Eight exact-answer JSON tasks, a 99% quality floor, and concurrency 1/2/4/8 stay fixed while the swarm searches.
B
SHARED-GPU TEST · BF16 WEIGHTS
Qwen3-0.6B
The smaller service in the separate two-model test. It must pass its own answer checks and shared-device latency limit.
C
SHARED-GPU TEST · FP8 WEIGHTS
GLM-4-9B
The second service in the sharing test. Both services must pass quality and stay within 10% of their isolated p95 latency.
Hardware: one 96 GB RTX Pro 6000 on Molab. The separate sharing test declares a 24 GiB memory budget; it does not emulate a smaller card’s compute speed. An earlier joint run passed; the latest repeat missed GLM’s latency limit.
THE EXPERIMENT MATTERS
A model is only
part of the workload.
Request rate, input length, output length, and burstiness all affect serving behavior. Sera evaluates the configuration against the workload and requirements supplied in the experiment.
Prompts and responses.
Long prompts concentrate work in prefill. Long responses keep decode active for more steps.
Memory and device time.
A model’s footprint matters, but sharing also introduces competition for device time.
Requirements for each model.
A shared configuration is accepted only when both models clear their individual gates.
These are the measured single-GPU paths. Real 2/4/8-GPU validation is not complete. The demo’s ordered stages are implemented and tested; a live staged throughput gain is not yet established.
SEE THE LOOP IN ACTION