Breaking the Scale-Out Barrier: Zero-Degradation AI Inference Optimization on a 32+ Accelerator Single-Server Node

Yes, we know 32+ accelerators on a single server is not normal. The industry standard eight accelerators dominate the market with little variance — but what if that paradigm is long overdue for a reset when it comes to inference and fine-tuning?

Large-scale training is bandwidth bound, but LLM inference scaling and fine-tuning deserve their own optimized ecosystem. In this post we review a recent study I/ONX was commissioned to conduct to prove out the impact of a completely re-imagined hardware and software orchestration approach, where a single server runs multiple large language models across dozens of accelerators — deployed in a headless chassis such that zero memory, storage, or CPU on the local chassis were used for the tests.

The Symphony SixtyFour supports up to 64 accelerators on a single server. This commissioned test had 34 accelerators configured at the time of the benchmark sweeps.

Executive Overview: Density as the Ultimate Cost-Saver

As enterprises and cloud providers push to operationalize increasingly massive LLMs, the industry's default response has been “scale-out” — buying endless racks of conventional 8-GPU servers and wiring them together with fragile, expensive InfiniBand or high-speed Ethernet fabrics. An internationally known NeoCloud provider commissioned I/ONX to prove an alternative theory: what if we could scale-up instead?

The mission was to validate whether the I/ONX orchestration platform could maintain enterprise-grade throughput and rock-solid workload isolation on a severely overloaded, single-node architecture — configured with an unprecedented 34 disparate accelerators on a single bare-metal host. The results were resounding: the I/ONX system successfully served an enormous Qwen3.5-397B-A17B (FP8) model under maximum stress with virtually zero performance degradation.

• Massive CapEx reduction: consolidating workloads into ultra-dense nodes bypasses the “host tax” entirely, saving up to $1.5M per inference cluster.
• Operational simplicity: removing multi-node network fabric translates to significant power, software, and human-capital savings — projecting up to $2M in OpEx savings over 3 years.
• Vendor fluidity: proven, unified orchestration of AMD, Tenstorrent, and Furiosa silicon operating simultaneously inside the exact same server.

The Technical Challenge: Beating the “Noisy Neighbor”

When you crowd 34 accelerators onto a single motherboard PCIe fabric, conventional wisdom says you will cripple performance. Shared resources — CPU threading, PCIe lanes, RAM bandwidth, and thermal envelopes — inevitably collide. Traditional platforms struggle with “noisy neighbor” degradation, where engaging a secondary accelerator grinds the primary workload's inter-token latency to a halt.

Test Environment & Topology

• Target model: Qwen/Qwen3.5-397B-A17B-FP8
• Inference engine: SGLang (v0.5.9-rocm720-mi30x)
• Attention backend: Triton (AITER compiled)
• Tensor Parallelism (TP): 8 · ISL: 8192 · OSL: 1024
• Max concurrency & max running requests: 64 · Total prompts: 640 (burst profiling)

The single bare-metal server was populated with 16x AMD MI300X GPUs (two distinct 8-GPU ‘Hives’), 16x Tenstorrent accelerators, and 2x Furiosa NPUs.

Benchmark Execution & Results

Phase 1 — Single Hive Baseline

Running strictly on Hive 0 (8x MI300X) with no secondary workloads: 822.10 tok/s output throughput, 74.22 ms median TPOT — the pure, uninhibited monolithic performance of the MI300X serving a 397B-parameter model.

Serving Benchmark Result — Single Hive Baseline
Successful requests:            640
Benchmark duration (s):         717.59
Output token throughput (tok/s) 822.10
Total token throughput (tok/s): 7410.19
Median TTFT (ms):               492.33
Median TPOT (ms):               74.22
Median ITL (ms):                44.37

Phase 2 — Dual Hive Isolation

Spinning up Hive 1 (8x MI300X) under identical load, concurrent with Hive 0: Hive 0 held 813.02 tok/s and Hive 1 held 804.64 tok/s — minimal-to-zero noisy-neighbor penalty. Two massive Qwen instances ran side-by-side in distinct logical hives with completely isolated pipelines.

Phase 3 — Extreme Stress Test (34 Accelerators Active)

While both MI300X hives continued serving their 64-concurrency Qwen workloads, the orchestrator fully engaged the remaining 16 Tenstorrent chips and 2 Furiosa NPUs with competing tasks on the same server. Under the maximum thermal, PCIe, and core-thread stress conceivable for a single host, degradation remained incredibly narrow.

Serving Benchmark Result — Phase 3 (34 XPU active)
Hive 0 output throughput (tok/s):  805.12
Hive 1 output throughput (tok/s):  789.46
Hive 0 median ITL (ms):            44.52
Hive 1 median ITL (ms):            45.50

The Engineering Value of a Single-Server Configuration

1. 01Skipping the network layer: fitting 64 concurrent streams of a 397B model into a single server bypasses cross-rack fabric entirely — the data never leaves the PCIe bus.
2. 02Unified fleet orchestration: I/ONX handles the translation, runtime, and resource allocation for AMD, Tenstorrent, and Furiosa inside one unified OS layer, targeted natively as a single server.
3. 03Absolute workload isolation: diverse silicon types co-habit the same motherboard without stepping on one another's memory or PCIe bandwidth.

The benchmarks prove that the easiest way to scale inference and fine-tuning out is to scale up. By centralizing management on the intense single-node capabilities of I/ONX orchestration, engineering teams deploy models faster, cheaper, and with far less complexity on ultra-dense, single-server nodes.