DIY 8x NVIDIA GB10 Cluster Achieves Local Kimi LLM Inference
Sonic Intelligence
An 8-node NVIDIA GB10 cluster successfully ran massive Kimi LLMs locally, exceeding official support.
Explain Like I'm Five
"Imagine you have a super-powerful toy computer (NVIDIA GB10) that can think very fast. Someone figured out how to connect eight of these together, even though the company only said you could connect four. They used a special super-fast internet cable (400GbE switch) to make them all talk to each other really quickly. This allowed them to run a giant brainy program (Kimi LLM) right on their own setup, like having a super-smart robot brain in your house!"
Deep Intelligence Analysis
Central to this achievement is the meticulous selection and integration of advanced networking components. Each NVIDIA GB10 unit, featuring a 'Grace Blackwell' SoC with 20 Arm cores, a Blackwell-generation GPU, 128GB LPDDR5X memory, and ConnectX-7 200GbE networking, provides a formidable foundation. The deployment of a low-cost MikroTik CRS804 DDQ 4-port 400GbE switch was instrumental in enabling RDMA networking for RoCE and NCCL scaling, allowing the eight nodes to communicate with the necessary bandwidth and low latency. This specialized networking infrastructure is the linchpin for aggregating the compute power of multiple GB10 units into a cohesive, high-throughput cluster capable of demanding AI workloads.
The implications of such custom-built, high-density AI clusters are substantial. For researchers and enterprises, it offers the strategic advantage of sovereign control over their AI models and data, reducing reliance on external cloud providers and enhancing data privacy. This approach facilitates rapid iteration and experimentation with large models, unconstrained by cloud egress fees or API limitations. Furthermore, it highlights a growing trend towards specialized, on-premise AI infrastructure, signaling a future where advanced AI capabilities are increasingly accessible to those with the technical expertise to assemble and optimize these powerful systems, driving further innovation in both hardware and software for local AI deployment.
Visual Intelligence
flowchart LR
A["NVIDIA GB10 Unit"] --> B["ConnectX-7 200GbE"]
B --> C["MikroTik 400GbE Switch"]
C --> D["8x GB10 Cluster"]
D --> E["Run Kimi K2.5/2.6"]
E --> F["Local LLM Inference"]
Auto-generated diagram · AI-interpreted flow
Impact Assessment
This custom 8-node NVIDIA GB10 cluster demonstrates the feasibility of pushing beyond vendor-supported configurations to achieve high-performance local inference for massive LLMs. It highlights the critical role of advanced networking in scaling AI compute and offers a blueprint for researchers and enterprises seeking sovereign control over their AI workloads.
Key Details
- An 8-node NVIDIA GB10 cluster was built with 1TB of memory and 160 Arm cores.
- The cluster utilized a low-cost MikroTik CRS804 DDQ 4-port 400GbE switch for RDMA networking.
- Each NVIDIA GB10 unit features a 'Grace Blackwell' with 20 Arm cores, a Blackwell GPU, 128GB LPDDR5X memory, and ConnectX-7 200GbE networking.
- The cluster successfully ran large language models Kimi K2.5 and Kimi K2.6 locally.
- NVIDIA GTC 2026 increased official GB10 node support to 4, but the custom build achieved 8 nodes.
Optimistic Outlook
The successful deployment of an 8-node GB10 cluster, running models like Kimi K2.5/2.6 locally, showcases the immense potential for custom, high-density AI infrastructure. This capability empowers researchers and developers to conduct cutting-edge AI work without reliance on cloud services, fostering innovation and data privacy for large-scale model experimentation.
Pessimistic Outlook
Building and maintaining such a complex, unsupported cluster requires significant technical expertise and resource investment, limiting its accessibility to most users. The reliance on pushing hardware beyond official vendor support introduces potential stability and compatibility risks, which could lead to unforeseen operational challenges and debugging complexities for long-term deployments.
Get the next signal in your inbox.
One concise weekly briefing with direct source links, fast analysis, and no inbox clutter.
More reporting around this signal.
Related coverage selected to keep the thread going without dropping you into another card wall.
