Our ML server: why we chose the NVIDIA RTX PRO 6000 Blackwell
In this article
In June 2026 we put our own ML server into operation. Every training run, every experiment and every customer inference has run on it since. The central decision while building it was the choice of GPU — and it became the NVIDIA RTX PRO 6000 Blackwell. Why we buy GPUs instead of renting GPU hours, why we picked this particular card, and why the missing NVLink interconnect does not bother us — that is what this post is about.
Why own GPUs
The obvious answer for an AI startup is the cloud: no capital expense, no maintenance, compute on demand. Three arguments spoke against a cloud solution for us.
Data sovereignty. We work with our customers’ sales, inventory and ordering data — data that reveals revenue, margins and supplier relationships. That data does not leave our house. It is processed on our own cards, on a server in Germany. This is not a marketing line; for many of our conversations it is the precondition for starting at all.
Cost under sustained load. Rented GPUs pay off when you need them sporadically. We need them continuously: training runs at night, hyperparameter searches run for days, inference runs permanently anyway. At that utilization, purchased cards amortize within a manageable period — and after that, an additional experiment costs nothing but electricity.
Speed of experimentation. The underrated point. When a GPU hour costs money, you think twice before starting an experiment. When the cards are sitting in the rack doing nothing anyway, you just start it.
Why the RTX PRO 6000 Blackwell
The real decision was not “buy or rent” but “which card”. The candidates were the usual data-center accelerators — the NVIDIA H100 above all — and the RTX PRO 6000 Blackwell in its Server Edition, the passively cooled variant built for rack mounting.
The number that decided it for us is its graphics memory: 96 GB of GDDR7 with ECC, connected through a 512-bit interface. Why exactly that number tips the scales becomes clear further down, where NVLink comes up — it is the reason we avoid an entire class of problems.
Beyond that, what mattered was the ratio of compute, memory and price. At the time of purchase the card cost us about a third of what a comparable data-center accelerator would have — while carrying more memory: 96 GB versus 80 GB on an H100. On single-GPU throughput it matches or beats the far more expensive card in independent comparisons.
That is the decisive arithmetic: for the same budget we get not one card but three. And three cards mean three times as much concurrent compute for our workload.
What the RTX PRO 6000 lacks is HBM — High Bandwidth Memory. In this memory technology the memory chips are stacked and sit directly on the GPU package, connected through an extremely wide interface. That delivers many times the bandwidth, but it is far more expensive to manufacture and one of the main reasons data-center cards cost what they do: an H100 reaches roughly 3.4 TB/s this way, our card with GDDR7 about 1.6 TB/s.
That bandwidth determines how fast weights and intermediate results travel from memory to the compute units. It matters most where each computation step reads a lot of memory and computes comparatively little — the classic case being inference on very large language models, where every single token has to pull all the weights through memory. Our time-series models are the opposite: a lot of computation per byte loaded. They are compute-bound, not bandwidth-bound — HBM would be a surcharge with no return for us.
Why we do not need NVLink
The most conspicuous omission on this card: NVIDIA dropped NVLink across the entire RTX PRO 6000 Blackwell line. Multiple GPUs communicate over PCIe Gen 5 only. Plenty of comparison articles treat this as the knockout criterion for training. For us it is not — and the reason matters more than the card does.
NVLink solves one specific problem: it speeds up data exchange between GPUs when a model does not fit on a single card. Then you have to split it — distributing individual tensors or whole layers across several GPUs (tensor or pipeline parallelism). With that approach, intermediate results travel between cards at every single computation step. Without a fast direct link, the interconnect becomes the bottleneck and the GPUs spend more time waiting than computing.
We do not have to split our models. Our time-series transformers fit entirely into the 96 GB of a single card, optimizer states and activations included. That removes exactly the communication NVLink was built for. Worth noting: because this card carries more memory than the more expensive alternative, our “fits on one GPU” headroom is actually larger than it would be there.
Our parallelism sits one level up — not in the tensor, but in the training run:
- Hyperparameter search. We search systematically over architecture and training parameters. That is hundreds of independent runs, many of them computing at the same time — one per card.
- Models per customer and per assortment. Different datasets, different models, no shared state.
- Ablations and comparisons. The same question with five variants, all in parallel.
These jobs are distributed by a Slurm cluster that queues them and assigns them to free cards. One card is permanently reserved for locally hosted open-source language models — so those run in-house instead of through a third-party API.
In all of these cases, communication between GPUs is effectively zero. Each card computes its own run, writes its result into our experiment tracking, and picks up the next job. An interconnect pushing 900 GB/s between two cards would simply have nothing to do here.
Put differently: buying NVLink would have meant paying for bandwidth our jobs never request — and getting a third as many cards in exchange.
What we accept in return
This calculation holds as long as its assumption holds. It has clear limits, and we would rather name them ourselves.
A model above 96 GB gets awkward. Without NVLink there is no shared memory pool across cards. A model that does not fit on one GPU would have to be split over PCIe — technically possible, but noticeably slower. Should our models ever grow into that range, that is a hardware decision, not an architecture decision.
Classic data-parallel training across many cards is not free. When we train one model across several GPUs, gradients have to be reduced after every step. Over PCIe that costs real time. At our model sizes the effect is small — but it is not zero, and it grows with parameter count.
Conversely, there is headroom we are not yet using. The Server Edition cards support MIG: one physical GPU can be partitioned into up to four logical instances of 24 GB each. For many small, parallel inference jobs that would be the next lever — we simply do not need it right now.
What this means for our customers
GPUs are not an end in themselves. Three things get concretely better because of them.
First, the data we work with stays in our hands — processed on our own cards, in Germany, with no detour through someone else’s cloud. Second, we can train models per customer instead of applying one uniform model to everyone, because additional training runs cost us nothing but waiting time. And third, we try more things. That is the difference hardest to quantify and the one that matters most: a team that simply starts an experiment finds better models than one that first calculates whether it is worth it.