On-premise LLM server
Last updated
An on-premise LLM server is a machine in your building that runs language models for staff. Size it by the memory the model can use. Olares One (24 GB VRAM, 4,999 USD list) fits models up to about 32B at 4-bit.
Olares One → Private AI server → Local models · accounts · backups → Company data stays in the company
How much memory does the model need?
Memory decides which models a server can run. A GPU has its own fast memory, called VRAM. Unified memory, used by Apple silicon and NVIDIA GB10 machines, is shared by the CPU and GPU and can hold larger models. By an approximate rule, a 4-bit model needs about 0.5-0.6 GB per billion parameters.
Add a few GB for context. The table applies the same approximate rule.
| Model size (4-bit) | Approximate memory |
|---|---|
| 8B | ~5 GB |
| 14B | ~9 GB |
| 32B | ~19-20 GB |
| 70B | ~40 GB |
| 120B | ~65 GB |
Which on-premise LLM server options are there?
Machines that serve language models on site fall into three groups. One is a GPU box with fast VRAM, such as Olares One. Another is a machine with a large unified or shared memory pool, such as NVIDIA DGX Spark, a Mac Studio or a Ryzen AI Max+ 395 mini PC. The third is a tower you build.
| Machine | Memory for models | Ships with | Price (USD) |
|---|---|---|---|
| Olares One | 24 GB VRAM (RTX 5090 Mobile) | Olares OS | 4,999 list |
| NVIDIA DGX Spark | 128 GB unified | DGX OS (Ubuntu-based Linux) | 6,950 list |
| Apple Mac Studio (M5 Ultra) | 96 GB unified (base) | macOS | 5,499 base |
| Beelink GTR9 Pro (128 GB) | 128 GB shared | Windows 11 Pro | 4,199 sale |
| Used RTX 3090 tower | 24 GB VRAM | None | 3,800 (SpecPicks guide estimate) |
Prices were read from each seller on 12 Nov 2026. On Ryzen AI Max+ 395 machines, the share of the 128 GB the GPU can use is lower than the total, according to GMKtec's page for the same chip.
Why is Olares One our pick for a small business?
We pick Olares One because it ships with the server parts already in place. Ollama and Open WebUI install from Olares Market in one click, each person gets an account, backups are built in and encrypted, and after a one-time activation it can work entirely offline, according to Olares' FAQ.
The trade-off is memory. By the approximate rule, 24 GB of VRAM fits models up to about 32B at 4-bit. Larger models spill into the 96 GB of system memory and slow down; in Olares' own test, gpt-oss-120b ran at about 36 tokens/s this way. If your work depends on 70B-class models, we would change the pick and buy memory first.
What should you check before you buy?
Before buying, list the models you need and their size at 4-bit, count how many people will query at once, and decide whether the server must run offline. Then plan accounts and backups. Those answers narrow the choice faster than any spec sheet does.
The offline guide covers running without internet. User accounts and backups cover the rest of go-live.
Questions & answers
How much GPU memory does an on-premise LLM server need?
Enough for your largest model plus context. By an approximate rule, a 32B model at 4-bit needs about 19-20 GB and a 70B model about 40 GB.
Is a Mac Studio a good on-premise LLM server?
It can hold large models in unified memory. The M5 Ultra model started at 5,499 USD with 96 GB on Apple's store, read on 12 Nov 2026. It ships with macOS, so the model server, accounts and backups are yours to set up.
Sources for this entry
- Olares One: maker's page (Maker-published)
- NVIDIA DGX Spark: maker's page (Maker-published)
- Apple Mac Studio (M5 Ultra): maker's page (Maker-published)
- Beelink GTR9 Pro (Ryzen AI Max+ 395, 128GB): maker's page (Maker-published)
- GMKtec EVO-X2 (Ryzen AI Max+ 395, 128GB): maker's page (Maker-published)
- Used RTX 3090 tower build: maker's page (Maker-published)
- Olares One FAQ (Olares docs) (Company-published)
- Olares blog: local AI hardware benchmarking (Oct 2025) (Company-published)
Continue this reading path
How the related websites connect
Cite this entry
Best Private AI Server. “On-premise LLM server.” Content reviewed 13 November 2026.
https://bestprivateaiserver.com/on-premise-llm-server/