This laboratory's architecture, at your place.

The problem it solves

Most organizations that cannot send data to a cloud face a false choice: give up on generative AI, or sign a confidentiality agreement and hope.

The third path exists and has become practical: open-weight models that are good enough, unified memory that is large enough, a runtime that is fast enough. What is missing is not the model — it is the infrastructure around it: what decides which model is loaded, what guarantees an answer is sourced, what stops an agent writing where it should not.

That layer is what I build. It applies to a single machine as well as to a small fleet.

What it includes

1. Resource-aware inference gateway: several models, one RAM budget
2. RAG where every answer is sourced — and which refuses rather than invents
3. Tool-using agents with explicit approval on anything that writes
4. Supervised services, health endpoints, privacy-respecting usage journal
5. Knowledge transfer: your team takes over, documentation included

What it costs in hardware

It is a CTO's first question, and almost nobody answers it publicly. Here is the order of magnitude, measured on my own machine.

On a machine with 128 GB of unified memory, I hold an 80 GB budget for models, the rest going to the system and applications. That keeps a quantized 35-billion-parameter conversational model resident, plus a second specialized model, with LRU eviction between the others.

Below 64 GB you have to choose: a single resident model, and accept the reload cost on every switch. That works for sequential use, not for several concurrent applications.

The figure that actually matters is not peak throughput, it is cold latency. Warm, a response starts in 5 to 7 seconds. On the first call after an eviction, expect roughly two minutes of loading. That second number is the one to design timeouts around and to tell users about — quoting the warm latency is the surest way to disappoint.

Read the full case study →

What I do not do

• No multi-node orchestration on NVIDIA GPU clusters. My specialty is Apple Silicon unified memory and single-machine inference.
• No pre-training or large-scale fine-tuning. I assemble, optimize and harden open-weight models.
• No solution that exposes your data to a third party, even partially. That is the entire point of the engagement.
• No delivery without knowledge transfer. A stack nobody can maintain is a liability, not an asset — transfer is in scope, not an option.