MLX optimization on Apple Silicon
Which model do you kill to load the next one?
The real problem
MLX makes local inference fast on Apple Silicon. It does not solve orchestration: several applications want several models, and the machine only holds two or three.
With no control plane, every application loads its own weights. Nine applications, nine copies: roughly 208 GB of redundant weights on a 128 GB machine.
How I solve it
An OpenAI-compatible gateway that owns the models, with:
- LRU eviction under budget, one pinnable model, honest accounting of in-flight memory
- Hexagonal scheduler: decision logic is pure and testable, side effects (processes, sockets) live at the boundary
- Guaranteed cleanup of in-flight requests via
BackgroundTask— zero reservation leak when a client disconnects - Structured output through constrained decoding, where the schema allows it
A trap learned in the field
The most useful counter-example I have measured: forcing constrained decoding on lyrics generation broke it.
| Tokens | Duration | Result | |
|---|---|---|---|
| Constrained | 8,192 | 140 s | truncated JSON |
| Unconstrained | 708 | 13.8 s | valid JSON |
Cause: the schema was unbounded everywhere — open objects and lists. Under a hard constraint the model has no reason to stop and loops to the token ceiling.
Rule kept and locked by a test: constrained decoding does not compensate for an unbounded schema. It amplifies it.
Neighbouring trap, on another project: xgrammar enforces alphabetical key order. A prompt written in logical order silently yields a result misaligned with the schema.
What it delivers, measured
- ~208 GB of redundant weights removed by pooling
- 37 GB returned by decommissioning a resident model nothing consumed
- Latencies quoted cold as well as warm: 5–7 s warm, ~2 min on the first call after eviction — the second number is the one to design for
- 295 tests on the scheduler alone
Where it runs in production
- Klody Core — Control plane
- local-suno — Music generation
- KlodMetrics — Text-to-SQL & semantic layer
Your data cannot leave the building?
That is precisely the problem I solve. A 30-minute call is enough to scope an audit.