An AI system degrades silently.

Why degradation is invisible

A classic service either falls over or holds. An AI system has a third state: working, but worse. Three main causes.

Corpus drift. Indexed documents age, new ones arrive, coverage shifts. The RAG still answers — with increasingly irrelevant sources.

Dependency drift. A library moves up a version, behaviour changes at the margin. Today's CVE is in code you installed six months ago.

Usage drift. Users ask questions nobody anticipated. The refusal rate goes up or down, and without measurement nobody knows which of the two is the problem.

What the retainer includes

1. Nightly eval harness, pinned baselines, explicit drift budget
2. Vulnerability and version watch, exceptions tracked by name
3. Observability: cold and warm latencies, refusal rate, real cost
4. Monthly report readable by a steering committee, not only by an engineer
5. Light on-call on regressions caught by the gates

The setup already runs — on me

I am not selling a theoretical setup. The nightly eval has been running on my ecosystem for months, in ten sections: answer quality, code executed under pytest, supply chain, system posture, journal freshness, and the rest.

A red verdict is not decoration: it has already triggered real fixes. That is the only proof that matters for a monitoring setup — a dashboard nobody looks at and that has never triggered anything is not a setup, it is scenery.

The RAG's arbiter is a regression gate with pinned baselines: coverage, refusal rate, zero hallucination. A score under baseline fails the night. No alert to triage the next morning: a hard failure.

The limits of the setup

• This is not 24/7 on-call. It is automated daily monitoring with business-day responsiveness, not an operations centre.
• A gate does not replace human judgement on quality. It catches regressions, not the problems nobody thought to measure.
• Without an initial baseline, the first month is spent establishing one. You only detect drift relative to something.
• If your architecture changes every two months, baselines never get to earn their keep. The retainer assumes a stabilized system.