122 → 0 CVEs, and the root cause was not in the code.

Context

Four separate virtual environments, one per production service: the inference gateway, the document RAG, the coding agent, the music pipeline. Each with its own dependencies, all installed incrementally over months, none ever audited.

This is the normal state of a system that grew by accretion. Nobody decides one day to accumulate 122 vulnerabilities — they arrive one at a time, each too small to justify stopping.

The constraint

None of these environments could be rebuilt from scratch. The gateway serves resident models whose reload costs roughly two minutes; the RAG indexes 1.7 million chunks. "Start again from a clean requirements.txt" was not an option — it had to be fixed without breaking what was running.

What I looked at first, and why it was the wrong place

The reflex is to sort by severity and start with the criticals. I did. It was useful for reassurance, useless for understanding.

Sorting by severity treats each vulnerability as an independent event. On an assembled system they are not: they arrive in clusters, because one cause produces several. Sorting by severity means fixing symptoms out of order.

The useful sort is by package and by environment: which packages appear in several environments, and why are they pinned to different versions?

The real root cause

That sort surfaced an anomaly: several libraries were held at stale versions in one environment, while the other three had them current.

The cause was not in application code. It was a command-line tool installed inside the service's virtual environment. That tool had no business being there — it was only used for occasional development tasks — but installing it had imposed its own version constraints on libraries the service also used. The resolver had done its job: it found a compatible set of versions. Stale, but compatible.

No amount of reading application code would have found this. The offending file was imported nowhere.

Removing the tool from the service's environment and moving it to a dedicated one unblocked the bulk of the updates at once.

What pushed back

Three CVEs had no upstream fix. The package was either unmaintained, or the fix unpublished.

The temptation is to ignore them: not exploitable in this context, the service is loopback-bound, real risk is low. That is probably true. The problem is not the risk assessment — it is that an unwritten exception becomes, six months later, a forgotten exception, and nobody can tell whether it was assessed or merely endured.

All three are therefore tracked by name, each with: the identifier, the package, the reason for the exception, the date of the next review. They are reviewed monthly. If an upstream fix appears, the exception falls.

The measured result

What prevents recurrence

A one-off cleanup is worth nothing: dependencies move again the following week. Three mechanisms, from lightest to strictest:

1. Pre-commit across five repositories — the audit runs before the commit leaves, not after.
2. An audit section in the nightly eval — it goes red the moment one reappears. A red verdict is not a notification to triage tomorrow: it is a hard failure.
3. The monthly exception review — three lines to re-read, the only manual task left.

Transferable to you?

Yes, and it is exactly the scope of the Audit engagement. What transfers is not the package list — yours will differ — but the method: sort by cause rather than severity, look for what has no business being in the environment, name exceptions rather than tolerate them, and put in a gate that fails red.

The number to remember is not 122. It is zero services interrupted: an audit that breaks production does not get run a second time.