The LiteLLM breach started with a key nobody revoked: 7 lessons for anyone running LLMs in production

If you run AI in production, there's a good chance LiteLLM is somewhere in your stack — it's the open-source gateway a lot of teams put in front of OpenAI, Anthropic and every other model provider. On 21 October, a poisoned LiteLLM 1.107.0 was published to PyPI and Docker Hub. It was live for about two hours and carried malware that steals credentials: cloud keys, and the API keys for the model providers it sits in front of.

LiteLLM has now published its root-cause analysis, and the first domino is depressingly ordinary: a key that was committed to a public repository in May, and never revoked.

I run systems that hold keys for half a dozen AI providers at once — speech recognition, language models, voice, telephony. Here is what happened, what to do if you were affected, and the lessons I'm taking from it.

What happened

From LiteLLM's own incident page and its canonical attack-chain write-up:

• 27 May 2026. A configuration file containing the master key for LiteLLM's internal Agent Platform (an alpha service) was committed to a public repository. The commit was later force-pushed over — but, in LiteLLM's words, "key revocation did not occur for the master key."
• 10 October — 136 days later. The attacker used that key against the Agent Platform's API. One endpoint returned the GitHub CLI token of one of the organisation's two GitHub owners. It was "not associated with a service account and was not scoped down": owner-level access to every repository.
• 10–12 October. Using that access, the attacker pushed branches whose CI workflows posted repository secrets to an external endpoint, collecting more maintainer tokens.
• 21 October. The attacker changed the private release workflow to build from a lookalike repository, used the owner token to disable branch protection, and used two maintainer tokens to trigger and approve the release — approval had been relaxed earlier that week so "any two engineers" could ship a backport. Seven malicious wheels reached PyPI at 11:39 UTC; Docker images followed at 11:46. The attacker then deleted the workflow run records.
• Within about 20 minutes an external researcher caught the release. PyPI removed it by 13:23 UTC; all Docker Hub tags were cleaned up by 14:20 UTC.

The malicious package installed files that re-run on every Python start (cook.pth, cook.py, sitecustomize.py), started a background loader for remote code, and exfiltrated credentials. LiteLLM's enterprise build, from its private registry, was not affected.

If you might have installed it

You were exposed if you ran an unpinned pip install litellm between 11:39 and 13:23 UTC on 21 October, or pulled the v1.107.0, latest, main-latest or main-stable Docker tags between 11:46 and 14:20 UTC. LiteLLM's guidance, condensed:

1. Pin litellm==1.106.2 (or another explicit earlier version). Don't run pip install inside an affected environment first.
2. Remove cook.pth, cook.py and sitecustomize.py (and their __pycache__ files) from site-packages, and kill the background Python process holding /tmp/remote_code_loader.lock — or better, rebuild the image or virtualenv from scratch.
3. If the host had outbound internet, assume the credentials were stolen and rotate everything it could reach: the cloud role, every provider key stored in the proxy database, LiteLLM master and virtual keys, the database password, and any secret in the process environment.
4. Check your cloud audit logs for those credentials being used from outside your infrastructure.

Seven lessons for teams running LLMs

1. Deleting a secret from git is not revoking it. The key was spotted and the commit force-pushed away. That removes it from the branch, not from the world: public commits can be scraped within minutes, and forks, caches and clones keep them. The only fix for a leaked secret is to rotate it, immediately, then clean the history — in that order. Make "rotate first" a reflex, and put secret scanning on every repository, including the experimental ones nobody thinks about.

2. Your AI gateway is a treasure chest. Size its blast radius. A gateway like LiteLLM exists to hold everyone else's keys: OpenAI, Anthropic, Google, your vector database, sometimes your cloud. That's exactly what this malware went for. Give the gateway its own scoped provider keys with spend limits, keep them out of developer machines, and make sure one leaked key costs you a budget cap, not your whole account.

3. Agents and agent platforms must never hold a human owner's token. The pivot from "internal alpha service" to "owner of every repository" was a GitHub token belonging to an organisation owner, stored where an API could hand it back. As we wire agents into GitHub, Slack, CRMs and cloud consoles, this is the pattern to kill: agents get their own identity, the narrowest scopes, and short-lived credentials — never a person's all-powerful token, and never an admin's.

4. Pin versions — and pin images by digest, not tag. Everyone who installed a pinned version was fine. Everyone tracking latest was not. In Python, pin exact versions with hashes (pip install --require-hashes with a lockfile); in Docker, deploy by @sha256: digest. Upgrades should be a deliberate pull request you can review, not something that happens because a container restarted.

5. CI secrets are only as safe as whoever can edit a workflow. The attacker didn't need production access to steal secrets: pushing a branch that ran a modified workflow was enough. Restrict which workflows can see which secrets, require review for workflow changes, and prefer short-lived OIDC credentials over long-lived tokens stored in CI.

6. "Temporary" relaxations are attack windows. Release approval was loosened for a backport the same week the attacker struck. Every exception you grant — a wider permission, a skipped review, a disabled check — should come with an expiry and an alert, because attackers holding stolen credentials notice when the gate is open.

7. Detection came from outside. LiteLLM learned about its own poisoned release from an external researcher. If you publish anything — a package, a container, a mobile app — monitor what actually got published against what you built, and alert on release events you didn't expect.

The uncomfortable part

None of this is exotic. No zero-day, no clever cryptography: an unrevoked key, an over-powered token in the wrong place, a relaxed approval, and an unpinned install. That's what makes it worth reading for anyone shipping AI. Our stacks have quietly become concentrations of high-value credentials — model keys with real spending power, agent tokens with write access to code and customer data — and most of us are still treating them like configuration.

The fix is mostly hygiene, and hygiene is cheap compared with rotating every secret in your company on a Tuesday afternoon.

References: LiteLLM: v1.107.0 compromised — incident page (timeline, affected artifacts, remediation) · LiteLLM: attack chain — canonical root-cause analysis (the May 27 key, the 136 days, the owner token) · Wiz analysis · StepSecurity analysis · BankInfoSecurity coverage. LiteLLM says its investigation is ongoing and further changes will be published; details may be updated.