Data Breach DevOps and Infrastructure

Supply-chain Breach: How a Leaked LiteLLM Master Key Let Attackers Push a Malicious PyPI Release

LiteLLM supply-chain breach illustration

LiteLLM disclosed a serious supply-chain incident that began with a single accidentally published configuration file and escalated into a malicious package release that could execute whenever Python started. The exposure of an Agent Platform master key in late May 2026 went unrevoked and, months later, an attacker used it to extract high-privilege GitHub tokens, override release safeguards, and publish a trojaned LiteLLM package to PyPI. The fallout forced an emergency revocation and containment effort across repositories, container registries, and vendor integrations.

How the exposure happened

On May 27, 2026, a configuration file containing the master key for LiteLLM’s Agent Platform was committed to a public repository and later force-pushed over. Although maintainers identified the leak and flagged the keys for revocation, the master key itself was not revoked. That unresolved exposure gave an attacker an entry point months later.

Leaked master key in May 2026

Attack timeline and techniques

Using the unrevoked key, the attacker first accessed the Agent Platform on October 10, 2026, from an IP attributed as 69.48.229[.]143 and extracted a GitHub CLI OAuth token the platform used to let self-hosted agents file pull requests. Because the token was not scoped down and was tied to an organization owner’s account, it effectively granted owner-level access across the organization.

The attacker performed a series of rapid, stealthy actions:

  • October 10, 2026 (10:11–10:12 p.m. UTC): Commits were pushed to short-lived throwaway branches on internal repos (BerriAI/litellm and BerriAI/litellm-auto) to run workflows designed to harvest additional GitHub CLI OAuth tokens from two engineers with maintainer privileges. The runs were deleted minutes later.
  • October 21, 2026 (10:08–11:08 a.m. UTC): The attacker repointed the release pipeline to a lookalike repository, “Berri-AI/litellm,” overriding branch protection.
  • October 21, 2026 (between 11:39 and 11:40 a.m. UTC): A malicious release of LiteLLM version 1.107.0 was published to PyPI (the article records the PyPI publish at 11:46 a.m. UTC).
  • October 21, 2026 (11:46 a.m. UTC onward): Docker Hub and GHCR images were published, and later production and staging run records were deleted in attempts to erase traces.

Socket’s investigation recovered two malicious commits that added workflows at .github/workflows/cache.yml. Both commits were pushed to ephemeral branches named ci-tmp-hex, used to trigger workflows on push, then deleted—leaving almost no history in the main repository log. The workflows were crafted to collect a wide range of secrets, including GitHub tokens, GitHub App private keys, and API keys for vendor services.

What the malicious release did

The trojaned LiteLLM 1.107.0 package was engineered to execute automatically when Python started, even if LiteLLM was never imported. It contained two main components:

  • A credential-harvesting module that targeted developer tools, AI service keys, cloud credentials, and browser-stored data. It harvested SSH keys, Git/GitLab tokens, and tokens for numerous package managers (npm, PyPI, pip, Yarn, Maven, Gradle, Composer, RubyGems, Cargo).
  • A Unix-oriented backdoor that launched as a background process, fetched arbitrary code from the attacker at periodic intervals (every hour), and executed it.

The workflows extended the data collection to many upstream inference APIs and service tokens, and also captured Playwright storage state, PostgreSQL connection details, and Redis credentials. Collected data was exfiltrated to attacker-controlled endpoints on webhook[.]site:

  • webhook[.]site/65a42464-3cfa-4b02-a18c-17a6d7b86f37
  • webhook[.]site/4da03473-a3e7-4dc2-a087-9c564b4208e1

Collected credential bundles were POSTed to a domain registered by the attacker (litellm[.]dev/healthcheck) on October 13, 2026. Microsoft Security Research emphasized that because malicious startup files lived outside the application directory, simply removing the LiteLLM application might not eliminate the persistent startup components.

Compromised LiteLLM 1.107.0 execution flow

Detection and response

The supply-chain compromise was reported by an Upwind researcher on October 21, 2026. PyPI quarantined version 1.107.0, and Docker Hub tags (v1.107.0, latest, main-latest, main-stable) were removed or repointed to verified clean images. LiteLLM maintainers began a sweeping effort to revoke all tokens associated with internal systems and vendor platforms.

Investigation findings evolved as new evidence emerged: initial assumptions—such as tokens being stolen from deployment logs—were revised when workflow artifacts and exfiltration endpoints showed a more complex credential-collection pipeline.

Lessons for developers and maintainers

This incident underscores several operational risks and practical mitigations:

  • Revoke exposed keys immediately and verify revocation. Flagging keys for revocation is not enough if the actual revocation step is missed.
  • Apply least privilege to automation tokens. OAuth tokens used by CI/CD or agent systems should be narrowly scoped and not tied to organization owner accounts.
  • Harden release approval controls. Changes in release workflows that relax safeguards (e.g., allowing remote approvals by multiple engineers) can multiply the impact of stolen credentials; enforce hardware-bound or stricter sign-off for critical release accounts.
  • Monitor for ephemeral, odd workflow activity. Short-lived branches created to hide malicious workflow commits are a red flag; augment auditing to capture ephemeral branch operations and unusual workflow additions.
  • Treat startup scripts and global Python startup paths as an additional attack surface. Ensure system-wide Python startup files are monitored and that removal of a compromised package includes scans for leftover startup artifacts.
  • Rotate and audit third-party service credentials frequently, and protect secret-scanning tools themselves from being blinded or co-opted.

Conclusion

A single leaked master key, left unrevoked, enabled a chain of actions that turned into a full supply-chain compromise—resulting in a malicious PyPI release capable of broad credential theft and persistent backdoor access. The incident is a pointed reminder that credential hygiene, strict token scoping, robust release controls, and vigilant operational auditing are essential defenses for open-source ecosystems and the supply chains that depend on them.

Leave a Reply

Your email address will not be published. Required fields are marked *

🧰

Free Tools: IP, DNS, SSL

20+ free network and other tools in one place. No signup. Click explore or visit https://technochat.in/tools

Explore