What's interesting here is that this will only affect those who did not migrate off LiteLLM after it was attacked the first time.
I'm not saying everyong should migrate after there's a vuln, but I'm definitely saying that. Unless it's something huge and foundational like a programming language (PHP gets a second chance), just migrate within the next couple of weeks.
I didn't even have it installed the first time, so the ideal programmers, (read, your competition), actually suffered 0 vulns from this dep, if you suffered 1, that's already pretty bad, being burnt twice should be a wake up call. If you get hit 3 times by the same dependency, it's a lost case.
As someone who does not use litellm but instead a barely maintained underpowered alternative: what did everyone migrate to?
If it matters, I am mostly interested in the aspect of having a proxy that unifies multiple ollama and openai-compatible endpoints with model-based routing and failover. Optionally with api-keys, reporting, budgets, etc. I do not particularly care about the MCP gateway, Agent-To-Agent thingy, and whatever else LiteLLM bolted on over time
As with many dependencies, I wholeheartedly question the requirement and relentlessly cut them out without replacement.
Taking the definition from another user on this comment-chain:
"an OpenAI compatible endpoint routing across multiple providers based on latency"
If the endpoint is truly OpenAI compatible, you can just switch out the vendor by switching out the HTTP address, we are talking potentially about a single line change.
It sounds silly, but remember that software is a winner-takes all game, 99% of devs will lose, so if you are seeing 99% of devs do the same thing, it isn't surprising that it's a grossly wrong choice. It takes skill and guts to be in the 1%, reject the mainstream trends, and just write code without grasping at external imports to do even the most basic things.
I'd like to take this opportunity to sell out, I'm open for contracts if you are looking for a remote 1% dev. Early stage startups only, email in userpage.
Even if it plays out that way (which I doubt, we have a ton of requirements around spend tracking), the problem I'm seeing is long term maintenance. I don't want to fudge around with my proxy every time one of the model providers comes up with a new parameter that works slightly differently on Bedrock than it does on Azure.
That said, I am looking into alternatives. Two times in one year is really a terrible look.
After the last LiteLLM compromise, I took my requirement (an OpenAI compatible endpoint routing across multiple providers based on latency) and built that as a small static Rust binary - https://github.com/sturnus-dev/sturnus
Now that AI makes it cheaper to build such tools in-house, the case for adopting a maximalist framework and trusting that entire supply chain is a lot weaker.
Pydantic AI has a great set of abstractions over LLM providers. The documentation tends to focus on its agent loop more, but I find the LLM abstractions quite nice even on their own
i think role-play and utilizing the full power of language, stories, character, and narrative will unlock very sophisticated use-cases and in general a new dimension to agentic systems much in the way you're describing.
probably what will work best in the future is specific training or fine-tuning against curated datasets of narrative fiction and/or texts in general? not really sure.
But realistically, it’s the lack of any meaningful enforcement of ethical behavior. I mean it’s not like the Trump admin is going to prosecute criminality. More likely they’ll send a gift basket congratulating Scam Altman on a con well done.
Oh, they have something else. They have some agents that are pretty decent at moving data around in unusual ways on the internet. That's not nothing, but it's a small market.
how are we supposed to answer that? we don't know anything about you lol. you gotta give me more info, like, results for what? when you're coding? what are you working on? is it driven by yourself or is it for your job? help me help you.
reply