Many SaaS vendors forbid benchmarking, I find it crazy that such anti-competitive terms are standard across the industry but they are. Generally the goal of such terms is to "control the narrative" around the product, regardless of the truth of performance being better or worse than competitors.
Interesting; was curious how this didn't fall into trouble with ToS. Apparently the "no benchmarks" clause was intended for "limited preview" audiences and didn't get removed at launch on accident.
Missing the point that they marketing, but so far I saw it perform good at some tasks, and pretty bad on others. I don't think it's all that general purpose they claim it to be.
fwiw, i am with thariq https://x.com/trq212/status/2092302273099796842 in that prompts should be tuned for models and in fact blindly applying agents.md is probably an antipattern unless you want all models to basically converge to some common ill defined of instruction following - good local minima, bad global minima for model diversity and exploration of intelligence.
aka, sometimes it really is too early to force a standard
If you want this it's trivial to add an AGENTS.md that simply says "if you're Claude read CLAUDE.md, if you're Astra read ASTRA.md". A common entry point is good regardless.
Wasting turns? That is silly, use a better harness. Also token usage can mitigated by incremental discovery instead of stuck 5k+ worth of tokens in the AGENT/Claude md file.
In a "one LLM only" environment, your instructions are by default tuned for said LLM.
In a multi-LLM environment, roughly nobody will keep separate sets of instructions for each. It's not a realistic take.
On top of that: If your LLM is so bad at reading that it can't follow a set of instructions that wasn't specifically written just for that one single precious LLM, I sure wonder what that says about your employers repeated statements that ASI is definitely right around the corner.
depends what you're doing. if you've got a specialized agent deployed in prod, of course your evals and prompts will be targeted towards 1 specific version of a model.
on the other hand if it's just a local coding/"use my computer" agent, i highly doubt the effort in maintaining different prompts is worth any gain in performance
19 out of 20 harnesses supporting the standard isn't "too early".
Tariq is wrong and it's not an antipattern. Reason being that a good AGENTS.md impacts all models in a positive manner. If it affects certain models negatively, it means you're putting the wrong things in it.
No thanks, I'm not tuning a bunch of files just for things to break when I switch models or a new one comes out.
I'll just use my one-size-fits-all AGENTS.md file and tweak it when the one of the clankers screw up. I don't have time for such busywork.
Actually, I will append extra rules to CLAUDE.md (which imports AGENTS.md) since there is a hook there, and Claude has its own foibles. So I'll backpedal a bit there.
> July 13, 11:10 PM: I reported the live basetenbot token, the public Harbor project, and the repository permissions.
> July 14, morning: Baseten made the Harbor project private. I flagged that the token itself still worked.
> July 14, 4:34 PM: Anton from Baseten Security confirmed the issue as critical and said they had made the Harbor project private and rotated the token. He also asked us to securely delete the images we'd pulled.
> July 14, 5:05 PM: We confirmed deletion and sent over two lower-severity findings from the same scan.
> July 17: Baseten closed out the remaining findings.
> September: We let Baseten know we planned to disclose the finding publicly and sent them a draft of this post.
They also sent us some T-shirts and sweatshirts as a thank-you for finding this critical bug.
well done all around. i think my only open question is what default security boundaries should all vibecoded internal agents follow as a learning we can take from this
Good in terms of prompt communication and fix. Absurdly bad in terms of reward.
Earlier in the article, it mentions that Baseten is valued at $13B. They can't dig into their couch cushions to give a few thousand dollars to the researcher privately disclosing a bug that let an attacker escalate to admin in their GitHub org?
This sends the message that honest researchers should not waste their time looking for vulnerabilities in Baseten, but it's a good target for criminals who want to monetize these vulnerabilities.
This is two companies working together. Most of the comments below are assuming this was an independent security researcher doing work on their own time. This was professionals doing work for their companies on both sides.
> This sends the message that honest researchers should not waste their time looking for vulnerabilities in Baseten, but it's a good target for criminals who want to monetize these vulnerabilities.
The reason they were looking for bugs was in the context of a B2B relationship, not as a someone independent on their nights and weekends.
If they give them any additional compensation it would probably be in some amount of free or discounted services, which is what they’d want anyway.
Swag packages like these are a token of appreciation not a reward.
The front page post in HN here is worth far more than few thousand dollars , don’t think either organization is operating under purely financial transactional nature .
Most people who find a dropped wallet will return it without evaluating the market value of your compromised identity or the contents of the wallet .
Grateful owners may buy you a beer that doesn’t make them cheap , not everything is evaluated in purely money terms, and that is a good thing ?
> The front page post in HN here is worth far more than few thousand dollars , don’t think either organization is operating under purely financial transactional nature .
not always, especially if its just someone independent. iirc there was a guy here not too long ago who started dropping Windows 0days because Microsoft couldn't be assed to process his bug reports
That actually supports the point that people aren't acting under purely financial motivations. If the guy was purely following financial motivations, surely he would have chosen to sell the vulnerabilities to the shadier side of things. Instead, he dumped them publicly, burning their value while amplifying the "fuck you" factor to Microsoft.
Ignoring reports, or just fixing the vulnerability without acknowledging the work put in by a researcher, is rude and invites rudeness in return.
Microsoft runs a bug bounty program. NightmareEclipse (that’s the researcher’s handle) allegedly participated and Microsoft did not honor their part of the bug bounty program terms.
This is a completely different situation - a company evaluates the security of a prospective vendor prior to entering a business agreement.
> iirc there was a guy here not too long ago who started dropping Windows 0days because Microsoft couldn't be assed to process his bug reports
Did that ever actually happen? I remember him threatening to start dropping 0days and getting a lot of press coverage for it. When I tried to look it up I didn’t find anything at the time.
“Nightmare Eclipse released these zero-day exploits as part of an ongoing dispute with Microsoft over the company's bug bounty and vulnerability disclosure practices. […] Since April, the anonymous security researcher has disclosed a long list of zero-day flaws, including ShieldBreak, LegacyHive, RoguePlanet, BlueHammer, RedSun, YellowKey, GreenPlasma, MiniPlasma, and UnDefend, targeting Microsoft Defender, BitLocker, and other Windows components.”
Wallets usually belong to real people with lives. We can empathize with them. Companies are not people. And they also don't and can't empathize with you.
Companies are 100% people. The fact that companies, their CEOs, and employees are not treated like people is exactly reason why humanity is in the shitshow show it is right now.
>Wallets usually belong to real people with lives.
So does data ? it belongs to real people.
I would imagine baseten's customers and eventually their end-users[1] were also grateful that their data was not compromised here and the disclosure was responsible.
[1] There is a pretty good chance you and I could be using services who are using baseten
"and can't empathize with you" - I don't really understand why such a perception of companies has been regurgitated and reinforced so much in US public, to the point where it's a blank excuse from ever expecting such a thing from a company. It's not true that it can't. The only reason to keep repeating that kind of worldview is to absolve companies behaving in shitty, toxic or downright evil ways.
The law doesn't say companies MUST choose the most profitable choice at every turn, and even explicitly allows for good treatment of customers, community, employees etc as a viable business strategy (even if it's sad that it must be justified in that way).
I agree that this notion that companies make these decisions is bad, bad on the grounds that it's people working for those companies that make the decisions, they hide themselves away, but the company itself isn't doing anything - is always a human making the decision
I think the point is that companies are purely legal entities, and as such, cannot feel, much less empathize, simply by virtue of them not being living things
That's nonsense. "Companies" are not something non-human, they are run by humans, who do feel, empathize, and are living beings. Without these living-being humans, there would simply be no "company".
Now how those humans that run the company behave is another thing - they are free to be greedy assholes, and a lot of them are, and some of them aren't - but that's still a human thing.
Yes, they are not humans. They are not even living creatures. They are mostly-legal entities mostly for the purpose of contracting with humans or other legal entities.
> they are run by humans, who do feel, empathize, and are living beings
This is usually true, but it is orthogonal to whether the company (a legal entity) itself is a biologically living creature, which is the only thing capable of feeling*. To put a point on it: my lawnmower is also run by humans, but it does not have empathy for any grass or people that gets in its way. It mostly just goes where it is steered. A company is like that, except with less touching grass.
* — unless you want to argue semantics about what "feeling" means, even though the discussion is about "feeling" and "empathy" in the way humans experience it, and how that form of "empathy" does not exist for a nonliving legal entity which may or may not employ any actual humans
I think the only misleading part of this situation is your naive and self-centered definition of "trust", and the assumption that so many others think similarly enough that they need to be warned.
I trust a business to fulfill their obligations as stated in writing for the money paid. I do not trust them in any other way. Nobody should "trust" or depend on undefined behavior. Common sense can only ever be as common as you expect.
> expecting compassionate or empathetic behaviour from companies, and [another thing, which is not the same thing as the first thing but merely similar or related]
First, you're being petty and just fighting fire with fire. Second, most of this research is fairly trivial.
What you're instead encouraging is a race to the bottom. You're not going to kill off the companies you hate by withholding information. You don't even have that power anyway because by its very nature, security research is not secret. You're really just encouraging pessimistic groupthink and bad faith. This is why businesses can't be more open about their flaws. It's not that they're stupid and incompetent, but that the pitchforks come out. These are the seeds of dystopia.
They would have eventually figured it out, but as an unfortunate incident with an outsized effect. As much as you wish it to be true, even the worst of these incidents will not kill their business. As much as you hate these businesses, their financial momentum will eventually cause the public to depend on them more. There's more at stake here than anyone's personal gain. It's naive to think otherwise.
You're just manifesting broken windows and ignoring litter thinking you're fighting the man. This is straight up ghetto punk ass behavior wearing a white collar.
> I'm saying serve yourself, not them. if you have say, a 0 day on your hands, do what serves you best. is that "ghetto punk ass behavior"?
Yes.
If you have say, managed to find an overlooked passage into an ostensibly high security building, "doing what serves you best" such as selling the information to some thugs, is in fact that kind of behavior.
for real security bugs, like, you can literally sell them to brokers who sell them to governments. would selling stuff to the CIA be ghetto?
morally, it depends. but after seeing so many posts of e.g. Google cheapskating on bug reports, it really makes no sense to me to participate in such a broken system.
this case however is quite different as it was a B2B encounter and during vendor vetting
like to me it just seems like a fair deal, if Google wants their bugs patched (which they can definitely afford to do) they'd just pay properly for serious bugs and so on, and everybody would be happy. it's not some kind of thing where they can't do anything about.
maybe you can understand the angle I'm coming from?
This was a potential customer reporting a result of an audit of a tool they are evaluating. This is frequent and normal activity in enterprise deals. Most of the time such reports are not critical vulnerabilities it would things like tenant configuration -what business would like versus what CISO will accept or risk acceptance of the product they are buying with monitoring or other prescription on access restrictions or a DPA and so on.
It would be novel business model to spend ton of money in getting a prospect to late-deal stage where they are ready to do a security audio for you just so that part is "free" .
Most companies wouldn't disclose(to the public) even if it was serious , that is not their job, they will report to internal teams and re-review on fix. Strix.ai has a benefit in doing so as they sell a scanning tool for this purpose so we get to hear of this.
Yeah companies need to quickly understand that having good actors try and hack you is a good thing - those hacks get reported and another door gets sealed shut for bad actors.
This is more true today than ever before as the bar for a successful attack has never been lower. We’ll see a resurgence of the script-kiddie, or shall I say, vibe-kiddie :-/
The researcher in this case was doing a security review for their company who was a potential customer. Sending potential customers more than a token amount of cash is usually prohibited by corporate ethics rules for obvious reasons.
That's incorrect. It's not only perfectly acceptable, but absolutely vital, to pay someone for their services (incl a customer) for assisting with an existential threat against the corporation.
Any counsel or HR who would draft a corporate ethics rule that wouldn't allow for a bug bounty to be paid out on a massive vulnerability, merely because the person was "a potential customer", should be immediately replaced.
if it were my company I'd not pay a dime if the researcher was going to make a big public blog post about a security issue in my infrastructure that I promised customers was secure.
I'm sure the cash value of the advertisement here is worth more than a bug bounty would pay.
> This sends the message that honest researchers should not waste their time looking for vulnerabilities in Baseten, but it's a good target for criminals who want to monetize these vulnerabilities.
of course, these companies want you to sell vulns to brokers and other orgs. they don't care about bug reports.
This is probably still considered standard response timeline, not a rapid one.
The time window allowing for CVEs + Vulnerabilities remediation has been collapsing to days and hours perhaps even minutes[1]. Anyone who has an OpenRouter account can start using Strix + GLM 5.3 Flash to do damages at frontier Mytho 5 level cyber capabilities. [2]
This cyber patching race is on, won't stop until all the software created for the past 70 years still in active use needs to be patched up. This is happening at EVERY SINGLE software company.
Meanwhile I have customers running legacy web apps last compiled over five years ago on end-of-life operating systems… and it’s crickets chirping. Dead quiet, not even a hint of an attack, let alone a breach.
I expected them to have been hacked to pieces by now, but even “maximally vulnerable” internet-facing apps seem to be relatively unmolested so far.
Maybe it’s still too expensive to go after “boring” enterprise targets? Maybe the bad actors targeted crypto systems first for the immense payoffs, if successful?
It's a risk-reward ratio, same as anything else, whether legal or illegal.
You wouldn't organise the equivalent of an elaborate bank heist to break into a child's piggy bank, it's just not worth it.
I have heard of a few high profile crypto heists that appear to be AI-assisted, some as far back as the GPT 3.5 era. There was an article I can't find any more about someone accidentally pushing a security fix to a public repo and getting their wallets drained via that specific mechanism within something like an hour.
Malicious actors are watching crypto like a cat in front of a mouse hole, because a "success" can net them the equivalent of hundreds of millions of USD that they can instantly transfer, launder, and spend.
For comparison, what would they achieve by hacking the web site of a local council or public library? Cause some embarrassment? Attempt to crypto-locker them? What are the chances of a payout? Certainly not a 100%, and you're also certain to get the attention of the local equivalent of the FBI or Homeland Security.
I’d treat a vibecoded agent like an untrusted CI job, not like a junior employee: repo-scoped identity, read-only by default, no inherited Actions token or production secrets. Any operation that turns a read into a write should require approval outside the agent’s control and produce an auditable diff. Network egress belongs in the boundary too. Read-only access is not much protection if the agent can send everything it reads to an arbitrary endpoint.
Of course, it's impossible to know for sure what was LLM processed or not, but some of your posts (like this one) have been getting classified that way.
no, it does not, you have to actually RTFA if you are going to try to TLDR a court proceeding. literally 3 paras down:
> the United States Court of Appeals for the Ninth Circuit vacated the preliminary injunction and remanded for further proceedings. The Ninth Circuit held that Amazon was unlikely to succeed on the merits of its claims because Perplexity did not “access” Amazon’s computers within the meaning of the CFAA or CDAFA; instead, the access was performed by the user employing the Assistant as a tool. The court found that the district court erred in its analysis of the equitable factors, which favored Perplexity, and concluded that an injunction was not warranted under these circumstances. The disposition was to vacate the injunction and remand.
perplexity won on appeal. if you stop at first para you are part of the problem
This could get pretty pedantic. They haven’t “won” yet, and the first few paragraphs do accurately describe the problem, but not the whole state of the case. The injunctions and appeals are very important, but they are details of the suit proceedings, not the case itself.
reply