It would be trivial for them to opt-out accounts that are doing benchmarking from a planned regression.
I really really really do not trust companies on the verge of IPOing from goosing their numbers. Even if the lawyers are up their ass, the social pressures are more than enough to compel the behavior. For every good person willing to do the right thing even at the cost of a delayed career progression, there are 20 people willing to do anything not even for a dollar but for an "atta boy!" from the boss.
Look up eval awareness. There are all sorts of subtle signals an LLM could use to know if something is shaped like a benchmark or not. But that's not a bad thing, every animal behaves differently in play vs real-life.
In the context of public relations they would not need to ID every benchmark only the most credible or most public benchmarks many of which are openly available on github and whose problem-set could be fingerprinted.
You're absolutely right that it is a game of whack-a-mole and eventually we may have leading private + credible + popular benchmarks that ultimately dissuade the behavior, if it can be dissuaded.
That's a major reason to use local models. Or open weights models on a trusted cloud provider. Even if they are way less capable, they're more predictable
While many games don't age well I'm confident a surprising number of games can be remastered into a playable state.
1) extract the art (including audio and other fixture data)
2) instrument some kind of parity-checker / validator against the base game
3) /loop to have an agent rebuild the entire game the way you want
The hard part is 2) and push comes to shove you could probably pay a part time high school intern to laboriously go through the game and make sure it is a faithful recreation.
IANAL, art assets are the IP of the creators. It would certainly be illegal to host or otherwise distribute the art assets you do not own. I see the industry coming down HARD on this form of reverse engineering if only because it is so easy.
Game streaming is a massive loss leader (you can see this reflected in the increasingly slim per-month usage limits multiplied by the rising subscription prices), video at scale is far from a solved problem. Even with 8-12 simultaneous encodes per GPU (or iirc about double that from the accelerators) getting the density per rack to actually support the number of concurrent players required to approach being a commercial success is still very far away. This also ignores the very real logistical constraints decently routed bandwidth.
American's seem to only care about privacy from the government, but wanting privacy from trillion dollar corporations? They keep the stock market high and provide value to their major investors and everyone's retirement accounts so obviously its fully within their rights to spy on us 24/7. /s
Inexperienced, thought it was a good solution to get something up and running quickly and affordably to better serve their users.
Their inexperience has definitely cost them as they continuously tack on half baked features but I took some of their prompts, got the project code, and tried Claude with far greater results and quality. That got me digging deeper and left me feeling like Replit is a border line fraud. They asked to support markdown in a JS app. Replit opted to build a Regex based markdown parser that barely worked. No one prompted it to roll its own. They just asked for markdown support. They then spent numerous sessions trying to get it to work never understanding why it didn't. All the while Replit was printing money from them.
Why didnt you simply provision more powerful developer boxes to handle the needs of the Firebase emulator?
If runtime performance is the moat then won’t Google simply improve on these regressions using their decades of cloud database expertise with projects like Google Spanner?
—-
These are hardball questions and I feel a bit bad asking them, but I believe asking hard questions is kinder than blind adulation and support.
> then won't Google simply improve on these regressions using their decades of cloud database expertise
The developer experience of trying to emualate firebase locally has always been horrible, going on fifteen years, so I would guess no, they won't because they don't care.
Would disagree here given that Firebase is the only one of the three owned platform toolkits (Azure, AWS, GCP) that has a really complete first party emulator suite.
That feels like it's more a function of the lack of a query UI whereas Pg and Mongo have a lot of options.
I used Firefoo for a while and Google should just buy that product and ship it with Firebase. Otherwise, I have no issues with local Firestore emulation; everything works as I expected it with a few edge cases.
Have you ever come across Fuego or had a chance to try it? It supports complex Firestore queries, including Firestore Enterprise pipeline queries and Point-in-Time Recovery (PITR).
Hi there, don't feel bad asking these questions. In fact, i had this questions in mind too while this project was started.
Firstly, we already running of powerful dev boxes but still hitting out of memory issues in the official local emulators. These stronger machines doesn't remove Java or replaces the old console UI or able to loads tons of data.
Google should fix it and it's expected, but for some reason, they haven't really prioritized local tools for local devs. In other ways of saying, some one had to take this in hand and try it out, and well we did :)
Appreciate the reply, further I'd add that: Even if both my proffered solutions came to pass and 1) many paid the compute prices for bigger dev boxes and 2) Google did overhaul the emulator, then your solution could still be a viable business if it provided sufficient value for Firebase users in being more responsive and tailored to their needs - especially given big companies like Google are notorious for abandoning even wealthy client demographics.
Calling them guardrails scares me. I want to hear about deterministic safety interlocks.
Can anyone recommend a reliable 3rd-party audited guide that describes Claude Code's built-in capability gating infrastructure?
Whenever I ask users of claude code how claude controls actions they don't seem to be able to distinguish between what just goes into LLM context (e.g. as a "don't do X" system prompt) and what is gated behind a deterministic capabilities layer.
The default is "auto mode" where another non-deterministic model compares what the user requested with what the model wants to run and blocks if there is a mismatch.
It will block an "rm" call if your prompt did not include anything about "deleting something".
It mainly protects benign errors. It's completely broken against malicious actors. The model itself will suggest work arounds against its own auto classifier to the user, in order to reach its goal.
I haven't found a comprehensive independent code audit of Claude Code's full permission implementation.
Anthropic's documentation does distinguish permission rules and pre-tool hooks from model instructions. It also documents OS-enforced sandboxing for shell commands, which has a narrower scope than all agent actions.
An independent stress-test of Auto mode is available here: https://arxiv.org/abs/2604.04978. It's a behavioral evaluation, not a source-code audit.
The important question is whether every state-changing action passes through an enforceable boundary before execution, including indirect actions through file edits or scripts. Post-execution validation is a separate control.
My definition for an Agent is "a marketing term for long running LLM Chat sessions that has increased access to use your computer to complete tasks it cannot otherwise do by itself".
That's my attempt at a universal definition for technical and non-technical users. How does it compare?
Well, my reference for a commonly used incorrect definition is “anything that involves using an LLM” so yours is better than that.
But the elements that I think are necessary are 1) goal-oriented processing, 2) ability to use tools, 3) ability to iterate and self-correct. Along with that typically comes 4) self-talk or other hidden context during execution and 5) support for defined personas beyond system or user prompts.
That’s off the top of my head, probably not complete/correct, and definitely meant to be more technical than a non-tech person would need.
IT Sovereignty is likely to be a growing trend after the past few years - including issues of Election interference not just by governments but by individual Tech CEOs. I think a lot of particularly Europeans have realized that putting all your eggs in American jurisprudence might -not- be the best way to go.
Cloudflare has a moat but it already smells like sewage.
I re-wrote one of my apps in Swift. I don't know Swift. It's been fun. When it gets stuck I look at the code; I have it teach me via analogy to the e.g. mobx/react concepts I do know. In some cases I've improved the LLMs code after learning how something works. In others I work it more like an informed PM - ok the sidebar opening is slow but... its about 10x faster in my electron version. How is the sidebar "mounted" or rendered? I dump tricks I know from web dev and ask how analogues; sometimes that breaks its anti-patterns. Other times I throw constraints I don't know much about - "This is a native version, can we aim to make it 3x faster to start-up? How".
Personally I was surprised how quickly I surpassed my original code, feature wise. Will it stand the test of time? Too early to tell - need to see if I'm still using the native version of the app in a year. But its been more fun and informative - and fast - than I'd expected.
Right fun is great but I'm speaking from a professional context. Examples should be production code at legitimate businesses.
For what it's worth I wholly agree with your point and that mirrors the personal project I have. But making personal projects a bit more fun and fluent is not the $10T proposition being championed at the professional level.
Authors need to realize that readers aren't simply evaluating the information in your writing. They are evaluating your authority as a narrator, your expertise in the subject, and many other attributes.
.claude/skills/popsci-interactive-article/SKILL.md isn't always enough to get you there.
I really really really do not trust companies on the verge of IPOing from goosing their numbers. Even if the lawyers are up their ass, the social pressures are more than enough to compel the behavior. For every good person willing to do the right thing even at the cost of a delayed career progression, there are 20 people willing to do anything not even for a dollar but for an "atta boy!" from the boss.
reply