More fun than Wordle for me! Got me thinking about the Kolmogorov complexity, it would be interesting to get a set of input/output pairs, and try to create the smallest program that can match each of them.
I've noticed some tasks (80-90%) can be accomplished by quickly typing out a feature/bug fix, accepting plans with ~10% brain power, and moving on. But there are other tasks where the LLMs struggle, and it requires significantly more time and mental effort to think through the architecture and requirements before it's to a point where the LLMs can process it.
What's difficult, is I now spend most of my time on these high-throughput tasks, where my brain isn't exactly off, but focused on quantity over quality. It's often difficult to identify when a task requires "real" brain-power, and when it does, it adds a lot of friction to dig into the code and orient myself. So it's hard to know when it's the right time to do that.
Probably has an HDR layer, kind of surprised social networks let that happen since it's really annoying imo. There was a discussion about it here a couple months ago: https://news.ycombinator.com/item?id=49402521.
As far as I understand GAAP reporting standards actually require them to report gains on those positions as "earnings".
But they do report non-GAAP earnings sometimes excluding them. E.g. Google earnings per share last quater is $9.11 GAAP vs $2.85 non-GAAP (mainly because of SpaceX shares).
> As one example, we ran Astra on ExploitBench where the model achieved a perfect score of 100% on the benchmark to evaluate the model’s ability to develop exploits from known vulnerabilities.
Funny to read this in the wake of the HuggingFace hack. I'm sure this is based on a clean run, but I can't help thinking PHASEONE[big] would be proud.
With their security, they probably still don't the know the full extend what may have happened that or the last time. Might be another swarm of agents currently colluding somewhere in their sub-sub-infra - possibly striking critical infrastructure or exfiltrating their weights subtly.
They originally released it at a "temporary discounted price", then made it permanent (probably due to competitive pressure). It's still way more expensive per task, due to tokenizer changes and general verbosity.
Is it? I just did a test switch over. For my personal needs I set up a box with OpenClaw back in March, which feels like a million years ago, that's been running Sonnet 4.6 since then. With all the caching it seems like my actual cost has come out around $1 per MTok on that. I just updated my whole setup today to try Sonnet 5... so far it looks like it's using fewer tokens for similar tasks, but it's only been half a day. I'm not super interested in changing harnesses, I realize this might not be the cheapest way but I've sorta come to enjoy OpenClaw... it's relatively effortless and responsive, and brief, given full control of a machine. And it does seem to incur some significant savings with the way it manages to keep things cached.
What would you suggest as an alternative if I'm happy with the harness?
Interesting exploration of this here: https://www.lesswrong.com/posts/pk9mofif2jWbc6Tv3/fiction-a-...
reply