At introduction, Terra was a good mid-tier model. Terra [high] was on the pareto frontier of DeepSWE's score over cost, if only ever so slightly.
When I had Sol orchestrate Luna and Terra as implementation agents, Sol was a lot happier with what Terra produced and would find far fewer issues than what was implemented by Luna.
But a few weeks after introduction, OpenAI slashed Luna's cost by 80% and Terra's only by 20%. Only then did it become uneconomical to run Terra and its reason to exist stopped.
There are cars with engines that will bring you to 250 mph without burning out, like the Bugatti Chiron. Unfortunately you will burn through your $40k set of tires after 15 minutes at that speed. But don’t fret, because your fuel tank will be empty after 9 minutes anyhow.
The plot doesn't appear to be in Amdahl territory yet. The single-threaded time in the plot looks to be around 39 seconds. A perfect division into 32 workers without overhead would make it 39 / 32 = 1.22 seconds. With the multi-threaded workload being reported as 1.5 seconds in the text, there's still only .3 seconds of overhead + serial instructions that can't be parallelized.
Every doubling of the number of workers halves the execution time cleanly in the plot, from 40 seconds to 20 seconds to 10 seconds. Eyeballing this for 32 over 16 workers is difficult, but it still seems close to halving the total time once again. So there's not a lot of Amdahl flattening, it's just the plain physics of looking at a inverse-proportional curve.
This is well-written. I could follow along quite nicely, from the setup through the bottlenecks and onto the resolution of the performance bug. Even the PRs are very pleasant to read: the majority of them is just a handful of changed lines with an added tests and a bit of documentation.
I was taken aback for a moment that this work originated from a report on StackOverflow. I had thought SO was effectively dead and abandoned by its community. But maybe I shouldn't project my own experience onto everyone else.
I’m not sure why it took me, a NumPy developer, looking at the benchmark numbers and saying “hmm, this is a bug”. But that is what it took. People are sometimes slow to treat behavior in dependencies like NumPy as bugs.
or maybe SO is back to its community sans clout chasers and tourists? If a vaninishing minority drive a given community in content generation and discourse, then lurkers etc leaving isnt as meaningful. you could lose 99% of users on many platforms without disrupting the core community and often having the added benefit of imoroving SNR.
It's a less marketable product in the modern attention economy, but that isnt the same as a dead community. It would be interesting to see a plot of posts/discussion liveliness per unit time and seeing is quality and depth of both questions and answers has changed and how they have changed.
When I had Sol orchestrate Luna and Terra as implementation agents, Sol was a lot happier with what Terra produced and would find far fewer issues than what was implemented by Luna.
But a few weeks after introduction, OpenAI slashed Luna's cost by 80% and Terra's only by 20%. Only then did it become uneconomical to run Terra and its reason to exist stopped.
reply