I understand the frustration this is born from, but find it hard to imagine a practical rule set that will actually do more good than harm.
1. To be meaningful at all, the legislation must only allow narrow ranges, e.g. 10%, so that employers can't put 30-90K in the description.
2. In addition, you would have to make the application very specific to a given position. If the employer can say "you don't qualify for L7 that you applied for, buuut... we could transfer your application to the L5 track", then we are back to where we started.
3. On the other hand, any super-rigid rule like this will hurt both employers and employees who are actually trying to fill a real position. It will hurt even more the few remaining companies that are looking for people and not just labeled cogs in the machine.
I want this to work, but it has the smell of compliance theater.
I dunno I see a lot of broad ranges in the US but it's still a lot better than nothing. And they can list multiple levels in the job ad. I can see these being pointless but I can't imagine it being harmful.
>2. In addition, you would have to make the application very specific to a given position. If the employer can say "you don't qualify for L7 that you applied for, buuut... we could transfer your application to the L5 track", then we are back to where we started.
In banking, there's various thresholds for transfers or deposits or withdrawals that require scrutiny or reporting. There's also laws that amount to "Don't fuck with us" around deliberately narrowly avoiding those thresholds.
I think that this law could have a similar "Don't fuck with us" clause, where a company can reasonably adjust what jobs they offer people that apply to their posted jobs occasionally, but if they're clearly deliberately misleading applicants, they get penalized enough to make it cost more than they benefit.
Only if you were planning to scrape by. If you actually know the material (the thing the exam is supposed to determine), you should do fine, and any good or bad luck will tend to average out by the time you graduate. Besides, there really aren't any alternatives left. Completed homework had some weak-ish signal for a while, but now it doesn't.
It's been suggested elsewhere that the AA-Omniscience benchmark [1] is more representative than the rest. It puts K3.1 about 6 months behind the frontier and the new version of DS 4.1-flash about 12 months. That roughly lines up with my gut feeling as well, at least when it comes to the trickier problems. Still rather impressive, but not quite frontier.
Although even this one needs to be taken with a grain of salt. Fable/Opus still act very differently to Astra and Argon imo. But maybe it's just because I am used to these. I wouldn't clearly put them above each other though, although I still prefer anthropics models, but that could be simply out of comfort.
Huh, this is an interesting hypothesis! It would make sense, to a degree. Part of "big model smell" to me is the model being able to pull a huge range of interesting algorithms, architectures, mathematics, and everything else that would come with greater world knowledge and having a much more sort of innate, like, full, well-rounded understanding of it instead of surface-level ICL pattern matching on web results.
The inspiration from the F35 seems to be quite literal. Spent a lot of time and money making vertical takeoff happen, but forgot to give the vehicle any useful capability once it's actually in the air.
This is false. It's being fielded in large quantities by militaries all over the world as a workhouse multirole fighter. Additionally, it's the best STOVL and SEAD aircraft ever produced.
I've had the same experience, and likewise I have no idea what to do.
Or rather, I do have ideas, but they do require substantial restructuring. E.g.:
1) Strict, in-person (no exceptions), electronics-free exams with written and oral portion.
2) Transition to guided independent learning: students are given written materials and recommended exercises to study. They are shown how to use AI assistants productively. Instead of lectures, there are scheduled discussions, where students can ask questions and listen to additional explanations from the lecturer. They can also seek feedback for completed exercises. However, none of this contributes to grades in any way.
With such an incentive realignment, I think it's possible that higher education could be salvaged. But... apart from general institutional inertia, there is also the political angle standing in the way. Everybody needs a degree in the "developed economy", and the above changes are basically the opposite of the nobody left behind policy that enables half the population to attain one...
I have stopped reading Ed Zitron quite a while ago because of his long-winded and polemic style. But counting his failed predictions is maybe not that useful, in the sense that his core thesis is really just the AI Bubble. While it hasn't burst, all of his predictions remain wrong; when (if) it does, he'll have been right. Trying to guess the exact time or failure mode (whether it's open models or corporate sticker shock or a bond crisis) is a fool's errand when there are so many powerful actors all-in on keeping up appearances. Exposing the financial and corporate shenanigans of the AI world is nonetheless useful. I just wish someone would do it in a more level-headed way.
Note to self and others who have read Richard Preston's "Hot Zone": The 1989 Reston (US) incident was NOT caused by a single-base mutation of Ebola, but by a distinct monkey virus. Preston seems to have made that up to dramatize events.
I read that book when I was probably way too young for it, and that incident made me spend a lot of time staring at the ceiling in the middle of the night. I'll have to dig into it more!
Cheating and overfitting, as discussed in the article, are the most obvious problems with benchmarking LLMs. But there is also the aspect that, at least for closed models, the tokens still have to be sent to the provider's servers for inference. This makes the holdout set not as held out as it may appear. OpenAI and Antropic probably don't care about your private set of regex benchmarks, but for the headline "closed" benchmarks, I'd be surprised if they haven't collected a nice representative set of "holdout" problems to be examined at leisure.
1. To be meaningful at all, the legislation must only allow narrow ranges, e.g. 10%, so that employers can't put 30-90K in the description.
2. In addition, you would have to make the application very specific to a given position. If the employer can say "you don't qualify for L7 that you applied for, buuut... we could transfer your application to the L5 track", then we are back to where we started.
3. On the other hand, any super-rigid rule like this will hurt both employers and employees who are actually trying to fill a real position. It will hurt even more the few remaining companies that are looking for people and not just labeled cogs in the machine.
I want this to work, but it has the smell of compliance theater.
reply