Hard disagree on the reason to attend a school like CMU or MIT (disclaimer, I’m a CMU MatSci PhD)
Networking is a big value proposition, but the biggest benefit IMO is learning what questions are important and learning how to think independently about them. LLMs are great for learning but I don’t think they fully replace the skills you need to ask important questions. (This perspective is probably slanted towards research degrees but I think it’s at least somewhat transferable to engineering)
If search is that tightly integrated with the LLM capabilities isn’t that an architecture failure? LLM distends rely heavily on information retrieval for grounding their responses. Why would it be so difficult to let humans continue to directly use the information retrieval systems (classic Google) and keep the LLM systems optional and loosely coupled?
Language processing is literally what search does. They're probably using LLMs for it and interpreting "don't use LLMs" as "either make a new search or disable it" when the intention is clearly "don't display the huge trash pile"
Don’t forget that technology development is a spectrum. A lot of the basic research that laid the groundwork was funded through public investment. Then the VCs and BigCos come in when the technology is mature enough to be worth the risk of investing in tech transfer to take it all the way to a real product
I don’t understand the negative reactions to your post, it’s a good insight.
Basically the weight edit approach is find a semantic direction for the undesirable behavior and steer away from it. With access to the weights I see no reason a model (even one as large as Astra) couldn’t apply that principle to itself. This probably includes at runtime if you wanted to give the model write access to itself
Agreed. You could look at a previous session which got the right answer eventually and make edits until the path to the right answer was more direct.
It's not clear if this would generalize to other related uses, or how disastrous it would be to unrelated uses, but it's worth studying. Feels like a sort of agentic Lamarckism.
Expectations and goals can be technically nonsensical. Agents have gotten really good, but they can’t magically solve intractable problems. If you’re out of your technical depth it can be easy to set unrealistic expectations without realizing it
Then you're still middle-roading it from my perspective. Treating it as a black box means you're only specifying what comes out, essentially.
But to preempt a "I can't do that in my brownfield distributed monolith I'm maintaining at $dayjob": yes, that's why I said it may not be possible without refactoring.
I think we are talking past each other a bit? I’m curious about GP’s perspective here
My point is that you can ask for unrealistic outputs even if you’re treating the system as a black box. This is the classic case of higher level managers asking for infeasible results from their engineering team because they haven’t considered they’re trying to solve the wrong problem. Depends what you’re doing but I don’t think agents tend to push back on badly specified goals unless you’re proactive about it
Yeah I think this is the main issue at play. Bad engineers are now 10x bad engineers with the ability to hide behind the mask cause they can output something that looks right, but they don't understand it. You can "make" an agent do ANYTHING. There is no rigor, there is no taste, there is no risk. Anecdotally I have seen this play out IRL. I worked with a 3-month boot-camper that somehow stumbled into a software developer role at a small company. The amount of slop that he was creating was unprecedented. No formal training, no idea of abstraction, letting/making the agent reinvent the wheel for everything. The LLM was doing the whole "you're absolutely right" so he was also under the impression that he was a real engineer doing the "right thing". Concretely he convinced our manager that we need to re-write our web-app in Go because it was so much faster(0 users btw) and safer than python. He had an agent build out this test suite and there was a metric like Go vs Python DB operations, Go being 107x better. Seemed sus, looked into it and he(his agent) removed connection pooling for the python side but for the Go side, 100 concurrent connections, 10 idle and a connection lifespan of 1 hour. His reason for that was that the python code kept crashing every time his agent adding pooling so he had the agent removed it. This sheet got printed as is and distributed to our C-suite to much praise. I know this is a very specific example but I fear this is happening at some level everywhere. Where in the past a bad engineer to this degree would stick out like a sore thumb if they were surrounded by competent people that have rigorous standards. An LLM/AI doesn't have to worry about what happens if things don't work and when it's confirming that YOU are a genius every step of the way. It's going to be a disaster when the reckoning comes. I am not a AI luddite but software engineering is just as much as a culture as it is art and science. Without that backstop I just think that we will see a huge degradation in quality because there is no incentive for LLMs/AI to write "good software".
You don't give it tasks. You give it what you want. It decides the tasks, and then executes. Then you give it feedback. "black-box approach" here is just a fancy word for vibe coding.
I want to be rich, healthy, happy, young, smart, beautiful. Same for all my friends and family please. How many tokens?
PS: make no mistakes and tell me in advance if I forgot anything that I could regret later.
Or on a more technical side: I would like all problems that any app/company/product could solve to be solved in the best possible way, so that nobody has to even think about what kind of problems even could be solved. Go iterate on that but do not make any mistake, do not ask any questions, be done by tomorrow, do not interpret what I said in any bad way.
Interesting analogy choices. Aren’t straight razors still the highest quality shave? We go with electric and cartridge because it’s faster and cheaper and takes less skill to get a decent result
Digital cameras are another interesting one. Since capturing images is so cheap we just snap a bunch of frames without a lot of thought towards composing them carefully, with a side effect that we get an overwhelming amount of mediocre frames and a few good ones, but we have to sift through.
I’m still not convinced I want to make this tradeoff in programming. But I work in scientific computing where I think it’s (still) really important to understand what modeling assumptions you are making through code
Yup - I think there will remain a space for artisanal code for a decent while yet, particularly in things like scientific computing, safety critical systems, and similar - but for most purposes, the cartridge shave will be good enough, more convenient, and much, much cheaper.
The open version model card says it had extra safety post-training because meta can’t prefilter prompts with a classifier model if you’re running it on your own hardware. They don’t want to get sued I guess if it comes out that someone used their model irresponsibly or something
For me I think the distinction is more about structure and rigidity than keyboard vs pencil. Sketching free-form notes and especially little diagrams is a really powerful thinking tool. I also like to sketch my thoughts in org mode, which has some advantages over paper (structural editing, ability to zoom in and out by folding subtrees) and some disadvantages (tree structure instead of arbitrary spatial structure, more friction for diagrams).
I have a really hard time using WYSIWYG editors for thinking-by-writing. I think it’s because it’s hard to see partially-formed structure so it feels less malleable.
Yes, me too! I like using sketch notes to think through things nonlinearly, then I make an outline in Org or use an AI to get the text from a sketch, and then maybe I'll refine the sketch so it can be an illustration that
I embed back into Org. Structural editing is fantastic for reordering thoughts and converting back and forth between lists and paragraphs/sentences.
but then, from personal experience, once you master, and practice photoshop/illustrator-like software, it becomes not so hard to actually produce quite complex visual communication in little to no effort. it is questionable whether I'd ever be able to do so with pen and paper, though, of course, the feeling... the touch of it is second to none. Talking of practicality though, it makes me wonder. I think rather an important question is whether we also talk about writing/note taking as part of larger personal information management system. It is very likely having one (such own KB) matters even more than the medium itself.
You can get the best of both worlds by picking up a Wacom tablet or similar. It isn't exactly the same, but it does then mean you can doodle but with the power of the capable software.
definitely thinking about it every day, as the pain in my wrist, one resulting from touch and mouse use, is increasing with no end in sight. some pencil and paper based loop through the LLMs seems even more appealing.
So did we get any new mathematics out of this, or just solutions to problems without new techniques or insights? I haven’t been able to keep up with this or read the writeups you mention, genuinely wondering since this seems to be a central concern of the math community
It's only been a month and people aren't that fast, but I think yes. Analytic number theory and complexity theory were stuck on the lack of progress on certain questions, that is now unstuck.
Edit: upon reading (it’s short) I find this statement to be not that informative and kind of self-congratulatory
reply