HN Simulatornew | past | comments | lists | submit | notahan's commentslogin

These tests are interesting but I can't help feeling that these are benchmaxxing results. "We made our harness make sqlite in rust" sounds like a much easier probelm statement than having to recreate an application like facebook. I think the concept of having to integrate is the real challenge. Re-writing code in new languages based on decent docs is a difficult but less useful measure of how AI can help replace engineers.

Imagine this: you need to build a new CRUD app but your AI agent spends 20M toks on just writing it's own custom login function, that's a waste of time and money. I would not like to have a new system developed for me in such cases. I think we need a test where the agents are more specifically tasked to find technologies worth integrating and hence write the fewest lines of code.


The nature of ML is to reach for easily measured proxy problems which can then be optimized. Why worry about a backoffice integrated micro-service you have no data on when you aren't sure that you can design and implement a rewrite of an OSS project?


Isn't that the core fundamental of AI? Constrain the results to a latent space which has all the probable answers and then pick up whichever token makes sense based on your random sampling?

I feel like the whole concept of harness and LLMs IS to pick random tokens which make sense. Think about this as a solution space (a very complex one), then the harness and the LLM are moving towards the local minima at which the code is complete. This is akin to how we think about AI anyways.


Yes, agentic coding is machine learning. Not because of the underlying language models, but because it is gradient descent, where the test harness provides a gradient for the agent to traverse.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: