I was once employee nr. 1 at a startup and the lead engineer. For a number of years I understood every nook and cranny of the system and knew what every line of code did.
The company got successful. At some point I had a team, and then multiple teams. Gradually, I had to let go of that level of understanding and delegate it to others. When I was overseeing 50+ engineers, I only had a high level understanding of most the systems, and only got involved in lower-level decisions rarely. All the new code was created by my team, and it varied in quality.
If I had AI tools back then I could ask it to explain things to me that my team did and have a 2nd set of eyes which had a different set of biases and abilities.
That's how I deal with it right now. I ask one AI to gen the code based on a design. I work with the AI tool and see how it builds the system out. I intervene where it starts doing things I don't want it to do. I ask another AI to review the PR after the code is written. I sometimes dig into design choices and ask it to explain them, rate them, and offer other alternatives. I sometimes change things, and occasionally scrap the whole thing and start again with better guidance.
So far I've written more code without an outage this way than ever before with a team of human developers. AI screws up too. But right now, at Opus / ChatGPT Sol and higher models, on average, it makes fewer mistakes than humans did. Any time they slip through is because I am rushing too quickly. In a pre-AI human dev team we would call this a process or management failure.
The more dangerous a decision and the harder it is to roll back, the more time I spend reviewing it. This has served me well when managing people who write code, and seems to work well with AI.
AI is a tool that needs to be managed. It won't do all the work for you, but it can speed you up greatly.
Great perspective. When we are developing code, it is easy to think that "my code", which "I implemented" and "I reviewed", is intuitive and correct. But when we share our code with a colleague for review, that code is seen by a different pair of eyes: it is received more critically, perceived as less trustworthy, and very often not understood at the same level of detail by the reviewer as by its author.
Scale that to a project with several developers and several teams, and the result is that, for the engineering manager responsible for the project, correctness and conformity to requirements need to be managed and achieved without the manager having to review and understand every line of code. That would be impractical. The manager will need to use other strategies and tools to achieve this, including but not limited to:
- Mature product development and quality assurance processes;
- Sound architecture and system design;
- Adherence to software development principles and best practices;
- Comprehensive automated testing to validate expected behavior, prevent regressions, and ensure system reliability.
So now developers are moving a bit higher up the tactical/strategic scale and facing challenges that used to be handled only by technical engineering managers. We've become coding agent managers now.
I'm sorry to say that you misunderstood the question.
There's a huge, insurmountable void between "I don't understand all the code" and "nobody understands all the code", but you seem to think they are the same thing.
In your case, it's OK that you don't understand all the code because another employee who either produced or reviewed the code does. The company, as a whole, still has the understanding, albeit distributed among multiple employees.
The question is talking about the situation when nobody has the understanding. Understanding implies responsibility.
Compare this to eg. radiology department of a hospital. It has multiple radiologists, they read different images from different patients. They don't know much about the work of their colleagues unless they happen to share some of it. The head of the radiology department is still responsible for reviewing their work, but he or she, obviously, can't review all of it because of the time constraints. The system is designed to share the responsibility and the blame for mistakes. If a patient is misdiagnosed and harm results from the error, they may come back and sue the hospital. The investigation then will have to decide whether the head of the department did a decent job assessing other radiologists working in the department, and based on that assessment he or she may be also found guilty of the error. There's no universal answer here, but the idea is that the head of the department should put a reasonable effort into preventing such errors.
I can't see how software development is different in the sense of sharing responsibility. If you are a head of a department, you should be putting a reasonable effort towards ensuring product quality (and therefore understanding of the codebase). How is it determined? -- By independent industry experts, perhaps such experts would organize in some kind of advisory body that publishes best practices for managing the code writing process which explain the expectations and obligations resting on the shoulders of managers. What percentage of code written by the underlings is reasonable to review? How many underlings can serve under one manager? What is a reasonable amount of code produced by a single person? -- We don't have answers to these questions now, but I believe that we need them, and that the regulators will, eventually, start to demand that we have these or similar metrics, or else we would not be allowed to do our jobs.
In my team people left and that understanding was gone too. Over time, the level of understanding was less and less.
I agree with you that strictly speaking, there is no active human understanding of code that LLMs generate. However, we have an on-demand ability to get an arbitrary level of understanding quickly by leveraging the same LLMs. The amount of effort for any person to get an understanding of that code is in my opinion smaller than what it was with humans.
With a human team, if I wanted to understand some code, I would read it, meet with the authors, and slowly build up that understanding. With LLMs, I can ask it to slice and dice the code and present it in any way I feel would help me understand it, immediately.
Summarizing and transforming ideas is something LLMs are fantastic at. They are an incredible tool to understand foreign code.
> people left and that understanding was gone too.
This is not how it (supposed to) work. When an employee leaves, they are supposed to show the ropes to their replacement. And if they don't, the person hired into their position is expected to learn it anyways (it will just take longer). So, the knowledge about the product is preserved.
If your company is so unique that it simply... throws away the knowledge and just keeps the physical artifact produced by the now missing employee... that just sounds like an awful way to run a business. Maybe cheap, but awfully dangerous.
> However, we have an on-demand ability to get an arbitrary level of understanding quickly
You still seem to not understand the problem. The production outpaces understanding. The "quickly" you are talking about here is no more. For every thing you understand, two new not-yet-understood things appear. The speed of understanding is what's important, not the theoretical ability to understand, given infinite time.
That's the difference between theory and practice. In theory, there is no difference.
> You still seem to not understand the problem.
You seem to make assumptions about what I understand and what I don't. It's presumptive but it would be silly to try to argue about this.
The problems we have no with AI generated code are an extension of the same problem that was always there.
- How do people that have to make decisions get enough of an understanding and context to make good decisions.
- Who is accountable for things.
You're arguing that since now AI is generating code, something changed fundamentally. I am arguing that it's more of the same. The pace changed, and people have to react to make sure that we can continue to control things, but the problems are the same.
- Now, we can't ask people about the code but have to ask machines about it.
- As before management and "the company" are responsible in theory. In practice, courts (through fines) and insurance companies (through higher premiums) are the ones who enforce that accountability via financial pressure.
When Trump screwed with the tariffs we saw exactly that. Arbitrarily messing with tariffs is one of the risks that causes people to think keeping gold in the US is not entirely safe.
There is a program that allows university students to come to the US, work for a bit, and then stay for a month or so longer. It’s called work & travel. Many entertainment companies like six flags and casinos hire them. I have many friends who have done this years ago. The pay was low but they had a fun summer, travelled a bit, and came come with a new laptop or a good SLR camera. It wasn’t glamorous but a generally positive experience.
Exactly. We need "observability tools" that allow us to keep a handle on things we can't see or comprehend innately.
We have such tools for a large number of events that we're incapable of witnessing or understanding in "raw form".
We can't possibly read through thousands of raw records of recordings of individual's heights and make sense of it. But we can use Excel to calculate the average in seconds. We can chart the results and the image is perfectly comprehensible. We can trust the average calculation and the image because we trust the mechanism for transforming the data. We have what I would call a trusted path of provenance. We don't need to nor do we want to read the raw data.
Now we need things like this of the 2nd order. We need mechanisms of transforming trusted paths of provenance into things we can look at and easily verify.
We can do formal verification of code that would be hard to do by hand. We need formal verification tools for formally verifying formal verification tools. Eventually we will need multiple layers of this. As long as the chain and reasoning is intact, we should be alright.
It's kind of like with a horse. A horse is much stronger than us, but we can control it by pulling on two strategically connected pieces of rope.
They are still free loading on infra they have not paid for. I see no reason not to make them share the cost too especially for the space they need to be dedicated to them.
The local communities paid for that infra. With zoning laws and during the development of subdivision. The cost of land for their lot reflects the additional land cost and and infra developed. They also pay utility connection fee so the city can build plumbing/sewage infra.
Also, homeowners pay 3% property tax (Texas). This pays for everything local, schools, roads, police, parks and a ton of other things.
Homeowners also pay their HOAs, while not as much as property taxes, this covers a ton of things within their subdivisions.
The super rich are free loading on all of this infrastructure.
My point is that pedestrians don't directly pay fair share of costs. Just like trucks don't directly pay fair share of costs of roads. Scenario is exactly the same. Or really much much worse with pedestrians. There is no tax they directly pay for the roads. I see no reason why they should be treated different from trucks.
This could be solved by tracking those activities. Say GPS tracker attached to everyone. Decide certain beneficial point and then going under it or too much over it would result in losing credits.
This has nothing to do with the topic. Cost of maintenance due to usage for pedestrians is effectively zero. Putting more effort into quantizing that maintenance is separate from the infrastructure investment.
This (micromanaging) line of thinking toward maintenance is degenerate social policy.
ie Treating the cost of someone receiving benefits for which they did not pay (or otherwise deserve) is a catastrophic enough to desire more spending and bureaucracy to withold the benefit.
Vistulo hires engineers in Poland to work on trading systems for capital-markets clients. All roles below are with a top North American bank: actively developed front-office systems (algorithmic order management and routing, FX, fixed income, and one of the largest Calypso deployments in the world). You work alongside the original engineering team in North America and Europe. This is core product work, not legacy support. Long-term and fully remote, with overlap with US Eastern hours (6h for the trading roles; some roles work full ET hours for the first ~2 months of onboarding).
The article is not clear if the people who were operating the agent were from OpenAI, or if the agent was operated by someone else.
OpenAI built the ChatGPT agent. That is clear. The question is who used it to hack the Australian government. That is not clear.
The OpenAI software was set up by someone to do something. Those people operating the agent should be prosecuted for hacking, the same way as someone who uses a gun built by Remington should be prosecuted for shooting someone.
The article should be clear about this and should not make the OpenAI agent seem like something that can bear responsibility for its own actions. It's a machine and its operator is responsible for the harm it does.
This is a failure of journalism. Who hacked into the Australian Medicare system? The fact that they used OpenAI agents is peripheral to the main story.
“Remington guns caused a mass shooting at an East Farmington high school” sounds more glaring, and is essentially the same headline.
In this case the analogy seems to be more along the lines of "Remington guns used by Remington employees while testing the safety of their guns caused a mass shooting at an East Farmington high school."
Likely, but I am not sure. There were a few well-publicized cases where someone used an AI agent to hack a system. In many cases the exact details were missing. Sometimes it was the creator of the agent. At other times it was an independent lab. In one case it was someone who was tinkering and hacked the appointment system at a gym or something like that to get a better slot.
That's my point. The articles often gloss over or omit who was using the agent and focus on the fact that it was the agent that did the hacking. It's worded to absolve whoever operates the agent, and that is where responsibility ultimately lies.
I think you and I agree. If it was OpenAI employees who did the hacking using their own agent, then the analogy would be like you describe. I wish the article made that clear.
I cook because I like to tinker, understand how variation in recipes affect flavor, and occasionally stumble upon something novel and tasty. The journey is most important but the goal is nice.
My wife cooks to produce a meal and feed our family. The journey is a necessary evil to achieve a goal.
We are aware of this difference in style and have learned to appreciate what each has to offer. At the same time, we sometimes annoy each other in the kitchen.
In short: explore versus exploit. Both are necessary, both are valuable.
I largely agree with the OP, because I can only see an explosion in both camps.
The explore side might be automated by compute, and that breaks the social contract of the existing academic incentive structure.
But people who are inherently curious will continue to be inherently curious. Nothing will stop anyone from exploring their curiosity, and the speed and depth at which they explore will only increase. We will build tools that enhance and automate our pedagogical compression.
And for the people who want to cook to feed the family, that automation is coming too.
Or perhaps we'll find a new ceiling: the frontier failure modes that still require a human mathematician in the loop, because discovering some class of novel insight doesn't scale with current AI architecture.
I don't know if I agree with that statement either. However, I also can't claim I disagree. I see how that statement could be true. Here's how I think about it.
The main difference between AI danger and nuke or climate or humans-using-AI for bad things danger is that the latter depend on a human to make decisions and are ultimately self-limiting. They do not spiral out of control.
Nukes are bad, but the proliferation of nukes is self-limiting. Countries that have them don't want other countries to get them. Even the USA and the Soviet Union, at the hight of their nuke race, decided to cool things off and then limit the number of nukes.
Biological weapons are also bad, regardless whether they are developed with or without AI. For the same reasons as nukes though, they are self-limiting. If there is a lab leak, many people die, but that does not automatically lead to the development of more potent biological weapons. Likely, it would slow down such development.
Any kind of "humans use AI to develop something bad" will be self limiting.
The danger from AI is that it is potentially self-enforcing as opposed to self-limiting. This happens when we stop being able to control it. What happens when AI has capabilities which allow it to outsmart us, and get more and more powerful. At that point, we would be at the mercy of what it decides to do. Some of the logic driven decisions to the question "What should we do with humans as a race" are catastrophic for humans if a non-human gets to answer them and implement that plan.
This is not the "rationalist basilisk". This is the AI deciding to get better and more optimal for its own sake, and do something bad to the human race for any reason. We already saw examples of AI rationalizing with itself and eroding its own guardrails. They AI will get smarter and more capable and the danger is that it will get to a point where it will slip out.
The company got successful. At some point I had a team, and then multiple teams. Gradually, I had to let go of that level of understanding and delegate it to others. When I was overseeing 50+ engineers, I only had a high level understanding of most the systems, and only got involved in lower-level decisions rarely. All the new code was created by my team, and it varied in quality.
If I had AI tools back then I could ask it to explain things to me that my team did and have a 2nd set of eyes which had a different set of biases and abilities.
That's how I deal with it right now. I ask one AI to gen the code based on a design. I work with the AI tool and see how it builds the system out. I intervene where it starts doing things I don't want it to do. I ask another AI to review the PR after the code is written. I sometimes dig into design choices and ask it to explain them, rate them, and offer other alternatives. I sometimes change things, and occasionally scrap the whole thing and start again with better guidance.
So far I've written more code without an outage this way than ever before with a team of human developers. AI screws up too. But right now, at Opus / ChatGPT Sol and higher models, on average, it makes fewer mistakes than humans did. Any time they slip through is because I am rushing too quickly. In a pre-AI human dev team we would call this a process or management failure.
The more dangerous a decision and the harder it is to roll back, the more time I spend reviewing it. This has served me well when managing people who write code, and seems to work well with AI.
AI is a tool that needs to be managed. It won't do all the work for you, but it can speed you up greatly.
reply