You can't go half-in with AI; need to go all-in. At 10:39, he's talking about how humans are reviewing the code. No. You cannot go half-in; have to go all-in and set up AI reviewers that match the best human code reviewer's heuristics. You cannot go half-in and have QA doing manual integration testing. You have to go all-in and AI driven, automated QA.
Speeding up delivery of code is forcing a lot of orgs to face a reckoning of inefficiency that they've built up over years that need to be fixed.
It wasn't until after factories were redesigned from the ground up to take advantage of electricity that the benefits of electrification were realized.
There's a persistent question in my circles: "what does it mean to be AI-native".
Your analogy here is apt because AI-native teams are the ones that are doing the impossible. I know 4 non-technical solo-founders (all had previously courted me pre-AI coding to be a technical co-founder) that are effectively shipping real products with no engineering team. One has 4 large universities signed.
These guys are all-in on AI because they could not exist without it; by necessity every leg of what they do is AI.
Is there anywhere I can find a matrix of what services are emulated and how complete that emulation is? I tried the docs but just ended up going round in circles.
Pass rates from the release's scores.json (commit 7753983), joined with the eval suite's manifest.jsonl. Tests = behavioral checks touching the service. All = overall pass rate. Happy/Reject = valid requests vs. requests AWS should refuse. Ctrl/Data = control plane vs. data plane. Solo/Integ = single-service vs. multi-service checks. Wire# / Wire% = count and pass rate of recorded request/response replay tests (a test passes only if every call matches).
I believe AWS already autogenerates their API clients (e.g. boto3) based on some authoritative internal service api spec so I imagine that spec could be used upstream of this tool as well.
Except, this is not defined in schemas, this is merely defined in prose around the documentation.
So your Rust API is full of Option, and it gets really annoying when you're testing things like say Bedrock, which takes 10 minutes, and then fails because what you submitted 10 minutes before doesn't match their schema.
I understand this can be fixed by reading the documentation in more detail, but it has gotten so verbose, and sometimes you just want to start something.
Yeah whatever they did internally to get service teams to actually adopt AWS Config ~5 years ago needs to happen here or it’s going to get stuck with marquee services and that’s it. Especially given this is a much much larger ask of each service.
AWS has lots of unfunded mandates like this and they historically just depended on bleeding the souls of engineers and managers to get done.
I suspect this is built by agents through bleeding the soul of GPUs until it gets too expensive (unfunded mandates are free after all on the OP1/2). Then it just decays into the place all good PR/FAQs decay once their sponsors are promoted, burned out, or are laid off.
There are many AWS services. Simply offering an alternative to a few of the most used could be a good business. It doesn't have to cover all AWS in all release versions, just enough to get and hold some clients
Nothing personal, but unless you can link us to a public announcement made by Amazon, your statement is worth less than nothing.
Official statements made by a company are still poor representations of their current business plans, and don't mean anything 3 months later (no matter how much they pinky promise). But a HN post isn't even that.
Additionally Swami, our SVP of AI blogged about it separately and our CTO is tweeting about it - which is more than a number of products get in terms of coverage.
I didn't mean announcing the tool, I meant an announcement committing to the maintenance/support that users like the GP poster were looking for. My bad on describing that clearly, I'm gonna get clowned on in the comments and I'm not turning this around.
I understood what you meant. I was just trying to add evidence to support my claim that it was a real product and would be maintained. And yeah, votes on comments can be fickle.
I think one thing that concerns many people is that this looks like AI slop, which it clearly is according to the blog post.
I think this is a super interesting direction, but how can we trust the accuracy of this simulator? Agentic loops are great and all, but they can still miss plenty of things.
Or maybe a better question: where is the accuracy of this simulator going to be guaranteed enough that we should trust it?
To me, something like this is actually what vibe coding thrives the best at; tight, looped, easily verifiable, test-driven dev work. In fact, this is probably the field it can do with little or no intervention, if you're willing to waste tokens.
Is it? How is it verified that this catches all the potential edge cases of interacting with AWS? Or is handling all the edge cases of AWS not in scope?
Like, what am I actually getting here? Is it just happy path operations being correct here?
In the near future, every case will be a real edge case and every one will be a guinea pig.
The border between AWS and AWS emulator becomes fuzzy. And the users of AWS will be agents as well. A completely offline emulator will save a lot of resources and could be a great training ground for more focused (offline) models.
Back in the day, we assumed that when a major company open sourced code, that code would have problems. We invented all these things like ‘issues’ to deal with this.
GP statement still stands, actually being strengthened. This blog post doesn't even mention nor seems to care about maintainability or long-term support or anything that users will care about. Only talks about how cool the team is and how amazing they're at using AI, multi-agent workflows, and this kind of bs. It's funny how author talks so much about testing when they themselves also boast how they haven't written or reviewed any of the code.
I mock (ask Claude Code to make something run in a container) my own AWS services today for CI and development. I'm excited to learn more about how AWS uses it and hopefully use this for CI and development.
Good! Been waiting for a first party thing like this.
It’s actually interesting that such a thing exists. It’s in Amazon’s interest that the only way to test a lot of services is to have a test environment in AWS.
>> Skeptical of using because it's not clear whether this was just an exercise or a real product they are going to ship and maintain.
> It’s a real product that we plan to maintain.
29,684 commits in a repository less than a month old does not instill confidence in this being a "real product." That is roughly 1,000 accepted commits per day.
Until the plan changes, which at some point, it will. Comment is still appreciated for what it is.
Edit: Comment is in the spirit of "it gets tiring to build on sand which is at the whims of corporate management folks far disconnected from the actual work." If someone is asserting "this will stick around," words are insufficient, words are free. Show me the contract, show me the penalties for failure to abide by the assertion.
Amazon team killed the Copilot CLI[0] which I thought was the single best thing they ever did for AWS infra. Made it tolerable to deploy on AWS; still salty about this one.
Yes, we are still struggling with a good replacement. The way they handled this - from steady release cadence to complete stop with no info at all but just speculations makes me very reluctant to adopt AWS maintained projects.
There has been no food replacement as far as I can see that doesn't involve writing a pile of code for what Copilot was doing with a few lines of config.
Google builds up a product over time and abandons it. This is annoying for long term adopters.
AWS has a habit of throwing a bunch of products at the wall, but only bothering to maintain & build up a few of them. This is annoying for early adopters.
Azure on the other hand. Will throw products out, ensure they are never consistent with other offerings on their platform, and then deprecate it from under you while promoting the same thing with a new name.
I think there was some truth in this in the early days of Azure (and all clouds) as the companies were looking to work out what cloud services were really needed. I don't see this as much any more though esp when a services makes it to GA they have published commitments for service availability and API compatibility. (I was in Azure for 10+ years so definitely let me know if you disagree)
Yeah I disagree. First of all, Azure is mostly garbage so avoid it if you can. Second of all, Azure deprecation follows Microsoft's general policy of deprecation which is The Fadeaway Policy. Don't publish anything publicly about it, gut the team to go work on something else, transfer the service to be one of 6 dozen other services some random offshore team is "maintaining", experience enough outages and trouble and during every support call they nudge you to move to something else, but the service is "totally not deprecated" because we haven't gotten a memo that says that, but we're only doing "security" fixes now. wink wink, nudge nudge.
Fine to disagree - but it would be better to share real insights rather than just blanket statements. Sounds like you have had some bad experiences - maybe you could share more?
Let me quickly compile a list of the last 700 failing bicep deployments with an “Internal Service Error” or No capacity errors along the all the no-op idempotent puts that take 30+ minutes for some reason, the broken networking system all up, the constant support advise to move to different regions, except half the things you use are not in that region, or the constant advice to just delete everything and start again or the half docs half marketing fluff docs. Azure only exists because large enterprise looks at a list of checkboxes before signing a contract. No reasonable person should ever use that garbage unless you’re offered deep discounts to do so, and even then, limit your usage to things you can migrate easily whenever the discount expires.
And Azure will send you an email every day about how you need to migrate X to Y without any way for you to know where those instances are or how to find them.
You're certainly not wrong. But in the wake of Localstack's changes, the landscape has been full of slopware and low trust anyway. I don't think this is worse, for now at least.
That is pretty much the key thing IMO. It needs to be able to run our actual code, locally, with fairly high fidelity to the real thing in terms of behavior.
For the AWS people behind this, it means going the extra mile(s), with a level of care and depth that you typically mostly see within great startups. Perhaps agentic coding will enable AWS to reach this, perhaps it will stay relatively abstract.
This site is riddled with companies and commenters officially representing massive entities that go back on their promises. Are you saying that we should trust them just because they represent their company?
It comes from a history of this sort of thing where an Amazon staff person will say that X will always or never happen and then their promises falls flat because of big company dynamics.
Same commenter a few months later: "I (work at Amazon and) already have to go in once a week, and I literally do NOTHING that day. [...] This is a disaster in the making."
I've tripped several times in my life while walking. I've even hurt myself a few times. I don't go outside wincing to myself about whether I'm about to fall and hurt myself.
You choose the attitude you want to have in your life to disappointment and the attitude you bring to this site. You chose cynicism. You're going to crowd out people who don't make the same choice.
I crashed my motorcycle years ago and have mild but still lifetime injuries. As a result, I've changed my behaviors and have put away my motorcycle. It's given me a place to reflect on what my own life is about, where I want to go, and the pain my family felt when they heard about the crash.
I choose the attitude of learning from the past and adjusting my behaviors.
I had a similar experience on my bike and chose to get back on, though it certainly took me a while, and I've stopped riding in weather and in other conditions. My family supported me in my decision. I guess you and I are just wired differently.
I try not to share my cynicism with the world too broadly because I think our modern era has too much of it. I think humanity is advanced by the wild dreamers.
You're making a lot of assumptions about me, what my risks are/aren't and... dude, can you not, please? I sold my motorcycle and once I recovered, I rented a motorcycle out in New Zealand as a sendoff and I reflect on that time with deep joy and appreciate the deep joy that others feel when they ride motorcycles. But I would not blame anyone for stopping, or infantilize them like you're doing to me now.
Yes, we're obviously different, but... dude.... dude. this is a damned conversation about a damned cloud server emulator that's not even close to 100% in-parity. It's perfectly healthy to inductively recognize that using such a system, like its sibling tools, might not continue receiving support.
> Sometimes things suck and it's okay to express our frustrations about them.
I guess I care because I've been on this forum for a very long time and everyone individually says that, but then the comment section as a whole turns into a hazard zone. Anyway I think I've engaged too much on this site for today.
Strongly agree. I've been wanting an official emulator from AWS for years. It looks like they've finally produced it, and it looks like somebody that works there is saying they're planning on maintaining it. And they've blogged that it was released. And somehow the comments are focusing on the fact that they haven't promised to maintain it for a long time. It's extremely rare to ever see statements about longevity when something is first announced.
We can appreciate it while also recognizing that history repeats itself into disappointment while also recognizing that Amazon has done large social media campaigns (remember the FC Ambassadors?). I would have been more excited about this ten years ago.
Amazon saying that they'll continue supporting an open source project and quickly forgetting about it isn't on-topic? It's why I asked the question to the AWS person...
In my experience Floci has been the epitome of vibe coded - huge amount of changes, with things breaking randomly between releases. Moved away from it as soon as I could. Still better than Localstack :)
I didn't have a good experience with test containers, but perhaps it'd work better with fast starting containers. We run our test fixture with compose, and have the tests manage their data in isolation - so we don't pay the startup penalty in our test suites.
I think I'd trust this project at least as much as Floci. Embedded Rust mode also looks very interesting for us.
I only use Testcontainers with "explicit" tests (do not run by default in CI; must be run explicitly). This lets agents exercise the integration with a realistic fake locally, but does not incur the cost in CI nor for any dev unless they run it explicitly (usually also with a category so it's explicit + category filter)
Do you have any recommendations to start learning cloud infrastructure with a almost real setup? ive heard of localstack but also a lot of negative reviews, so im not sure how to get started..
Not really. I was trying to use their own Q AI to help me get something set up and it was constantly wrong about what screens to go to, what was on the screens etc... If their own AI doesn't understand AWS, I don't have confident this will either.
No docs site linked on the GH. 3 weeks old. Blog post talks more about the process than the tool and opens with:
> How we built the agents that built AWS Emulator
> Thousands of autonomous AI agents wrote every line of code in AWS Emulator, a new local emulator for AWS now available for download.
Feels more like a science experiment than a product from that announcement.
Doesn't feel like a real product. But I also didn't look very hard for docs; I kind of expect it to be linked on the GH in the site and have a nice `llms.txt` for me to give to my agents.
Sure they are, but they could have polished their product and showed some love by configuring the repo details and put the website there. Did you look at the blog post as well? Does it look like a product that some team there loved and will care for? Or the output of some exercise for some enterprise content?
This post https://www.aboutamazon.com/news/aws/how-autonomous-ai-agent... doesn't set the right tone for me at all and it has nothing to do with agents and everything to do with the lens of looking at this as a science experiment instead of a real enterprise tool.
> they could have polished their product and showed some love by configuring the repo details and put the website there
This feels like such a weird nit-pick when AWS is criticized for not moving quickly enough. Why should I care if there's a link in the side bar, when there's plenty of links in the `README`? If anything, I'm happy that there's alternatives to Localstack that are supported by AWS themselves that I can begin incorporating. They have a well written `README` that I can start working with and a User Guide to get me started. What else could I want?
> Did you look at the blog post as well?
I did.
> Does it look like a product that some team there loved and will care for? Or the output of some exercise for some enterprise content?
Do you even have the grounds to cast judgement like this when you admittedly didn't even look hard enough?
I looked, but you're mistaking my critique here: I value polish.
This lacks polish.
Small details like getting your GH repo set up for a public audience. Lacking. Compatibility and support matrix clearly advertised? None.
If this public facing artifact exhibits a lack of care and love, what does it say about the code? I am all for vibe coding and agent software factories, but then the humans should show some love with all of that extra time saved.
If your complaint is about polish, you’re a very poor writer. It should not have taken this many replies to get that point out. But regardless of your communications skills, I disagree - this product looks fine for something released at this stage.
You might want to reconsider how you present the critique then, because that's not how your root comment came across or any of your subsequent messages, especially when engaging with (presumably) actual AWS employees in this thread replying to you.
Do you agree that there's a polish here that shows a team's pride in that work that is clearly missing in this work?
> ...especially when engaging with (presumably) actual AWS employees in this thread replying to you
??? You and I are the customers. What even is this statement? Are AWS employees our benefactors? Wow. I do not care if their feelings are hurt because this "product" clearly shows a lack of polish and love.
> Do you agree that there's a polish here that shows a team's pride in that work that is clearly missing in this work?
No and I think the inverse is also possible and if anything more likely. You can make a website look really pretty, really easily and have the product still suck. This is "product as a slide" in a new age where making visual eye candy with AI has never been easier.
> I do not care if their feelings are hurt because this "product" clearly shows a lack of polish and love.
The point isn't about hurting feelings. The point is about providing critque and feedback based in reality, not based on "vibes." You're trying to gatekeep people releasing software because you dont like the way they're launching it lol.
There's a right way and a wrong way to assign a website URL in GH.
The team didn't even take the time to review these small details of the repo, you know, by editing the repository details to place the docs URL there.
I mean, I don't know about you, but I feel like I still like to see people care about the stuff they produce and missing something as basic as the repo website URL in GH settings shows a lack of polish and love.
I just write my TS like I write my C#. They are similar enough, IMO. I ended up writing this as a precursor to our team's move from TS to C#: https://typescript-is-like-csharp.chrlschn.dev/
The main things I don't like about TS are the volatile tooling, bad habits from other devs (not declaring types), compromised ORMs, require lots of package hunting, and module system.
AI is great at following patterns and loves to write tests. Start with good patterns that most teams would not have the discipline to write by hand and it is possible to get very robust code.
AI will write tests to the level of depth that you ask it to.
Try:
- Clearly identify the invariant being tested focus on exercising the invariant
- High volume, low signal tests increase CI and build times
- Avoid low signal tests; raw coverage is not the objective
- Avoid write-then-read tests when risk is low (no transformations, no business logic in application code on write or read path)
- Use explicit test execution [Explicit] when test code is used to exercise code in local environments only and should not run in CI
What's useful is to have a second agent that reviews the tests impartially and trim the ones that are low signal.
You also need to understand why agents write high volume, low value tests: because other ways of exercising the code (e.g. Playwright, E2E, curl) are even more expensive and not as consistent. So you have some recourse here by: 1) giving agents another way to exercise the code (e.g. provide it a REPL to your code) or 2) tell it to write some unit tests as explicit only (does not run in CI).
For our tests, we also added a small helper library[0] that cleans up tests to make them easier to read (smaller context for both humans and agents) as well as to write. Agents write average code by default. Your job as an engineer in this age is to help agents write good code.
> VCs can't hop on a flight to do some due diligence?
I think it's more of an issue of familiarity with the regulatory environment. VCs and PE know how to operate in specific legal and regulatory landscapes. With standard paper.
Asking them to operate in new jurisdictions requires a lot of effort and legal know-how; it's not just a matter of cutting a check, but understanding how to navigate the legal landscape for these financial transactions, rights of the investor, tax implications, legal recourse, etc. There's going to be a lot of friction such that they'll either: 1) not seek those opportunities, 2) ask the founders to move to the US if it's a really promising team (see this a lot; was in one startup where a co-founder moved from AUS to the US).
Melanie Perkins talks a bit about this in the Canva episode of How I Built This[0] and how SV wanted them to relocate. AUS government ended up matching some of the investment through a new program so they reached their target for their raise.
> A trading floor is a bad place for unreviewed code written by people who can't read it, and the information barrier problem would be reason enough on its own. Requiring that agent-written code come from people trained to review it is a sensible rule for a regulated business.
Jokes on them: devs are also losing the capacity to read the code. Can't mix move fast, high volume with "read code".
If a team wants the benefits of velocity with AI, the consequence is volume.
To deal with the volume, the system needs more checks than humans can handle. Additionally, the humans no longer understand the details at the low level of implementation at scale. Therefore, the system only needs to assert that it is following agreed patterns.
So the initial scaffolding needs to be agent friendly. The processes need to be aligned to agents. The tooling needs to be agent operable and put on rails. The patterns need to be documented to be enforced.
Otherwise, teams get volume that cannot be handled by humans. So full speed agentic engineering requires teams to be all-in.
A piece of this is to leverage the harness to normalize or standardize output. Standard patterns, standard architecture, standard UI libraries. These are all much easier to enforce with agents than with humans, but requires extensive platform and harness engineering.
Teams do not really understand this yet. Engineering teams still think their job is to build product faster. By the 10th time you tell an agent "implement issue #15639", you realize anyone could have done that. Engineering's job now is to own the enterprise harness (context layer, ephemeral agent env, ingress channels, auth proxies, MCP servers, quality gates, agent ops, etc) so anyone can build the product. That's what "all-in" means.
Once this system is set up and well executed, it really doesn't matter who writes the initial prompt. An enterprise harness designed to enforce safety, security, and best practices applies to a senior dev or a junior dev or a product owner all the same. All-in on engineering the enterprise harness, not the product.
That's the part I don't think requires all-in, because what you're talking about is just traditional enterprise dev right now. Having good infra in no way requires your workforce to have any % commitment to hands-free generation.
You have to go all in on agentic engineering. And if you're all in on agentic engineering and doing it well, then anyone should be able to build safely and no one needs to look at code. The code should look highly consistent and agents should be enforcing best practices, approved architecture, running security scanning, etc.
The gains at the middleground compromise are not really significant and, IMO, creates frustration and friction.
Works best to go all in. GS should let everyone have CC and GH and focus on making their enterprise harness safer for everyone.
Key is to get folks to unload their heuristics and rules they use for their daily workflow and judgement.
What does good code look like? What are you looking for when reviewing code? What anti-patterns are your senior devs looking for? How can you see the evidence of agent adherence?
Some of it comes down to system architecture and essentially creating rails. Agents are very good at following patterns and copying existing code (it's all just text). Establishing strong patterns, documenting them, and creating deterministic checks (linters, analyzers) helps unlock broader audience of builders.
There's nothing that fundamentally cannot be written down as a set of heuristics and given to agents to enforce except that most folks in software are very bad at writing.
In a single session? Agree. The agent is bad at enforcing rules against itself. In a multi-agent, disconnected session setup? You'll see that this works very well.
Paseo has been pretty amazing at getting more autonomy. It's the way the agent profiles work. It is effectively a fully adaptive "auto-harness" with just the tiniest amount of config.
Fully switched from TUI 4 weeks back while testing it out for another use case.
But only in documents that make it through RAG’s hard cutoff. There’s an explanation of this on the related Stanford site at https://memory.stanford.edu/ (see “How MEMORILLA differs from RAG”)
Do you have more background on this? My team actually ran into some issues with GH when we tried this earlier this year (felt like it was around that same time).
We have a few simple rules for Accounts on GitHub's Service.
You must be a human to create an Account. Accounts registered by "bots" or other automated methods are not permitted. We do permit machine accounts:
A machine account is an Account set up by an individual human who accepts the Terms on behalf of the Account, provides a valid email address, and is responsible for its actions. A machine account is used exclusively for performing automated tasks. Multiple users may direct the actions of a machine account, but the owner of the Account is ultimately responsible for the machine's actions. You may maintain no more than one free machine account in addition to your free Personal Account.
One person or legal entity may maintain no more than one free Account (if you choose to control a machine account as well, that's fine, but it can only be used for running a machine).
Speeding up delivery of code is forcing a lot of orgs to face a reckoning of inefficiency that they've built up over years that need to be fixed.
reply