Depends on your build, system configuration, etc. I worked at a place where the data sci’s had a remarkable talent to build docker images based on python images they found in random places and you could wind up with charsets you wouldn’t expect. Managing the problem in code didn’t entirely work because if some third party lib print()ed something that had characters not in the charset it would crash.
It was the interpreter's default for reading Python source code files since 3.0 (https://peps.python.org/pep-3120/), but not the default for built-in and standard library functionality. But it may have been hard to notice on many systems.
Even on Linux it was not. Python would use the locale’s encoding, which is not necessarily UTF8. The POSIX locale was special cased to enable UTF8 mode tho.
That benchmark could really be a good AGI test. Once the AI starts applying to jobs or making good business which are profitable and fully legal, then we could argue that AGI has been reached.
If you could construct a sandbox to test this, where it doesn’t touch the real economy, then yeah it’s a great benchmark.
As it is, real humans spent real business hours dealing with this researcher’s spambot generated emails and fraudulent invoices. Individual recipients reported feeling harassed.
This isn’t a good benchmark. It’s a series of socially destructive crimes committed by the researchers and then documented and published on the internet.
Ask the AI to create a detailed spec according to a few simple requirements. Review the spec yourself and correct what you want changed. Then ask the AI to implement the spec. Each time you request something new, ask the AI to update the spec as well.
