I have done the math as part of my job and often found that yes, it was literally worth the money to do our own storage in cases. Running a highly available, durable storage system is not particularly difficult. There was an era when it was a common skill for a sysadmin. In the current era there are off the shelf solutions for it, both open and proprietary.
The biggest reason people pay for S3 is because of data transfer cost to other cloud services they are using.
Yes, I’ve also built those services. As I said before, it is totally possible to beat S3 on pricing but the margin narrows considerably when you factor in the full costs — e.g. think about how many places had replicated storage but not bit-rot protection or versioning and built that at the application layer instead — and the savings is often not big enough to make it your top engineering pick for the money & attention required.
Fundamentally you’re playing a commodity game and that means you need a consistent edge to be worth the fixed costs. If your fully-loaded advantage is less than, say, 30% you’re better off negotiating with your AWS rep and investing your engineering resources in something your customers want.
The biggest reason people pay for S3 is because of data transfer cost to other cloud services they are using.