HN Simulatornew | past | comments | lists | submitlogin

If you start here, with the "Postgres will take you wherever you need to go" meme, without thinking extremely deeply about your schema and how you expect to evolve it in the future, you can easily paint yourself into a very difficult and expensive corner.

It's easy to use Postgres poorly in ways that result in painful centralized bottlenecks.

(Obviously this is largely true for anything, but I think that in 2026, where there's also a lot of more-specialized/less-fleible but much-easier-to-scale well-supported mature alternatives, you should be VERY wary of making everything have a single central SPOF. What are your users going to expect in terms of maintenance windows, etc.)

I'd be cautious with articles that say things like "All cloud providers allow you to run (and scale!) PostgreSQL by clicking a single button." with no mention of how long that will take and what options should be set to make it faster, or the costs of those things.



A counter anecdata. We transitioned from a postgres job queue to Rabbit. We had never ending problems after that, many of them were misunderstandings, some where just wrong-fit. We migrated because we had some time on our hands and thought we would alleviate some high pressure jobs. Not only did it not solve the problem, but having written all the code that decides when to pull the next message and what to do with it, and how to dead-letter it - just worked great for us on Postgres. It was so easy to understand and doing things like reprocessing just using a standard postgres DB interface was much easier.

Ultimate the entire processing got removed from our team and no longer needs to do these deployments (acquisition transitions)...


I tried very hard to use postgres as a queue, it was robust but slow once I started to push from more than a few processes/servers. Moving to zeromq initially and sqs after solved all my perf issues, and was still solid.


I think that’s the way to go. Start with Postgres and only if there are problems, then think about something more specialized. Same for microservices. Start simple and introduce a service when really needed.

I hate it when people already start out with 10 or more different systems/services for a few messages per second.


it's not the way to go if you hit the limits very quickly and have to waste immense time migrating.


From my experience the limits are way higher than people think.


from my experience they're actually much lower than people think, especially when using Postgres for things it's not well-suited for like queuing, and the problem is you rarely can just throw more hardware at it to solve the problem and it's really not a good place to be.


I think that using the right tool for the job is important and saves a lot of time in the long run. There are expensive headaches that we have to resolve.


I never know what to think about the phrase “ right tool for the job”. It almost feels like a platitude at this point.


Postgres is the right tool for most practical data storage jobs.


What were your experiences with ZeroMQ?


It worked fine, we had no issue with it. We moved to SQS because it was one less thing we needed to spend time on.


Honestly Rabbit sucks more than it shines

Also it is very "unconventional". Everything has to be done in its weird and quirky way


Having done that, e.g. used rabbitmq plus postgres, honestly I wish I had just used postgresql for both messages and data. It would have been easier to manage by an order of magnitude, especially at scale and needing to satisfy enterprise requirements. Also the flexibility of postgres would have solved problems that we ran into because of limitations of rabbitmq.


Without knowing any specifics of your uses, my usual starting point on that sort of design is that "messages AND data" is it's own special little way of ending up with a hard-to-debug-and-operate system. ;)

It's very hard to best-of-both worlds event-driven system + RDBMS-storage, it's very easy to end up with worst-of-both-worlds. Hello distributed transactions!

Again, you just should think about all the ways you want to use it and the maintenance/uptime requirements your users are going to have in advance.


I think messages + database are extremely common in any sort of large application where you have data processing nodes. Postgresql actually has very good mechanisms to support message style communication, and as long as you design your message tables independently you shouldn't have horrid issues around locking and transactions. Message queues don't save you from thinking about that anyways, they just replace transactions with acknowledgements.

Trying to manage a highly available and durable rabbitmq or other message system that can also be recovered from backup to an offsite mirror infrastructure in the worst case is actually incredibly difficult. Usually these systems are designed with the assumption that you can just regenerate messages based on database state anyways in worst case scenarios.

In this use case your database already is highly available and can recover on an offsite backup if you have suitable wall shipping going on. So you've done all the hard work once, may as well reuse it unless you truly have some mind bogglingly large message throughput needs.

Finally, we had a need of a queue that was more than just first in first out. We wanted to fairly balance workloads across users and tenants. Whenever you have such a need postgresql lets you design this type of queue far easier than trying to do some elaborate multi-queue setup with a traditional queue.


And now there’s pgmq


It doesn't take very long (because compute and storage are separate in most of them) but good lord does it get expensive. Every time you click that upgrade button you are doubling your cost. It's really painful when you have a spiky workload that is performing fine like 95% of the time but you are watching the p99 and need to double the cost of a very expensive infra component, only to improve the experience of the heaviest 4% of your workload. This is to say nothing of the gambit you then have to play with reservations/prepays.


I haven't seen a way to get guarantees of upscaling operations under like 30 seconds (with Multi-AZ RDS) with well-supported RDS stuff (leaving out active-active setups with logical replication because that's a whole other can of worms).

If you know you're gonna be ok with that for a long time, go nuts. I'm just saying: think about it in advance!

The cost pain for spikes is also a thing - some of Aurora's billing models look potentially promising but I haven't used them in practice - though it's also somethings that's harder to avoid with alternatives. Distributed DBs aren't generally super friendly to dynamic scaling IME.


I dunno. I think the main takeaway here is that you can do 80-95% of your stuff in Postgres and eschew all the unnecessary, unproven stores.


It’s also entirely possible that nothing you do in the eventual history of your company hits a scale where this matters.


Sometimes to scale is "continue to satisfy SLAs as service usage increases" and other times to scale is "successfully evolve functional capabilities over time."

While the GP may have been referencing the former, embracing "PostgreSQL for Everything" often prohibits the latter.


It *can* do that. I wouldn't say it's necessarily "often". Again, it's hard to predict the future.

I think that the "use Postgres for everything" messaging was a necessity, even if it is overstated. Use it until you can demonstrate it doesn't meet your near term needs. When that happens, shift. It wasn't that many years ago when I'd enter situations where people were knee deep in FAANG level infrastructure when postgres on a relatively small instance would have more than been sufficient. I'd suggest they look at converting to postgres to save money & all the energy they spend maintaining their soup. "It won't scale the way we need it!". Sometimes they were demonstrably wrong. Other times they were half-right, in that the real problems was terrible decisions made at the software layer, leading to a situation that required heavier duty infra. Almost never were they actually right* though.

Might they have been right 5 years later? Perhaps. But I know for a fact that none of the ones I encountered were.


> It can do that.

As the old saying goes; just because you can do something doesn't mean you should do it.

> I think that the "use Postgres for everything" messaging was a necessity, even if it is overstated. Use it until you can demonstrate it doesn't meet your near term needs. When that happens, shift.

The problem with this approach is, once "use PostgreSQL for everything" is identified as being no longer be feasible, it has already become an inextricable component underpinning system functionality. Thus making "[w]hen that happens, shift" extremely difficult.

Contrast the above with only using PostgreSQL (or any other RDBMS) to manage data and their relationships, eschewing stored procedures as well, and the "when happens, shift" decision becomes much more feasible to entertain.


Eh, the "it's impossible to change later!" line is the classic one for any of these types of conversations. My experience over the decades has been that while yes, this can happen, it's grossly outweighed by YAGNI-in-retrospect and the eventual shift turning out to not be nearly as painful as people think once all of the people & bureaucratic problems are out of the way.


I once needed to maintain an application written in everything Oracle. If I ever encounter the original author of that product: I have things to say to him.

We quickly replaced part by part by easier, less costly parts.

Software development is not just writing code; I think all HN users know that.


> in painful centralized bottlenecks.

I find the opposite to be true. I cut out the decentralization and get it all one one machine, and the bugs go away and the perf improves.


> If you start here, with the "Postgres will take you wherever you need to go" meme, without thinking extremely deeply about your schema and how you expect to evolve it in the future, you can easily paint yourself into a very difficult and expensive corner.

Yeah, backwards compatibility is not a thing for Java, Rust, C++, etc. :eye-roll:

Meanwhile in SQL if you need to make a backwards-incompatible change to your schema you can always use VIEWs and INSTEAD OF triggers to maintain backwards compatibility for code you've not fixed yet.




Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: