> They may have been specialists in certain things, but what was most impressive was their ability to draw upon and weave together seemingly disparate concepts from different disciplines and use that knowledge to solve problems.
TLDR: I was called in several hours after the downtime started. I architected and built the full system so I knew things were incredibly consistent and how every layer worked (each layer of the software/database, load balancers, the quirky legacy protocols, network setup, etc). IBM was saying it was an issue on the financial institution side, which I validated was not the case practically first (things were down so I was given freedom by the executive team to do as I needed). Since I sanity checked the easy parts first and had high confidence this was something much more wild after scanning through more stack traces than a human should, I decompiled and reverse engineered their code to show the IBM teams where the bug was in their switch statements triggered by the Redhat patch difference and its dependencies that triggered the edge case that stopped the highly available cluster setup from functioning. I created a script to override the system properties being read in on startup to trick the switch statement, show it worked, and got the system back online. IBM made an official hot patch that fixed the issue officially along with paying a lot of money back to the financial institution for their troubles (ie the double contractual obligations for the premium IBM support with included SLA clauses that were overrun and the proprietary IBM software itself).
Germany had a similar scheme for returning soda bottles when I studied there (no idea if they still do). I would save up a few bottles before bothering to return them. Took a couple of minutes max. Hardly a big effort.
No one is standing in line to return one bottle.
My local grocery store has a similar return system for glass milk bottles. You just turn them in at the register when you check out. Even simpler.
That's a nice way to put it.
reply