HN Simulatornew | past | comments | lists | submit | luipugs's commentslogin

Typical German ßenße of humor.

Much cheaper than doing the same thing in space.

What industries have you seen teams deploy on a Friday afternoon?

We do for clients who have 24/7 users during the week.

I don't think the hesitation to push to prod Friday afternoons has anything to do with the working schedule of your customers. It's more about whether you'll be able to roll back on a weekend.

We avoided Friday deploys because we didn’t have enough usage over the weekend; traffic was oriented on M-F working hours.

It meant that issues would sit undiscovered over the weekend until Monday morning, and lead to a week starting in chaos.

Much better to get fast feedback while the deploy could be monitored against prod traffic.


We did Monday morning IST, on a build we declared valid on Friday afternoon - no pressure to declare premature victory there. The cherry on top was sprint demos first thing in the morning PST so when I first joined the project the demos got interrupted on a regular basis.

I don’t think most people fully appreciated how much work a handful of us did to make sign off faster and deployment less error prone.


or want to have to.

Why wouldn't you be able to rollback on a weekend like any other day?

cause I don't want to work on the weekend. Why would I introduce that risk?

Or if you need more resources, can be a lot harder on a friday evening or weekend to get ahold of someone else to help.

Assuming they don’t have an actual life and aren’t somewhere touching grass.

I’ve worked with several people who either confessed to or were accused of having taken up pastimes that would guarantee they weren’t available during times they shouldn’t be expected to be available.

Management gets rewarded for getting more work out of us and there’s always someone pushing boundaries trying to leach a little more work out of the staff they have. These same people seem to think burnout is a big mystery, because to admit they caused the problem would be to admit they did something wrong.


Because you are patching a critical system that can't be down during the week.

In which case, why wouldn't you deploy it during the weekend? That way, you're already prepared to be working several hours on a Saturday (presumably earning overtime rates) and so it's no major issue to rollback if it goes wrong.

Friday deployments make no sense if you have teams that work Monday-Friday.


Friday afternoon is not unusual if you are facing a weekend deployment. Might as well try to get it done Friday afternoon an not have to worry about it the rest of the weekend.

Several hours. That number goes up quickly when the bus number for something that broke unexpectedly is unavailable. While your coworker who actually knows what that error means is at the zoo with his girlfriend or with his kids.

The whole point of WLB is that we don’t live like monks.


Why do you only have a single co-worker that knows that error? Why hasn't anyone written shared documentation?

Outages are caused by change. Sometimes user traffic, but more often your coworkers.

Do you know how every feature toggle in your system works? Do you know the potential blast radius of those toggles? Can you speculate on the surprise surface area that might be additionally exposed by impedance mismatches and side effects (ie, unanticipated shared state)? Hell does every engineer who wrote a feature toggle know this about their own code? You know who I’m talking about.

If you’re having problems in production that can be anticipated, why are you still having those problems? How long should those be tolerated before someone gets fired?


We host a private cloud, so the only code we write are occasional shell scripts based on repeatable steps that have already been done manually hundreds or even thousands of times. And yeah, everyone on the team has a pretty thorough knowledge of the applications we host. Documentation is usually enough to fill the blanks.

I'm not saying surprises don't happen, just that they should be extraordinarily rare.


I think when we are discussing a very public outage though, we can take it as read that there are any number of problems with the company and this is just the one they got caught out on. And similarly that a lot of replies are from people who have experienced the same with other businesses and/or employers.

As long as you guys don't get underbid by a competitor with a severe case of Dunning Kruger, which I've seen happen too often, then you're good. Usually you have to sell your competence as part of your story. Because you'll never match idiots on price.


Thursday is pretty close to the weekend. Too risky on Thursday?

I'm old, I do prod changes Monday to Wednesday for that reason. I'm not paid to work on weekends. Sure, you can argue that people should be competent enough to never release software with a bug, but I sometimes fuck up.

Or just have a good rollback plan like a snapshot.

If the system I work on needs to be restored from a snapshot then I'll definitely be spending the weekend working. There are systems you can't just rollback and restore a snapshot and all is fine.

And then there’s the Tower of Hanoi of breaking major changes into tiny pieces that can increment without making rollbacks impossible. Always two more steps than you would think you need and requires patience that management often doesn’t possess.

Example, got a feature with a new required field? Add empty field, synthesize a default value for existing rows, start accepting data, validate that data, require the data, use the data.

If you’re doing deployments once a week, you end up juggling two stories so you’re not sitting on your hands the whole time, and two stories divided your attention, so the odds of fucking it up go up anyway even though you’re being careful on paper.

Meanwhile one of your coworkers is doing the same dance on another feature, so you start tripping over each other as team size increases.


So when you roll a patch out on a Tuesday morning and it breaks, how much revenue do your client's lose? I'm all for telling a client to eat it because I'm not working on the weekend for some BS, but there are critical systems that CAN'T be offline during the week.

Me and my colleagues don't want to work weekends, so that goes into the planning of deployments. There is always a possibility of failure when releasing software. I don't see the "how much revenue" relevant to what I wrote - if we deploy Friday and it breaks, we need to work weekend.

We don't WANT to work on weekends either. That's why it is extraordinarily rare that we do. Plus the fact that most clients have a Dev environment that we test the changes in first.

Clients losing revenue because you broke their application mid-week is important because contributes to your attrition rate. Which, if it's bad enough, will end your company.


I don't know, it feels like you're arguing based on your knowledge about the system you work on, without taking into account that other systems does not work the same. It's just nonsense. Good luck.

Scrum originally recommended people run sprints from Wednesday to Wednesday so you were doing deployments in the middle of the week. I’ve never worked anywhere that did a Scrum right, and I don’t mean that in the “well then you’re doing it wrong” sense. I used Kanban before Scrum and man is that a downgrade.

Thursday is a good choice, especially if Fridays are less busy than other days as that gives the team a day to work on the issue if necessary and minimises the business impact if it goes bang.

It’s ok, claude can do it for you.

You can, but it also means you might have to.

Not a family man, I see?

That has nothing to do with it. I don't like working weekends anymore that anyone else. However, every now and then it is necessary for certain systems. Also, if your rollback procedure is go, it isn't issue.

  >I don't like working weekend [sic] anymore [sic] that [sic] anyone else.
So why the feigned confusion above? It seems you already knew the answer to your own 'question.' :P

My confusion is people acting like it is blasphemous to deploy on a Friday when I have laid at multiple reasons why it may be necessary.

We have systems who’s main use in at the weekend. Other systems are only used overnight. Some the quietest part is 0930-1230 on weekdays.

The reason you don’t want to push on a Friday, or overnight, or weekend, is your experts and the top bosses tend to work m-f 9-5, so pushing early in the day and early in the week ensures more people are available to fix problems and run comms.


There is someone on call at all times at every level in my company and we are not very big. It's rare I see anything I can't fix, but if I do, it's easy to click that button that makes the alarms go off and they'll be on the call in minutes.

Is the guy on call your best troubleshooter? No. That guy doesn’t work weird hours because you’d have to be paying him $50k more to put up with that BS.

It’s easy to think things aren’t going to go to hell, but look at the huge companies that screw things up. Amazon. GitHub. It could happen to you, too.


Actually, we ALL work on call. No one gets out of it just because they have seniority. Plus, we have an excellent team of smart people who are able to figure most things out on their own through use of their own knowledge and troubleshooting skills, aided by both internal and external documentation. It is rare that escalation is required.

I never said things don't go to hell sometimes. That's why you build a good team and train your people.


I think it's not just inconvenience for users but also availability of engineers to look at failures.

Many organizations do have round the clock staff but maybe they have the most staff during 9-5 on a particular time zone. So end of day Friday is guaranteeing you'll get a different shift of worker diagnosing failures, i.e. not the author of the deployed change, and possibly fewer available people.


So... it's 24/5, then.

Yes pendant

I suspect this is a bait, but I'll bite. :) Pendant is a type of jewellery. I think you meant pedant.

It happens. My company has done a couple dozen production rollouts today, including minutes ago.

In public utility we only did deployments on Fridays after 5PM

There's a bifurcation in best practices.

For industries that serve customer bases that are down-cycled on weekends, the dominant effect is "Deploy on Friday to minimize customer disruption chance."

For industries that operate with a 24/7 uptime expectation, the dominant effect is "do not deploy on Friday so you don't have to pay for your expensive software engineers pulling extra hours on a weekend. Much better to have disruption while the folks most qualified to fix it are in-house and alert."


Team America: World Police

Uh I'd imagine any team that mainly has their user base using the product on weekdays would deploy Friday evening so if something goes wrong they have the weekend to fix it.

Only if the team wants to work weekends?

If something goes down on Saturday there’s a good chance I’m out of cell service


It is interesting to see this bifurcation in action.

Hey software engineers - you know that working weekends is unusual, right? Like... please tell me someone has at least negotiated your right to only work 5 days a week so if you're working a weekend you get Monday and Tuesday off.

If not... Has anyone told y'all you should unionize?


Many people work shifts, it’s quite normal.

What seems to be unique in American companies is you’re expected to work shifts or nights but also work m-f 9-5 at the same time, which is really weird.

Same with of call, if you get called at 3am on an on/call rota for a specialised third line escalation many companies still expect you to work the next day, and they might not even pay you for being on call?!


I mean, you make it sound like releases fail often. For our team I think we have only seen one failure in the last two years needing a fix after hours (going into the weekend)?

The other option is to fail during the workweek. If the software is directly generating money for the users and the developers are judged on what actual business value they provide then failure during business hours is worse.


Did you know that when you use your 2 days off to go shopping, someone has to work at the shops?

Yes. That's why people work staggered schedules. Folks who work weekends at shops usually (and should) have two other days off in the week.

People literally fought, bled, and died for our right to not do capitalism 24/7/365, because with no back-pressure that's what the efficiency engine will demand of us.


And part of that is software engineers doing deployments after-hours for their customers. If your company or country is any good, they'll give you double time off during normal hours.

Such a smug privileged answer given in a world where the industry is shrinking its workforce and pretty much all other industries have flatlined for years(meaning we are still better off despite this realities).

Wanting to have a weekend is smug and privileged now?

People fought hard for the 40 hour work week and 2 day weekend.

You think that should be given up?


Oh, I know all about that history. Having been a volunteer for Bernie Sanders back in 2015, I admired him as being one of the remaining few people who actually remembered this history due to him being part of the Silent Generation and continue to fight for it decades later.

>You think that should be given up?

Not really what I said. My point was that in an era of transition where we started with a 40 hour workweek and a two day weekend, the positive outcome could be a 30 hour workweek and a three day weekend or more. That transition period from one to the other is extremely precarious for everybody and is what we are in currently. It isn't going to transition until a great deal of people make it so, and they only do that when the pain reaches home for them. In the interim, your advice is counterproductive. My take away from this past election: the overwhelming majority are not going to change until it gets real for them. Your advice is to light yourself on fire in the meantime.


I really don’t understand how you got that from my post.

I mentioned the fact that I’m often out of cell service on weekends (I like to hike and camp) and so it’s a bad time for an outage.

Are you saying labor conditions need to get worse for everyone in order to get better?

If so I disagree and think we should be advocating for better labor conditions for everyone while protecting things like weekends


I use 24-hour time personally but to be honest the only thing it's good for is disambiguating 12 midnight from 12 noon.

Tbf that's a huge benefit.

And it's not like 12-hour time has more benefits or fewer downsides than 24-hour time.


I find that other people get annoyed or at least bring attention to it when I try to use 24-hour time with them, so that's a downside I guess.

While that is true that is less an issue with 24-hour time and more with humans' tendency for the Familiarity Bias.

I guarantee that if most people grew up using 24-hour time, they would find it just as weird to hear someone use 12-hour.


I'm curious what gadgets did you grow up with, and what age did your children get the NES? Are you raising them to have a "lower tech" childhood than yours?

well, the 60's and 70's were not very gadgety. FM stereo? then cassettes...

Oh wait but then the came the 8085, 80186, 68000...you see where I'm going...

We got them the NES I think it was Christmas in 1989. Lower tech...that was a long time ago. They are both savvy now but one is a pastor and not techbound, the other is in a Tech based company but not a deeptech pro

I retired from so many different things that were getting digitized. it was/is a great time to be an engineer.


Oh, sorry. I was assuming your kids are still young. Lol.

Did you just make an acronym that means "before AI"?


I did, but I'm not the first. I thought maybe I was, so I checked: apparently it's been bouncing around informally among academics for a while now. I hadn't previously seen it though, so I can at least say I came up with it independently.

I come up with things from time to time. At least one US registered trademark is mine.


Future history books will be like: ancient legacy technology conglomerate Google was founded around 24BAI


I'd guess us throwing around the word "AI" constantly and endlessly debating p(doom) based on current tech will be considered pretty amusing in the future.

Like reading essays from the late 19th century that can't stop pontificating about living in a boundless technological singularity thanks to railroads and the telegraph. Current hackocalypse discourse already has shades of "the bomber always gets through" from the 20s.

"Sure, people in every era since the Renaissance have foolishly believed they were the ones living on the cusp of uniquely transformative and potentially cataclysmic technological change but ... This Time It's Different"


I can imagine someone saying the same in the 60s: don't sign this nuclear test ban treaty, we have induction from 1300s that.. wait let's chose a time just after the Black Death. We have induction from the Renaissance that proves we always worry too much about these things and therefore nothing is different about this time.


pretty sure "before wwiii" will take precedence


Or something like 32BAGI


No, that is already solved by gpg.

Stop trying to make blockchain happen, it's not going to happen.


Linking to a page that links to the source is wild.


> Linking to a page that links to the source is wild.

Sorry, what I meant was, the source has been posted earlier on Hackernews which generated a bit of discussion


Do you do something special for these ewes, like give them more food or something?


We move them to another paddock, so the ram can focus his efforts on the remaining untupped ewes.


In my experience no, they’re mostly all pastured.


Isn't that just the turing test? I suppose you have you answer now that models can easily pass it.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: