Even if you release multiple times every day, refusing to release on Friday still makes sense. It's not about expecting bugs, it's about guaranteeing that your devs' time is their own. If you aren't okay with paying your devs for time they spend dealing with their own problems at home (without charging them their PTO time for it!) then you shouldn't be okay with making them work on weekends, no matter how rare it is.
The author ended up creating a strawman. Allen's argument was pretty clear: if your deltas are small and your deploy system is fully automated, then no one should be afraid of the risk of deploying.
Given that, if I deploy on a Monday morning and there is a bug on the new release, you revert, reproduce the issue in staging and push only new code when it is fixed. Same thing if I were deploying on a Thursday afternoon or a Friday at 7PM.
Only inexperienced developers* are unafraid of deploying right before leaving the office.
There's an entire untapped universe of possible new ways that things can go horribly wrong.
*Experienced developers who hate their boss and their colleagues, too, technically.
possible new ways
Name two, please.
XML
How is that not easily reversible?
It's not about how hard the problem is to reverse, it's about respecting the team enough not to call them on Saturday.
Again: if the changes are small enough and you have automated checks in place, they should not require manual intervention.
Plus, what happens if a deploy on Thursday has a bug which only is manifested on a Saturday?
Again: if the changes are small enough and you have automated checks in place, they should not require manual intervention.
You've used the magic word "should". "Should is famous last words." The trick to keeping developer talent is not to risk the developer's weekend plans on "should".
And yes, maybe I'm only risking our cloud ops person's weekend plans. Same principle applies.
Every change that isn't already an active disaster recovery can wait for Monday.
Every change that isn’t already an active disaster recovery can wait for Monday.
I honestly fail to see the difference between "don't deploy on Friday if this can wait until Monday" and "don't deploy on the evening if it can wait until the next morning".
The idea of CD is that changes are small and cheap. No one is saying "it's okay to push huge PRs with huge database migrations on a Friday", what is being said is "if your team is used to ship frequently and incrementally, it won't matter when you ship and your risk will always be small."
I honestly fail to see the difference between "don't deploy on Friday if this can wait until Monday" and "don't deploy on the evening if it can wait until the next morning".
Both are top tier practices.
If your team is used to ship frequently and incrementally, it won't matter when you ship and your risk will always be small."
Yep. That's all great advice.
But I'm just a veteran saying that all the preparation in the world doesn't compare with simply not inviting trouble right before the evening or the weekend.
Organizations that feel that they desperately need to take that risk, are doing it because they disrespect their team's time.
It can be the smallest risk in the world, but it's still a risk, and it's a completely unnecessary one (outside of an active in progress disaster recovery).
Organizations that feel that they desperately need to take that risk, are doing it because they disrespect their team’s time.
Or they are aware they are not in a position to block deployments for 60 hours every week? I've felt more discouraged working at companies that blocked Friday deployments because "it could wait until Monday", and then when Monday came half of the team was blocked or waiting for some new data report that could have been running during the weekend.
It can be the smallest risk in the world, but it’s still a risk.
And it's up to the Engineering manager (or at least the Release Manager in places where that role still exists) to evaluate what would be the trade-off. If you say that a bug coming from a Thursday deployment could've waited until Monday, why can't a bug that has come from a Friday deployment?
I guess my issue is not in saying "Some things should not be deployed on a Friday", but with the generalization. Of course there are things that should be okay to deploy on a Friday, or a Thursday night, or when the manager is on vacation... Being strict about it seems anything but "respect for the team", but a general distrust of the people and the process.
Good question.
Since we're doing a deep dive, I'll share some additonal context. I'm the manager of the developers. On my team, that means the call comes to me first.
I have had Thursday deploys that resulted in bugs discovered on Saturday. Here's how the conversation on Saturday went:
"Thanks for letting me know. So we didn't notice this on Friday?"
"No, it's subtle." Or "We noticed, but didn't get around to letting you know until now."
"Okay. I'll let the team know to plan to rollback at 0900 on Monday, then we will start fixing any damage that happened on Friday, and the weekend."
Can i work with you please? That sounds heavenly
In one of my jobs, they were automatically locking any push 2 hours before the shift ended.
Now that's a team that knows what they're doing!
If something goes wrong, devs have to give up their weekend to fix the issues, leading to burnout and dissatisfaction.
Correct in spirit, but the words "burnout" and "dissatisfaction" are weasle words that spinless middle managers use.
The correct terms are "abruptly quiting without notice leaving the company fucked and our stock worthless".
A minor point, but worth clarifying.
As someone who has experienced burnout before: that’s exactly what it looks like.
Even if you release multiple times every day, refusing to release on Friday still makes sense. It's not about expecting bugs, it's about guaranteeing that your devs' time is their own. If you aren't okay with paying your devs for time they spend dealing with their own problems at home (without charging them their PTO time for it!) then you shouldn't be okay with making them work on weekends, no matter how rare it is.
The author ended up creating a strawman. Allen's argument was pretty clear: if your deltas are small and your deploy system is fully automated, then no one should be afraid of the risk of deploying.
Given that, if I deploy on a Monday morning and there is a bug on the new release, you revert, reproduce the issue in staging and push only new code when it is fixed. Same thing if I were deploying on a Thursday afternoon or a Friday at 7PM.
Only inexperienced developers* are unafraid of deploying right before leaving the office.
There's an entire untapped universe of possible new ways that things can go horribly wrong.
*Experienced developers who hate their boss and their colleagues, too, technically.
Name two, please.
XML
How is that not easily reversible?
It's not about how hard the problem is to reverse, it's about respecting the team enough not to call them on Saturday.
Again: if the changes are small enough and you have automated checks in place, they should not require manual intervention.
Plus, what happens if a deploy on Thursday has a bug which only is manifested on a Saturday?
You've used the magic word "should". "Should is famous last words." The trick to keeping developer talent is not to risk the developer's weekend plans on "should".
And yes, maybe I'm only risking our cloud ops person's weekend plans. Same principle applies.
Every change that isn't already an active disaster recovery can wait for Monday.
I honestly fail to see the difference between "don't deploy on Friday if this can wait until Monday" and "don't deploy on the evening if it can wait until the next morning".
The idea of CD is that changes are small and cheap. No one is saying "it's okay to push huge PRs with huge database migrations on a Friday", what is being said is "if your team is used to ship frequently and incrementally, it won't matter when you ship and your risk will always be small."
Both are top tier practices.
Yep. That's all great advice.
But I'm just a veteran saying that all the preparation in the world doesn't compare with simply not inviting trouble right before the evening or the weekend.
Organizations that feel that they desperately need to take that risk, are doing it because they disrespect their team's time.
It can be the smallest risk in the world, but it's still a risk, and it's a completely unnecessary one (outside of an active in progress disaster recovery).
Or they are aware they are not in a position to block deployments for 60 hours every week? I've felt more discouraged working at companies that blocked Friday deployments because "it could wait until Monday", and then when Monday came half of the team was blocked or waiting for some new data report that could have been running during the weekend.
And it's up to the Engineering manager (or at least the Release Manager in places where that role still exists) to evaluate what would be the trade-off. If you say that a bug coming from a Thursday deployment could've waited until Monday, why can't a bug that has come from a Friday deployment?
I guess my issue is not in saying "Some things should not be deployed on a Friday", but with the generalization. Of course there are things that should be okay to deploy on a Friday, or a Thursday night, or when the manager is on vacation... Being strict about it seems anything but "respect for the team", but a general distrust of the people and the process.
Good question.
Since we're doing a deep dive, I'll share some additonal context. I'm the manager of the developers. On my team, that means the call comes to me first.
I have had Thursday deploys that resulted in bugs discovered on Saturday. Here's how the conversation on Saturday went:
"Thanks for letting me know. So we didn't notice this on Friday?"
"No, it's subtle." Or "We noticed, but didn't get around to letting you know until now."
"Okay. I'll let the team know to plan to rollback at 0900 on Monday, then we will start fixing any damage that happened on Friday, and the weekend."
Can i work with you please? That sounds heavenly
In one of my jobs, they were automatically locking any push 2 hours before the shift ended.
Now that's a team that knows what they're doing!
Correct in spirit, but the words "burnout" and "dissatisfaction" are weasle words that spinless middle managers use.
The correct terms are "abruptly quiting without notice leaving the company fucked and our stock worthless".
A minor point, but worth clarifying.
As someone who has experienced burnout before: that’s exactly what it looks like.