This is an anonymous post sent in by a member who does not want their name disclosed. Please be thoughtful with your responses, as these are usually tough posts to write. Email sloan@dev.to if you'd like to leave an anonymous comment or if you want to ask your own anonymous question.
How do you manage big decisions during production?? How do they happen? How do you manage it? And how can you communicate any big changes to team mates? Looking for some helpful guidance!!
Top comments (7)
The question can have different points of view, are you talking about technically, having some issue in prod, or from the product iteration point of view? I'll try to cover both.
Decisions are linked to the environment. If you can push a fix quickly without restarting the service, the down time will be much smaller (mainly in scripting languages like Javascript, Python, PHP, Ruby or the deprecated ASP) while you need to define more steps on other cases, like push the code somewhere, build that, pick that build and deploy it to production environment (in case of Java, C# and so) and talking about a hotfix as example.
If there's a bug in PROD the decision is clear. Find the origin, patch it asap even not being the most elegant solution, push this fix into production and then, get some time to fix it properly (if the issue is more complicated and it's not done yet).
Processes that are not hotfixes are usually defined at the beginning of a project and any improvement that you feel useful can be raised on a meeting witht the team.
The best advice I can provide is to analyse the use-case with what you want to achieve, get some metrics, info and so and prepare a way to deliber that information (a document in google doc, a new entry in GitLab Wiki...? whatever) then explain it to them.
i.e.
Guys, I've been toying with [TOPIC] and I feel it useful so I took some time to analyse it. Let me explain it so we all have the same information before taking a decision: [Make a description]. Any questions on that? [Quick Q&A].
[CONTEXT] -> In our specific use-case I'd do [those steps] to reach this so we get [those benefits], wdy think?
Dont' get disgruntled if someone point out issues that you didn't mind or whatever, say thank you, pick this information, take some notes and see how it fits within the rest and, if it's still a nice to have, solve that issues in paper and come back with that. Again, is an iterative process and is always better to cover all possible points in analysis and design stage, before implementing.
If we talk about product iteration; That's having the product working in production and planning changes into it, you need to analyse what is done, why it was done like that and plan future features and changes with that knowledge in mind.
Also, gathering user data will definetely help to point you in the right direction. Analytics, heat maps etc... so if users are not using a feature or there are just few using it, try to find the reason, A/B test it, analyse it from the UX point of view and then you'll have a good amount of data to back up your decisions or to know what decisions need to be done to improve your product.
It can be a good idea to have a meeting with the team, point out those issues or show up the desired roadmap and stablish the milestones, setting what needs to be analysed as TBD and start working into it, you cannot cover everything at the same time because any change you do, can take effect somehow to the rest of the project.
Just to clarify, this data gathering, data analysis and decision making is a periodic and iterative process, a companion for the development process.
Avoid taking decisions by opinion, is best to take decisions by experience, and you need information to get experience and know what's going on behind the scenes.
Also, your preferences doesn't matter. You may like something but it could be against the product itself.
2 quick examples on that:
1- You can like the color green but you reach an expert in branding and he/she point you to a blue color for different reasons that fits better with the brand, the brand in the market and so. You can be a silly stubborn and insist in green or you can let the experts do their job and agree to the blue.
2- You like tailwind because you tried it and you jump into the community hype, didn't even knowing that it can't handle inheritance and what you did (to use @apply indiscriminately because otherwise the html gets shitty in minutes) is a bad practice. Still, motivated by the ignorance you push and insist to add it to the project without even analysing the pros and cons.
Those are examples of the recipe for failure
Hope I did cover the questions, otherwise sorry for the bible 😂
I made one big suggestion of changing code right before our go live. I never regreted it. The code was in not good state when I had joined the team and it was hot fix on hot fix on hot fix, ready to break every minute just to meet the deadline. But what then, when we are going to go into phase two with this code?
With some thougtful planning all hot fixes were rewritten into good and readable code, and even without meeting the dealine we could assure code quality for phase two! I do now feel comfortable when git blames me for the latest changes.
I feel like this is one example:
That Time When the Only Test and Development Environment Was Production
Jeremy Friesen for The DEV Team ・ Jul 3 ・ 4 min read
Underpinning that is the idea that while you are working in production you need to be very deliberate. Say what you are doing before you do it, so that others can hear it and affirm. (Or you can catch yourself).
Making big decisions in production should not even happen. If the feature is adequately validated with the team in dev/staging environments before going live into production, then it should be all good.
But still, we know that in real life, this happens, so in my experience, my team used to roll back to the previous version of the application. So we could gain more time to think about what happened and try to reproduce the error in lower environments (like dev/staging or pre-prod). And finally, define what steps we need to do to solve it (like, if it's a huge mistake with a feature, then we would review the small tasks related to it and resolve the code).
And again, we would test it in a lower environment to prepare it for production.
One of the concepts that are really important to know for the delivery process is DORA metrics. For example, you can measure the meantime to recover your application from a crash in production.
I invite you to look at this video if you want to learn more about it.
Just give it your best.
crashing the production does not end the world....
Guess you don't work on nuclear weapon launch control systems?
no sir, i certainly did not... :P