Lessons learned while working on large-scale server software

#todayilearned #career #security

TL;DR notes from articles I read today.

Always have a plan for worst-case scenarios for error conditions, and find a general solution, such as automatically shutting down all operations and return an error code with when to retry or a contact to call.
Document your decision making and add idempotence wherever possible.
Approach debugging in a scientific way: first, gather data, form the right hypothesis and design an experiment to prove it, and then apply your fix; use tools to dig deep into traces and memory without stopping the system.
Impose a strict implementation of Postel’s Law: Be conservative in what you send, be liberal in what you accept.
Be wary of a major deployment that seems to go smoothly. Errors are inevitable and the bigger and quieter the error, the more dangerous they are. If you are not sure how to handle an error, let the system crash. It makes it easy to catch and correct errors.
Be prepared to restart the entire system from a blank slate under heavy load.
Notice technical decisions and components that have global effects, not just global variables.
Build channels for persistent communication as new people are onboarded and leave teams. When building systems, do not assume operators will do things correctly and give them the tools to undo mistakes.

Full post here, 10 mins read

Be ready to listen. Your mentee should always feel free to ask you questions, especially in the first few weeks of onboarding.
Help anchor them socially in the new place. If possible, carve out a weekly team ritual.
Let them feel safe enough to pursue novel solutions by showing them how to evaluate risks within the context of your organization and team.
Explicitly inform your mentee of technical safeguards in place as well as providing non-technical assurance of help with failures.
Understand their strengths and weaknesses, and encourage as well as challenge them technically with tasks that let them level up while having fun, gradually adding more unfamiliar or challenging jobs. Offer code hints but don’t unnecessarily prime them with warnings of difficulties ahead, and occasionally add stretch goals and praise their resolution to build confidence.
Have 1-on-1 discussions and reinforce when you notice they are doing a great job while challenging them to do even better and share business context they might not readily see, as well as the engineering context of their project predating their on-boarding.

Full post here, 8 mins read

DEV Community