How to better estimate your tasks

First of all, an estimate is just that - an estimate. Most of the time engineers struggle with estimation because of maybe some unforeseen side effects, or because of underestimating the task itself, or a plethora of other reasons.

Secondly, as an engineer you have to keep in mind that your primary goal is not to estimate the task itself, but to get a clear idea of what you will be implementing. The more details you know about what you are going to implement, the better an idea you will have about how much time it might take. Here are some steps that will help you achieve that:

1. Ensure a central list of requirements, e.g. acceptance criteria is in place. if you can identify any edge cases within the requirements immediately, add them to the list. Solid requirements will help you build a solid feature. It is also up to the engineer to analyse the requirements and add to them, because the product manager might not come up with all the various cases that need to be handled. This will also help QA and engineers devise the test cases for unit tests, manual tests and integration testing.

2. Once you have all your requirements in one place, start thinking about the data flow. What is the trigger point? What is the initial source of data? What kind of data will you need to store, and where (in db or cache)? How will the client eventually get access to this data? What kind of processing do you want to do to this data so that it meets the given requirements? Thinking about the data flow from start to finish, and writing down the steps will help you arrive at the overall design.

3. Schema changes - now that you know what the data flow will be, you can use that to write down the schema level changes e.g DDL statements, any new or alter table commands, adding indexes, partitions, etc.

4. API Design - from the data flow, and schema, you will get to know what APIs need to be implemented or updated. Think step by step what APIs you need to achieve your requirements. Are you creating new resources? updating existing ones? or fetching the new data to show somewhere?

5. Integration points - think about where will this new feature be integrated, whether your APIs will be consumed within the same module, or by another backend, or by a frontend UI. Think about the challenges the integrations might face, for example how will authentication/authorization work, does it need pagination, does your module need to be idempotent, what will happen if any external system makes concurrent requests with the same data, etc.

6. Impact analysis - how will your change impact the rest of the system? Chances are you are changing an existing system (or extending it). What will be the side effects? Do you need to emit any events to let other modules know what you are doing? Will your changes break any other module flows? Here analysing the existing system is important to understand how your changes will fit in, and how you can ensure continuity to the existing flows.

7. Task breakdown - Finally, now you should have a complete idea of what to do in order to implement this feature. If you did step 2 right, then this should be easier now. Using all of the analysis from above, start listing your todos / subtasks one by one in order in terms of what is needed to be done.

8. Finally cross check your todos/subtasks to make sure all the acceptance criteria are being met. If you find any further edge cases like "x flow should not break", or "y system should be notified", add those to your acceptance criteria. By the end of this exercise, you will have a well-documented process for how to implement the feature.

Now that you will have a good idea about what needs to be done, and will have a granular level breakdown of the tasks/subtasks, you can ballpark each subtask (should each be within 1-4 hours), then add up the hours to get a final estimate. Don't forget to account some time for writing unit tests / integration tests, and then also some buffer for some unforeseen circumstances (it happens). By now you will have a more reasonable estimate of how long the whole thing should take.

Key words are "more reasonable". However, a fair warning - you still might not be able to complete the task in the time you thought. It happens to everyone, welcome to software development :)