"Ship small! fast!" and "ship to learn!"
You've seen that shouted everywhere on Twitter. But what does it actually look like in practice?
I'm on ...
For further actions, you may consider blocking this person and/or reporting abuse
Great post! I have a question though that maybe wasn't addressed clearly. When you say 'ship small' I take it to also mean the ability to break the feature down into smaller logical parts and make small pull requests that can get reviewed faster and merged into production.
I constantly run into this situation where I sometimes receive/make massive pull requests because, well, all the code written was needed to make the feature functional (or makes logical sense together).
Is this a situation you encounter and if so how do you prevent it? Do you manage to keep all your PRs small enough to make the review process better for everyone?
Definitely! Happens all the time and I still do this occasionally myself.
I have a few ideas around this.
Also - sometimes making the PR small is more effort than it's worth. So sometimes it's necessary to have people review a large PR. I def try to avoid it though if possible.
Hey Mike, thanks for your great post.
What's small for you? Do we talk about a PR per day? Per 3 days? 1 Week? Or is it more about the amount of code?
Sometimes there is a lot of code to write for a very small thing.
Any guidelines on this?
Hey!
For me, it's generally a day or two of work for a PR. I like to think about the person who is going to be reviewing it. Will it be easy for them?
Using feature flags and being OK with shipping partially working features helps make everything smaller.
I think it's OK to ship an empty page. Or a page that has static fake data, that needs to be connected to a database later. As long as it's all behind a flag and users don't see it, it helps the development process move faster.
Thanks a lot,
I will keep that in mind!
Thanks for this, Mike! Really enjoyed this post. I have one question about the "parallel work streams" you mentioned.
What does the frontend work look like while the UI design is still being worked on? Is this like a "wiring up" of the initial boilerplate/wireframe that gets put up?
Thanks again!
❤️✨
Depends on how design heavy the feature is. Usually we have low fidelity sketches done during planning. Then we have our own design system, Primer, that we can copy/paste components from to build out the page. We can get really far with that alone.
This page for example: github.com/actions/starter-workflo...
I remember when we started it, we had some rough design ideas. We were confident we'd be showing "boxes" with workflows in them. So one engineer got started by passing an array of fake data to the frontend, and then looping over it to render a partial for each item in the array.
As this was happening, the designer was iterating on what the final page should look like. Once they had that, we were able to go in and adjust that partial/layout to look how the designer wanted.
My recommendation for people would be to always start on the piece you feel you have enough information to be confident in implementing (even if it looks terrible first pass). Just getting fake data on the page can sometimes be more work than we anticipate, so getting that out of the way and working is usually a nice win and saves time later.
Great article!
What about improvements to existing features? Do you also use feature flags the same way?
And regarding Pull requests, if you work incrementally doing very small changes, while each PR might be much easier to digest, there will be many of them per day.
Can you talk a bit about how is your code review process is organized so that the amount of PRs waiting for review doesn't become a bottleneck?
Thank you! ;)
Yes, we often do. It really depends on the risk and is a judgement call for the team.
A recent example, we added this "filter bar" to the Actions tab.
We feature flagged this one because it can generate complex queries and the risk is it could cause performance problems when used against huge data sets. We also wanted to be able to flag in specific people and have them test the functionality before rolling out to everyone.
Yup! A tip I learned from @andreasklinger a few years ago. When you work on a remote team you need everyone to dedicate time to doing code reviews. Unblocking each other is a priority, so most people will dedicate time in the AM (or whenever works for them) each day and they'll go through all the PRs and review them.
I find that when starting these projects, usually the first few PRs are a little more high effort to review because we're setting the foundation for the feature. But after that, things are small incremental improvements to the base. Since people are familiar with the area of code, they get pretty quick to review.
We rarely see anything go unreviewed for more than 24 hours. It helps that we are all on the same project and are motivated to get it done. If someone started a PR, but is now outside of their working ours, we'll often have someone else push commits to it to finish it up and ship it. Seeing multiple peoples commits on a single PR is pretty common.
One more thought on being "unblocking". We almost never block a PR unless we see something in the code that would cause a huge problem. We'll approve it, and leave "non-blocking notes" for things to improve. Then it's up to the submitter to take that feedback and apply it if they want. This reduces the amount of back and forth (critically important when in different timezones).
Thanks for the response. Perfectly clear ;)
Just one more question about feature flags.
How do you manage the cleanup process of older feature flags that are no longer needed, for example after the feature is generally available in production?
We're not great at that :). We often leave them in code for a couple months. Then we file issues to remove them and people will pick those up when they're looking for a quick/easy task to do.
Thanks for your sharing, it gives me hope that one day, I'll work in teams like that, maybe in 20 years :)
Got a question about production visibility, can you describe (or maybe you did talk about it elsewhere, or will) how works your "feature flags", because it's the main advantage you have to push code to production without giving access to "real" users.
Thanks ;)
:)
We use Flipper + some custom UI built internally. It works for Ruby apps.
If you're not using Ruby, I recommend looking at launchdarkly.com.
Great post and this is exactly how we are working at Adsdaq too. Since we introduced feature flags we not only can deliver/deploy fast but we completely eliminated our "long-living" branches which are a real pain.
There is one thing not covered here though which is very important to know when starting this direction and that is "backwards compatibility" ... Sometimes you need to change an existing feature which impacts something either in the api or the database. Since you are deploying and hiding with a feature flag, changes to those things must make sure the existing stuff keeps working! This can sometimes be very challenging. But not doing it immediately breaks the "deploy quickly and often" paradigm.
How are you guys managing your feature flags? We Open Sourced our Feature Flag platform Bullet Train last year and would love any feedback you have.
Do you think its better to roll your own feature flag system or rely on a third party?
Haven't heard of Bullet Train, looks nice!
We use Flipper, which is open source: github.com/jnunemaker/flipper. We have a custom UI on top of it for administration.
I'd always use something open source or paid. Building one doesn't seem like a good use of time when there are so many great OSS ones out there.
I found Unleash - Open source enterprise ready feature toggle service
unleash.github.io
Really good post! Thank you for your effort.
I have a question regarding feature flags in the backend. What if your new feature requires relatively big or significant database migrations. How does that impact the process? Do you maintain two data structures (old and new) till the feature is done?
That really depends on the migration + feature.
We can never take downtime at GitHub. So when making any DB change, it has to work with both the old and new code.
A recent example, we just had a feature that required us to move data to a new table. The process was like this.
Awesome Read!