My name is Sam Stagg and I work for Pusher as VP of Engineering. My responsibilities are to grow the engineering team, and to coach project teams and individuals to do the best work of their careers in a fast-paced and highly technical environment. My background is in software engineering in the video games and data industries.
Pusher provides hosted building blocks to help productive app developers move faster and write less code. Our core product enables developers to easily create interactive features for their applications, such as in-app notifications, activity streams, real-time dashboards, live trackers and more.
We just released Pusher Chatkit in public beta. It is a specialised Chat API solution that makes it easy for developers to add cloud-hosted messaging services to their mobile and web apps.
My AMA will start at 1PM EDT today. Ask me anything from scaling engineering teams to building real-time messaging applications!
Top comments (40)
What do you think is the best way to select tech. Leads? Where I work we are trying to do better at this. In the past, it was kind of up to supervisors, and they just asked people if they wanted to be a lead. Recently, we tried doing an internal job announcement and treated it kind of like a promotional opportunity, with in person interviews (even though it is not a promotion here). In the end, both stratgies have pros and cons, and the general consensus is that the interviews were the better option. I'm curious to know what you think, and what some of your experiences are in this area.
Great question. I think we are at a similar stage. The most important thing IMO is to make sure the possibility of being a lead is clearly open to everyone. We also like to run short projects that let people try on the tech lead hat without committing to being a team lead forever.
Internal job adverts sounds interesting - is that working out?
The internal job announcements were a mixed bag for selecting new tech. Leads for us. Everyone involved told me that they liked the process better than what we had done in the past. It seemed more fair, and it was easier for people to make their intentions clear (i.e. in the past, if a supervisor didn't know someone wanted to be a lead, they'd get passed over; now they could apply for it, because it was pretty clearly announced to all). The big problem for us, was that we had 4 positions, and 3 people applied, and none of them were selected. 2 of the applicants weren't really qualified, and the third was already a lead on a different project. We are about to go through this again, and I want to avoid the situation of not selecting anyone again.
I'm not sure if we can set it up to have tech leads hold the position for shorter durations, with the way we are currently structured, but it is something worth thinking about. I like the idea of people not being tied to being a lead long term, especially if they find they don't like that role.
How much of your team's efforts are devoted to features, developer experience, scaling infrastructure, and all the other things involved in writing code for a company like Pusher?
How do those responsible for scaling the infrastructure think about the end-user experience?
We actually spend most of our engineering capacity on innovation at the moment, which is great. The core Pusher product is scalable and maintainable without constant firefighting, and we want to keep it that way! So it's less than 10% on that, with most people working on our newer products like Chatkit.
All our engineers spend time on support so they get a good idea of how our customers use our product, and our sales team are really great at publicly highlighting customer use cases.
Hey Sam! I think most people switch from app dev to the game industry and usually not the other way around. What made you want to get out of the video game industry?
Nothing in particular. I had a great opportunity to move to London, and I liked the idea of working in an industry that wasn't quite so hit driven and revenue was a bit more predictable.
Ah, makes sense. What's the London tech scene like compared to where you were originally from (guessing the States)?
Here's an interesting article about the Ruby scene in Europe vs US that I read a year ago:
The European Ruby Revolution
Actually I'm from the rural Midlands of the UK, which has a lot more games companies than people realise 😄
Interesting article. Ruby is still very big for us and our customers.
What's the Pusher stack and why were the various technologies chosen?
The core of the main Pusher platform is written in Ruby using the EventMachine framework. We also use Redis extensively for the internal messaging part. We picked this in 2010 because we spun out of a Ruby on Rails agency, and we never had capacity to change it 😁
In the last year we've been building a new platform built in Go to support our new products, which we're finding much more suited to building scalable distributed systems. We use Kubernetes for infrastructure management and React on our front-end.
I'm surprised Pusher is written in Ruby. What are Pusher's main performance bottlenecks and how do you deal with them?
It is quite surprising, but EventMachine is actually pretty good for letting us handle a large connection load on a single threaded Ruby process.
We try to handle all performance bottlenecks with horizontally scalability. And Redis is awesome for handling the internal heavy lifting.
Thanks, I'm definitely going to check out EventMachine.
I have to say (as a lead developer at Pusher) that I wouldn't recommend EM for new projects. It was a leader in the evented-code-in-dynamic-languages revolution and inspiration for node.js, but it did not take off in the same way. Nowadays it feels quite archaic and unloved. The abstractions have not kept up with the other ecosystems in the space.
If I were to start a concurrent ruby project today, I'd try Celluloid (github.com/celluloid/celluloid), but as Sam mentions, we've mostly moved on to Golang for high-performance work.
What would you say is the difference in role between a VP of Engineering and a CTO?
Great question. Luckily for me there's already a great answer that Greg Brockman wrote in 2014: blog.gregbrockman.com/figuring-out.... In fact in was this post that made me realise I was headed on this track in my career.
I can't add much to it TBH, but basically my role is to grow the team and keep it running smoothly. I don't have a major input on technical decisions but I need to make sure they are made when they are needed, especially as we don't actually have a CTO at Pusher!
How should the hiring process for technical positions change (if at all) as the overall team grows?
What are the key lessons to keep in mind as an engineering team grows from 1-5 to 10-50 from a hiring perspective?
The main thing is that you can't rely on the "star interviewer" who can interview everyone up to 10 people. And then you have to think about how to build an interview process that more people can participate in while keeping results comparative.
We also found that above 10 people you start to hire more specialists (or at least specialising generalists). So we had to tweak our interviews to work for these new roles (e.g. mobile experts).
Finally, an internal talent person is absolutely amazing above 10 people (props to Tom, our talent hero).
Not really. I'm informed about architecture reviews, and it's ultimately one of my responsibilities to make sure they happen, but I don't do it myself. We try to make architecture reviews one of the competencies regularly practiced by senior engineers and above.
Do you do any coding these days?
If yes, how often?
If no, do you miss it?
Yes, a lot less but I still love it. It's mainly personal stuff now. I'm not sure the engineers would let me near the Pusher codebase 😉
ha! What sort of personal stuff are you working on? Exploring any new technologies?
Do you have 1:1 check-ins with your team? If so, do you follow a certain format?
Yes, I do. I try to keep them regular although the frequency has dropped to around every 6 weeks for most people as we've grown our team (27 now).
We now have a few mentors throughout the team - tech leads and lead engineers - who hold more frequent 1:1s with their team. I now focus almost exclusively on career development, personal growth and making sure people are challenged by the work they're doing.
Does game developers write tests for their games? If so, is it similar to web development where there's a unit test and feature test?
Yes, for sure. I also suspect automated testing in games development has moved on a lot since my days of the PS2 and the original Xbox 😁
Is there any advice that you stood by when you were in the game industry that you still find relevant in your current day to day experiences?
Your best advocates are your players 😃
As VP of engineering, are you involved in project direction and goals as well or are more of the coach and guide for the engineering team?
Terrific question. I'm definitely involved, but as with many things my primary responsibility is to make sure these kinds of things actually happen. It's surprising how easy it is to kick off a project without goals or direction so I suppose I'm auditing that more than actually contributing.
Coaching and guiding are a big part of the role in my experience.
What does VP mean? What responsibilites do you have as a VP?
Do you have any opening for young engineers?
where can i software engineering on the web?
Have you found that Pusher's core product works equally well with a seasoned developer as it does with someone who has less experience? What are the learning curves, if any?
It works equally well. We strive to make our APIs and SDKs a delight for any level of developer. And the scalability of Pusher means even the most experienced engineers get a lot of value from the product.