I am the cofounder and accidental CEO of honeycomb.io, where we are thinking hard about how to help you debug and understand the complex systems you have now && the even crazier systems you're going to have soon. I think a lot about observability and how to help software engineers own the code they write without losing their quality of life. Before this I was an engineering manager at Facebook, built systems at Parse and Second Life, and spent most of my time worrying about databases. I miss being on call.
![Cover image for I'm Charity Majors, Ask Me Anything! [FINISHED]](https://media2.dev.to/dynamic/image/width=1000,height=420,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fthepracticaldev.s3.amazonaws.com%2Fi%2Fa14csi1piwptc9ou45a3.jpg)
For further actions, you may consider blocking this person and/or reporting abuse
Top comments (58)
Hello Charity - thanks so much for doing this! I'm interested in how you got started - did you come from a Computer Science background? How'd you land your first job? (I'm always interested in origin stories 😂)
Also, any advice for programmers who are thinking about a future in management or leadership?
Thanks again! You're awesome :D
Hell no, I am a music major dropout (with side gigs in classical latin and greek, philosophy, etc). I never had computers growing up. I ran away to college when I was 15 though, and developed a crush on a boy who spent his time in the computer lab.
The lab stuck; the boy didn't.
Advice for those wanting to move to management: question yourself about why you want to do people management. Lots of people want it for the wrong reasons. And it's very different than wanting a future in leadership, which every single senior eng can and should have.
There's no scarcity of leadership work. I think it's terribly toxic to confuse management with leadership though.
Thank you - that's a very good point! Also high five for art backgrounds 🙌 I love that. Thanks again for doing this AMA!
Word. I think the liberal arts are the best education that 90% of humanity can possibly get. All hail philosophy majors. 👏
What is an area that new software engineers often overlook when writing code?
What happens to it after you deploy it.
It's not enough to compile and pass tests and not get paged about it. You should have muscle memories for going to check up on it -- did what you expected to happen, actually happen? Did anything else change?
What were some of the unknown unknowns you've had to deal with as a CEO?
oh jesus. much like observability, being ceo is all about the unknown-unknowns. I always feel like I'm failing, because I'm only ever working on whatever is broken in the org ... as soon as it works, it's someone else's job.
But the biggest unknown-unknown was having to be CEO at all. I distinctly did not plan on and did not want the job. I wanted to be CTO. It's always been my intention to have a primarily technical role for the rest of my career. And I knew CEO was a shit job and that I did not want it. But shit happens.
Ha! Is there anything you like about being CEO?
I read your interview with Business Insider where you mentioned how smooth the fundraising process went for honeycomb. What's your advice for folks that aren't quite as connected but are trying to raise?
I believe in what we're building, like a lot. That makes it worth it to me. There is very little about the job that I find rewarding or interesting on a day-by-day basis, and I'm still petrified about being relegated to PM roles after this -- what if nobody will ever hire me as an engineer again??? My 3 am self is in agony over this.
It was smooth this time because I have a COO who has been a VC and done startups before. It was not that smooth when I was trying to do this on my own. Moral of the story: admire business people and don't talk down to them, so the good ones flock to you. I think that's the moral of the story anyway.
My advice to anyone with fundraising questions is always, "talk to Ginsu". 😵 Dear god what a lifesaver good business people are.
Beyond that ... think big, pitch big, tell a story. I have to try very hard not to use my normal sardonic self-deprecating sense of humor around investors. Things that I think are fucking hilarious and realistic, they read as lack of ambition. Sigh.
What are your biggest goals for honeycomb's next 1-2 years?
First: nail user experience. We've built a service that's so fucking powerful that a few people will crawl over rusty nails and shards of glass to use it. Now it's time to circle back and make it more broadly available.
Instead of painstakingly instrumenting everything, we are building onramps to particular communities, so you can e.g. do "npm install honeycomb" and get all the basic stuff for free. I think this will help close the gap.
Second, I really deeply want to make honeycomb a mecca for distributed systems art and visualization. We need more visual metaphors for teasing out signal from noise, and more cutting edge art drawing on recent academic advances in data vis.
People are getting hit by this freight train of infra complexity, and I don't think A.I. is gonna save us in the next decade or so. What will save your ass is good design paired with intelligently plumbing your network. Turn it into a social graph question: when you get paged, what do you want to see? Obviously you want to see whatever questions were asked by the last person who got paged about this problem.
We have to look for ways to bring everyone up to the level of your best debugger, in every subject area. We have to get better at empowering teams, not individuals. Hell, I learned Linux by reading other people's .bash_history files. I want Honeycomb to feel like it's that embedded in your team and your social circle.
Third, I want to help software engineers demystify production. I want to take the CI/CD revolution another big step forward, and make it table stakes for every engineer to explore the consequences of their code in production, every time they deploy.
I think production should not just not be on the other side of a wall, but it should feel like the "fourth trimester" -- when you ship your code to prod, it's a blind, wailing, helpless, probably broken piece of shit when it first encounters real users, real data, real services, real network. You have to nurse it into growing up.
Software needs owners, not operators. Everything I'm consumed with is just details towards that goal.
Always count on a startup founder to have lots to say about these sorts of things 🙂
... i can keep going ... 😇
Damn, thanks for such a great response.
Do you have any articles you've written or talks where you've discussed something similar to the "trimesters" of development? I love that insight, and the analogy of needing to nurse your systems/tech to maturity.
And thank you for doing an AMA!
I have a draft. I hope to finish it this week. :)
Thanks for having me!
What can an organization do to be more inclusive of their DBA’s into their devops initiatives?
Use the same tools! The edges of tool usage is what creates silos.
Also, be willing to learn things like query optimization, etc so you don't have a DBA SPOF. People seem remarkably eager to wave a hand at it as tho it were black magic. IT's not. It's easy.
Which is more important - observability of systems, or bourbon?
Trick question: it's a false dichotomy. Everybody knows ops teams run on malt whiskey.
Do you think technologists make better CEOs than someone coming up through say marketing or finance ?
No, not necessarily. I've more often seen them be worse, because engineers tend to underestimate how important and/or challenging business stuff can be ("you just make the best product, and that will always win, right??"). Engineers have often underinvested in developing their powers of persuasion and other social skills as well.
But being a good CEO is all about surrounding yourself with people who are better than you, and knowing when and how to interfere.
For me, being on call is sort of nice and fun, but I think it's hard to make plans and sometimes affects my work-life balance. What do you miss about being on call?
A guilty little secret of mine is that I love firefighting. Some people jump out of airplanes ... I love the panic and glee of knowing everything is on fire, the company might not survive if I don't fix it correctly. I love high pressure and high stakes.
... But I will normally deny this (if sober), because in modern day infrastructure you are supposed to hate firefighting and want to write software all day.
On call sucks for a lot of people. It's a shame. I really believe it doesn't have to, it should be a net plus. Sounds like you're at least partway there. :)
I kind of also love firefighting. What's one of the worst fires you've dealt with? What did you learn?
Um.. the worst outage of my life was probably at Second Life around 7 years ago. We tried upgrading the primary from 4.1 to 5.0; all the secondaries had been upgraded painlessly, and all the benchmarks said 5.0 was faster. When we upgraded, the grid stumbled to recover; we ended up being mostly down for over 24 hours, and losing all that data when we had to roll back to the last good 4.1 secondary (due to binary incompatibility, couldn't roll back in place).
I spent a year developing capture/replay software for mysql and testing various configiurations and workloads before finally upgrading successfully.
And what did I learn? To be desperately paranoid of all database upgrades, and assume that anything that can go wrong, will go wrong.
Hah, totally feel you about loving firefighting. There's a real thrill of it for me, too, and it's really fun just being on the edge of the seat, where every solution you push out might be the one!!!
I might be partway there, not sure yet honestly. I can never reason out how it can be a net plus though, since (at least in my mind) it's a net-plus only if something broke and it was solved, right?
Net plus? Because for a week you have justifiable cause to run down any rabbit hole, do extravagant perf tuning, and other shit that normally isn't important enough to preempt your project work. :)
What are the biggest misconceptions about observability?
That it's a synonym for monitoring.
It's not -- although there's a lot of overlapping domain knowledge, and you might say that monitoring is a subset of observability.
Monitoring is heavily biased towards alerting, downtime, outages, and above all actionable alerts. I think of monitoring as being to ops what tests are to developers -- once you know about a problem, you can monitor for it, you can test for it. Monitoring is about known-unknowns.
Observability is a property of systems, and it primarily is concerned with unknown-unknowns. A system is observable if you can ask any new question you want out of it-- if you can understand the insides by interrogating the outputs, without needing to add new customt instrumentation for each question.
Can you suggest a talk or two to check out? By you or by someone else or both.
Oooohh boy. Do you like articles? Cindy Sridharan has been on fire lately, with her pieces on the subject. Honestly I had kind of stuck my neck out there, insisting this definition makes sense, and all the monitoring boys were coming down really hard on me saying it was bullshit. Cindy researched the topic, read a bunch of the stuff on both sides, and wrote some brilliant long pieces with even more detail than me where she came to basically the same conclusions. It was super validating. I think observability is an idea whose time has finally come.
As for talks ... I'm not aware of anyone else talking about this just yet, though I hope that changes soon. You might check out my Strangeloop 2017 talk. I was super specifically pitching it to software engineers, trying to meet them where they're at. Let me know if you like it or not, please!
Thanks for the tips