DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

Cover image for I'm Charity Majors, Ask Me Anything! [FINISHED]
Charity Majors
Charity Majors

Posted on

I'm Charity Majors, Ask Me Anything! [FINISHED]

I am the cofounder and accidental CEO of honeycomb.io, where we are thinking hard about how to help you debug and understand the complex systems you have now && the even crazier systems you're going to have soon. I think a lot about observability and how to help software engineers own the code they write without losing their quality of life. Before this I was an engineering manager at Facebook, built systems at Parse and Second Life, and spent most of my time worrying about databases. I miss being on call.

Top comments (58)

Collapse
 
annarankin profile image
Anna Rankin

Hello Charity - thanks so much for doing this! I'm interested in how you got started - did you come from a Computer Science background? How'd you land your first job? (I'm always interested in origin stories πŸ˜‚)

Also, any advice for programmers who are thinking about a future in management or leadership?

Thanks again! You're awesome :D

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

Hell no, I am a music major dropout (with side gigs in classical latin and greek, philosophy, etc). I never had computers growing up. I ran away to college when I was 15 though, and developed a crush on a boy who spent his time in the computer lab.

The lab stuck; the boy didn't.

Advice for those wanting to move to management: question yourself about why you want to do people management. Lots of people want it for the wrong reasons. And it's very different than wanting a future in leadership, which every single senior eng can and should have.

There's no scarcity of leadership work. I think it's terribly toxic to confuse management with leadership though.

Collapse
 
annarankin profile image
Anna Rankin

Thank you - that's a very good point! Also high five for art backgrounds πŸ™Œ I love that. Thanks again for doing this AMA!

Thread Thread
 
mipsytipsy profile image
Charity Majors Ask Me Anything

Word. I think the liberal arts are the best education that 90% of humanity can possibly get. All hail philosophy majors. πŸ‘

Collapse
 
pinkpushpop profile image
Katrina

What is an area that new software engineers often overlook when writing code?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

What happens to it after you deploy it.

It's not enough to compile and pass tests and not get paged about it. You should have muscle memories for going to check up on it -- did what you expected to happen, actually happen? Did anything else change?

Collapse
 
jess profile image
Jess Lee

What were some of the unknown unknowns you've had to deal with as a CEO?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

oh jesus. much like observability, being ceo is all about the unknown-unknowns. I always feel like I'm failing, because I'm only ever working on whatever is broken in the org ... as soon as it works, it's someone else's job.

But the biggest unknown-unknown was having to be CEO at all. I distinctly did not plan on and did not want the job. I wanted to be CTO. It's always been my intention to have a primarily technical role for the rest of my career. And I knew CEO was a shit job and that I did not want it. But shit happens.

Collapse
 
jess profile image
Jess Lee

Ha! Is there anything you like about being CEO?

I read your interview with Business Insider where you mentioned how smooth the fundraising process went for honeycomb. What's your advice for folks that aren't quite as connected but are trying to raise?

Thread Thread
 
mipsytipsy profile image
Charity Majors Ask Me Anything

I believe in what we're building, like a lot. That makes it worth it to me. There is very little about the job that I find rewarding or interesting on a day-by-day basis, and I'm still petrified about being relegated to PM roles after this -- what if nobody will ever hire me as an engineer again??? My 3 am self is in agony over this.

It was smooth this time because I have a COO who has been a VC and done startups before. It was not that smooth when I was trying to do this on my own. Moral of the story: admire business people and don't talk down to them, so the good ones flock to you. I think that's the moral of the story anyway.

My advice to anyone with fundraising questions is always, "talk to Ginsu". 😡 Dear god what a lifesaver good business people are.

Beyond that ... think big, pitch big, tell a story. I have to try very hard not to use my normal sardonic self-deprecating sense of humor around investors. Things that I think are fucking hilarious and realistic, they read as lack of ambition. Sigh.

Collapse
 
cooper_kunz profile image
Cooper Kunz

What are your biggest goals for honeycomb's next 1-2 years?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

First: nail user experience. We've built a service that's so fucking powerful that a few people will crawl over rusty nails and shards of glass to use it. Now it's time to circle back and make it more broadly available.

Instead of painstakingly instrumenting everything, we are building onramps to particular communities, so you can e.g. do "npm install honeycomb" and get all the basic stuff for free. I think this will help close the gap.

Second, I really deeply want to make honeycomb a mecca for distributed systems art and visualization. We need more visual metaphors for teasing out signal from noise, and more cutting edge art drawing on recent academic advances in data vis.

People are getting hit by this freight train of infra complexity, and I don't think A.I. is gonna save us in the next decade or so. What will save your ass is good design paired with intelligently plumbing your network. Turn it into a social graph question: when you get paged, what do you want to see? Obviously you want to see whatever questions were asked by the last person who got paged about this problem.

We have to look for ways to bring everyone up to the level of your best debugger, in every subject area. We have to get better at empowering teams, not individuals. Hell, I learned Linux by reading other people's .bash_history files. I want Honeycomb to feel like it's that embedded in your team and your social circle.

Third, I want to help software engineers demystify production. I want to take the CI/CD revolution another big step forward, and make it table stakes for every engineer to explore the consequences of their code in production, every time they deploy.

I think production should not just not be on the other side of a wall, but it should feel like the "fourth trimester" -- when you ship your code to prod, it's a blind, wailing, helpless, probably broken piece of shit when it first encounters real users, real data, real services, real network. You have to nurse it into growing up.

Software needs owners, not operators. Everything I'm consumed with is just details towards that goal.

Collapse
 
ben profile image
Ben Halpern

Always count on a startup founder to have lots to say about these sorts of things πŸ™‚

Thread Thread
 
mipsytipsy profile image
Charity Majors Ask Me Anything

... i can keep going ... πŸ˜‡

Collapse
 
cooper_kunz profile image
Cooper Kunz • Edited on

Damn, thanks for such a great response.

Do you have any articles you've written or talks where you've discussed something similar to the "trimesters" of development? I love that insight, and the analogy of needing to nurse your systems/tech to maturity.

And thank you for doing an AMA!

Thread Thread
 
mipsytipsy profile image
Charity Majors Ask Me Anything

I have a draft. I hope to finish it this week. :)

Thanks for having me!

Collapse
 
mattstratton profile image
Matt Stratton • Edited on

What can an organization do to be more inclusive of their DBA’s into their devops initiatives?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

Use the same tools! The edges of tool usage is what creates silos.

Also, be willing to learn things like query optimization, etc so you don't have a DBA SPOF. People seem remarkably eager to wave a hand at it as tho it were black magic. IT's not. It's easy.

Collapse
 
mattstratton profile image
Matt Stratton

Which is more important - observability of systems, or bourbon?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

Trick question: it's a false dichotomy. Everybody knows ops teams run on malt whiskey.

Collapse
 
vijayasankarv profile image
Vijay Vijayasankar

Do you think technologists make better CEOs than someone coming up through say marketing or finance ?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

No, not necessarily. I've more often seen them be worse, because engineers tend to underestimate how important and/or challenging business stuff can be ("you just make the best product, and that will always win, right??"). Engineers have often underinvested in developing their powers of persuasion and other social skills as well.

But being a good CEO is all about surrounding yourself with people who are better than you, and knowing when and how to interfere.

Collapse
 
andy profile image
Andy Zhao (he/him)

For me, being on call is sort of nice and fun, but I think it's hard to make plans and sometimes affects my work-life balance. What do you miss about being on call?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

A guilty little secret of mine is that I love firefighting. Some people jump out of airplanes ... I love the panic and glee of knowing everything is on fire, the company might not survive if I don't fix it correctly. I love high pressure and high stakes.

... But I will normally deny this (if sober), because in modern day infrastructure you are supposed to hate firefighting and want to write software all day.

On call sucks for a lot of people. It's a shame. I really believe it doesn't have to, it should be a net plus. Sounds like you're at least partway there. :)

Collapse
 
angaither profile image
Andrea Gaither

I kind of also love firefighting. What's one of the worst fires you've dealt with? What did you learn?

Thread Thread
 
mipsytipsy profile image
Charity Majors Ask Me Anything

Um.. the worst outage of my life was probably at Second Life around 7 years ago. We tried upgrading the primary from 4.1 to 5.0; all the secondaries had been upgraded painlessly, and all the benchmarks said 5.0 was faster. When we upgraded, the grid stumbled to recover; we ended up being mostly down for over 24 hours, and losing all that data when we had to roll back to the last good 4.1 secondary (due to binary incompatibility, couldn't roll back in place).

I spent a year developing capture/replay software for mysql and testing various configiurations and workloads before finally upgrading successfully.

And what did I learn? To be desperately paranoid of all database upgrades, and assume that anything that can go wrong, will go wrong.

Collapse
 
andy profile image
Andy Zhao (he/him)

Hah, totally feel you about loving firefighting. There's a real thrill of it for me, too, and it's really fun just being on the edge of the seat, where every solution you push out might be the one!!!

I might be partway there, not sure yet honestly. I can never reason out how it can be a net plus though, since (at least in my mind) it's a net-plus only if something broke and it was solved, right?

Thread Thread
 
mipsytipsy profile image
Charity Majors Ask Me Anything

Net plus? Because for a week you have justifiable cause to run down any rabbit hole, do extravagant perf tuning, and other shit that normally isn't important enough to preempt your project work. :)

Collapse
 
ben profile image
Ben Halpern

What are the biggest misconceptions about observability?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

That it's a synonym for monitoring.

It's not -- although there's a lot of overlapping domain knowledge, and you might say that monitoring is a subset of observability.

Monitoring is heavily biased towards alerting, downtime, outages, and above all actionable alerts. I think of monitoring as being to ops what tests are to developers -- once you know about a problem, you can monitor for it, you can test for it. Monitoring is about known-unknowns.

Observability is a property of systems, and it primarily is concerned with unknown-unknowns. A system is observable if you can ask any new question you want out of it-- if you can understand the insides by interrogating the outputs, without needing to add new customt instrumentation for each question.

Collapse
 
ben profile image
Ben Halpern

Can you suggest a talk or two to check out? By you or by someone else or both.

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

Oooohh boy. Do you like articles? Cindy Sridharan has been on fire lately, with her pieces on the subject. Honestly I had kind of stuck my neck out there, insisting this definition makes sense, and all the monitoring boys were coming down really hard on me saying it was bullshit. Cindy researched the topic, read a bunch of the stuff on both sides, and wrote some brilliant long pieces with even more detail than me where she came to basically the same conclusions. It was super validating. I think observability is an idea whose time has finally come.

As for talks ... I'm not aware of anyone else talking about this just yet, though I hope that changes soon. You might check out my Strangeloop 2017 talk. I was super specifically pitching it to software engineers, trying to meet them where they're at. Let me know if you like it or not, please!

Collapse
 
ben profile image
Ben Halpern

Thanks for the tips

Collapse
 
liana profile image
Liana Felt (she/her)

Hey Charity! What's the hardest part about dealing with the engineering stuff and the business stuff?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

I don't really deal with the engineering stuff, right now. I would only annoy people if I stuck my nose in with opinions. I don't really deal with business stuff so much either though ... hmm.

I guess I would describe my role as always plugging the weakest important part of the ship, and paying lots of attention to product and strategy.

Year one was about, "do we have a product? is it different enough?"; year two was about .. "okay, now do we have a business, and will people pay for it?" and year three is indubitably going to be about user experience. So I'm spending most of my time learning things about that.

Not sure if this answers your question or not, sorry.

Collapse
 
jess profile image
Jess Lee

Note quite the same as accidentally becoming CEO, but thought you might like this article about a woman who joined a COBOL bootcamp, became a programmer, and ultimately found herself promoted out of programming.

Collapse
 
jess profile image
Jess Lee

I studied classical piano in college too! I always have Arabesque No. 1 by Debussy stuck in my muscle memory -- first thing that comes out when I get to the bench! Do you have a go to piece?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

Niiiiice. :)

Rachmaninoff has always been my guilty pleasure. I spent two years learning the Revolutionary Etude when I was ~12, and I find myself compulsively fingering those long descending sequences when stressed. It's like a nervous tic.

Collapse
 
jess profile image
Jess Lee

YES!!! Everything c minor please. That one's so much fun but my fingers definitely aren't in shape for it anymore!

Thread Thread
 
mipsytipsy profile image
Charity Majors Ask Me Anything

it's actually really depressing to play ... what comes out of my fingers doesn't sound anything like i remember it sounding πŸ€”

Collapse
 
cooper_kunz profile image
Cooper Kunz

What's your favorite thing to do when you're not working, that's non-tech related?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

For the last two years I've been walking 5-10 miles a day. I've always hated gyms and fitness in general, so it's nice tp find something that really works for me. It's practical, I can do it when I'm roaming strange cities, I can write emails and blog posts and do twitter and read ... basically I look down at my phone, and when I look up an hour has passed and I'm there. It's amazing.

I also love, love, love fancy whisky, and bonding with a good friend while sipping late into the night. I <3 my scotch malt whisky association pass. The tasting notes read like a markov's chain. _^

Collapse
 
maestromac profile image
Mac Siri

How do you price a software product/service?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

Start by bleeding out your eyes and tearing out your hair for six months or so.

It's incredibly hard.

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

More seriously, ok, I guess I would say:

  • know who your customer is. you are either going to be leaving shit tons of money on the table, or shit tons of easy customers on the table. pick one.
  • know what it costs you (don't forget the personnel costs) and bend towards sustainable.
  • align your incentives. things that hurt your platform should be expensive, things that are easy should be cheap. engineering cycles are always the hardest part.
  • you can't get every customer. be willing to walk away for the right reasons.
  • i still don't know how to do this thing btw
Thread Thread
 
maestromac profile image
Mac Siri

Thank you for such a thorough answer! Boy this is tough πŸ˜₯.

Thread Thread
 
mipsytipsy profile image
Charity Majors Ask Me Anything

so tough <3

Collapse
 
andy profile image
Andy Zhao (he/him)

What was it like being homeschooled? I'm thinking about homeschooling my future kids. Also, sometimes I wonder how I would've turned out if I was homeschooled.

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

Terrible. I was deeply isolated. I didn't really have any friends or learn to interact with other humans for years after leaving home.

There was also the religious fundamentalism, though.. it may be hard for me to disentangle the two.

But seriously: kids need strangers and peers and friends. Family isn't enough. Beyond that, I do think it's a crime how many hours/day kids spend sitting in classrooms reviewing stupid shit, and all having to go at the same dogged pace. It's not a good use of anyone's time. It's like a warehouse more than an institution designed to inspire love of learning.

Collapse
 
andy profile image
Andy Zhao (he/him)

Oh dear, sorry for all that...

But seriously: kids need strangers and peers and friends. Family isn't enough. ... It's like a warehouse more than an institution designed to inspire love of learning.

100% agree. Thanks for the answer.

Collapse
 
andy profile image
Andy Zhao (he/him)

How'd honeycomb.io get it's name? I think it's pretty sweet.

...πŸ˜‰

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

The first name was bloodhound.sh.. we thought a detective dog would be cute, but it sounded vicious. So we shortened it to hound.sh. But then we got a legal challenge from hound-ci. So the four of us (myself, Christine, Ben, and Toshok) sat down with a lot of beers and tossed out all the ideas we could come up with one night.

The next morning we sat down with coffee and sifted through them for possible contenders. One was trufflepig.io, the other was honeycomb.io. But it was owned by Slack... :(

However, our lovely friends at Slack agreed to sell us the domain (and tossed in hny.co) for just the cost of the lawyer's fees! It was amazingly generous of them, and we are incredibly grateful.

I like honeycomb because of the associations with structured data, sweetness, hard work, and oh yes queen bees. <3

Collapse
 
ben profile image
Ben Halpern

One of the early name ideas for Amazon was relentless.com. Check out what happens when you click that link πŸ™‚

They too found it kind of vicious in the end.

Thread Thread
 
mipsytipsy profile image
Charity Majors Ask Me Anything

OMG. TIL!

Collapse
 
andy profile image
Andy Zhao (he/him)

Yes! All great reasons! That makes a whole lot of sense. That's great that it worked out.

Collapse
 
mtbsickrider profile image
Enrique Jose Padilla

No question. Just wanted to say i loved reading your responses and you sound like a hella of a fun character :)

Collapse
 
peter profile image
Peter Kim Frank

How is kubernetes evolving? Is it becoming easier to use?

Collapse
 
mipsytipsy profile image
Charity Majors Ask Me Anything

Yeeeeeeeessss... slowly and painfully. Much more slowly than I would have thought. I actually recommend most folks use a hosted solution, sigh.

🌚 Browsing with dark mode makes you a better developer.

It's a scientific fact.