Page It to the Limit

Wake Up! With Corey Quinn

Mar 3 '20

Cloud Influencers and Life Before PagerDuty

Corey discusses how he became a cloud influencer, what that means for his credibility, and how people now listen to him.

Corey: “I have been a fan of PagerDuty for a long time, largely because I remember a time working on systems before PagerDuty existed.”

Corey discusses why waking the right people up at the right time makes it much less of a “Call of Duty” situation.

Business Hours and Business Outcomes

Corey wanted a problem that was strictly business hours only, Julie and Corey discuss how it all ties together with being on call.

Corey: “One of the interesting pieces that we see around all of those different business functions is that in most companies, engineering is usually the only group that needs to be awake at two in the morning that needs to deal with an emergent event.”

Corey goes on to explain that there are exceptions to this rule, and how businesses should staff accordingly if it is a 24 / 7 critical system, and how empowering folks to fix things that are broken is the start of cultural change.

The Cost

Corey: “Everytime you wake someone up, you can contextualize it as pushing someone one step closer to rage quitting.”

Corey talks about looking back and removing the noise by understanding and analyzing what is alerting, and knowing when to wake people up.

Debunking a Myth

Corey debunks the myth that is the “central conceit of AWS billing”.

Corey: “People talk about how the bill is too expensive… The pain as articulated by the business, is that someone wound up receiving a bill from Amazon that was 20% higher this month than last month… contextualizing what the Amazon bill is for someone that does not work with technology is not exactly a straight shot.”

Corey goes on to tell us what kinds of questions to ask when the bill is 20% higher, and how tying the AWS bill into PagerDuty is a “DON’T DO THAT” situation, due to how billing works at AWS, as the bill is always a trailing function which requires more analysis than waking someone up at 3am.

Advice from Corey: “I would be fine with that going to a slack channel during business hours.”

Don’t Train People to Ignore Alerts

Corey and Julie talk about how people get trained to ignore alerts, either through distribution lists or API quota limits.

Corey: “If something blows up and this is hitting your email, surprise suddenly you’re not getting actual important emails for a while until everything stabilizes.”

Corey talks about how alerts that wake people up should be emergencies, things that are actually damaging to the business if they don’t get fixed… with exceptions. Corey goes on to talk about how peoples patience can get eroded and terrible patterns can emerge.

Do Something About it, Owning your Code and Services

Corey talks about the nuanced story behind why things don’t get fixed, and why it becomes a challenge to tune services.

Corey: “What made it terrible [the on-call rotation] was that the VP in charge of engineering was a big fan of a software package called Big Brother… this VP was also a fan of AWS… the answer that the braintrust came up with was to have three of them running simultaneously, so when you get paged you don’t have to worry unless you get paged by all three of then and then you know it’s real.”

Corey about how driving cultural transformation does require executive sponsorship and that buy-in will get you everywhere.

Dogmas

Julie and Corey discuss ingrained dogmas and Corey talks about how we try to reinvent everything, and how that isn’t necessarily.

Corey: “There’s this ongoing narrative within this space of if you are not reinventing everything you are doing then there’s something awful that’s happening.”

Corey goes on to discuss how you don’t have to throw everything out you have done to be successful.

The Greatest Cloud Influencer in the World

From suits to personality, Corey talks about what he wishes he knew years ago. Corey talks about owning the “thing that makes you special”, and finding your niche.