Corey discusses how he became a cloud influencer, what that means for his credibility, and how people now listen to him.
Corey: “I have been a fan of PagerDuty for a long time, largely because I remember a time working on systems before PagerDuty existed.”
Corey discusses why waking the right people up at the right time makes it much less of a “Call of Duty” situation.
Corey wanted a problem that was strictly business hours only, Julie and Corey discuss how it all ties together with being on call.
Corey: “One of the interesting pieces that we see around all of those different business functions is that in most companies, engineering is usually the only group that needs to be awake at two in the morning that needs to deal with an emergent event.”
Corey goes on to explain that there are exceptions to this rule, and how businesses should staff accordingly if it is a 24 / 7 critical system, and how empowering folks to fix things that are broken is the start of cultural change.
Corey: “Everytime you wake someone up, you can contextualize it as pushing someone one step closer to rage quitting.”
Corey talks about looking back and removing the noise by understanding and analyzing what is alerting, and knowing when to wake people up.
Corey debunks the myth that is the “central conceit of AWS billing”.
Corey: “People talk about how the bill is too expensive… The pain as articulated by the business, is that someone wound up receiving a bill from Amazon that was 20% higher this month than last month… contextualizing what the Amazon bill is for someone that does not work with technology is not exactly a straight shot.”
Corey goes on to tell us what kinds of questions to ask when the bill is 20% higher, and how tying the AWS bill into PagerDuty is a “DON’T DO THAT” situation, due to how billing works at AWS, as the bill is always a trailing function which requires more analysis than waking someone up at 3am.
Advice from Corey: “I would be fine with that going to a slack channel during business hours.”
Corey and Julie talk about how people get trained to ignore alerts, either through distribution lists or API quota limits.
Corey: “If something blows up and this is hitting your email, surprise suddenly you’re not getting actual important emails for a while until everything stabilizes.”
Corey talks about how alerts that wake people up should be emergencies, things that are actually damaging to the business if they don’t get fixed… with exceptions. Corey goes on to talk about how peoples patience can get eroded and terrible patterns can emerge.
Corey talks about the nuanced story behind why things don’t get fixed, and why it becomes a challenge to tune services.
Corey: “What made it terrible [the on-call rotation] was that the VP in charge of engineering was a big fan of a software package called Big Brother… this VP was also a fan of AWS… the answer that the braintrust came up with was to have three of them running simultaneously, so when you get paged you don’t have to worry unless you get paged by all three of then and then you know it’s real.”
Corey about how driving cultural transformation does require executive sponsorship and that buy-in will get you everywhere.
Julie and Corey discuss ingrained dogmas and Corey talks about how we try to reinvent everything, and how that isn’t necessarily.
Corey: “There’s this ongoing narrative within this space of if you are not reinventing everything you are doing then there’s something awful that’s happening.”
Corey goes on to discuss how you don’t have to throw everything out you have done to be successful.
From suits to personality, Corey talks about what he wishes he knew years ago. Corey talks about owning the “thing that makes you special”, and finding your niche.
Corey is the Cloud Economist at The Duckbill Group. Corey specializes in helping companies improve their AWS bills by making them smaller and less horrifying; hosts the Screaming in the Cloud and AWS Morning Brief podcasts; and curates Last Week in AWS, a weekly newsletter summarizing the latest in AWS news, blogs, and tools, sprinkled with snark.
Julie Gunderson is a DevOps Advocate on the Community & Advocacy team. Her role focuses on interacting with PagerDuty practitioners to build a sense of community. She will be creating and delivering thought leadership content that defines both the challenges and solutions common to managing real-time operations. She will also meet with customers and prospects to help them learn about and adopt best practices in our Real-Time Operations arena. As an advocate, her mission is to engage with the community to advocate for PagerDuty and to engage with different teams at PagerDuty to advocate on behalf of the community.