• TeraSky
• Mitchell Hashimoto’s post, My AI Adoption Journey
• OpenAI’s follow up, Harness engineering: leveraging Codex in an agent-first world
• Scott Rosenberg, Backstage As The Ultimate MCP Server
• Learn more about TeraSky’s Backstage plugins
• Our prior episodes on Platform Engineering feature Abby Bangser on what is Platform Engineering, Dave Bresci on IDPs, and Avantik Iyer on Backstage
• Join the PagerDuty Commons!
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Developer Relations: Trust is Built in the Trenches, Not Through Marketing. The Apple story Tessa refers to.
• Jobs to be Done.
• Built for Devs
• Built for Devs on Product Hunt
• Check out Tessa’s Developer Adoption Playbook
• See the whole episode recording on our YouTube channel
• Join the PagerDuty Commons!
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Built for Devs
• Built for Devs on Product Hunt
• Check out Tessa’s Developer Adoption Playbook
• See the whole episode recording on our YouTube channel
• Join the PagerDuty Commons!
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Knightmare: a DevOps Cautionary Tale
• Knight Capital Group at Wikipedia
• Case Study 4: The $440 Million Software Error at Knight Capital
• The Knight Capital Disaster: How a Deployment Error Cost $460 Million in 45 Minutes
• SEC report
• Join the PagerDuty Commons!
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Get a copy of Frictionless for yourself, including the workbook!
• The DORA metrics are now at DORA.dev
• The SPACE Framework
• DX, the Developer Experience Platform
• Our episode with Helen Beal on Value Stream Management
• Our episode with Steve Pereira and Andrew Davis on Flow Engineering
• The Phoenix Project
• Some more depth on the RICE Scoring Model
• Join the PagerDuty Commons!
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Innostax
• Mindset, The New Psychology of Success by Carol S. Dweck
• Join the PagerDuty Commons!
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Mate Security
• Meta’s Agents Rule of Two: A Practical Approach to AI Agent Security
• Join the PagerDuty Commons!
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Lorin Hochstein “TTR: the out-of-control metric” from the Surfing Complexity blog.
• DORA and the DORA Metrics
• A bit more about statistical process control at deming.org
• Moving Past Shallow Incident Data by John Allspaw
• Join the PagerDuty Commons!
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Cisco AppDynamics
• OpenTelemetry
• Join the PagerDuty Commons!
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev
• PagerDuty Commons

• Check out Boaz’s webinar with Moshe Bar and Matt Johansen: The Science Behind Alert Fatigue in Security Teams: How to Beat It
• How Neuroscience Can Help Us Battle Alert Fatigue, at Dark Reading
• Understanding Alert Fatigue and How to Prevent It in the PagerDuty Resources
• OX Security
• PagerDuty Home Page
• Join the PagerDuty Commons
• Episode edited by Mandy Moore
• Transcripts by Rev

• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev
• PagerDuty Commons

• Learn more about Backstage
• Backstage GitHub
• Leveraging Internal Knowledge: Building AiKA at Spotify
• High Performance Backstage for High Performing Organizations
• The State of Backstage in 2025 @ KubeCon 2025
• Connect with Avantika
• PagerDuty Backstage Plugin
• PagerDuty Home Page
• PagerDuty Commons
• Episode edited by Mandy Moore
• Transcripts by Rev

• Watch Andy session at PagerDuty on Tour
• PagerDuty On Tour
• PagerDuty Home Page
• PagerDuty Commons
• Episode edited by Mandy Moore
• Transcripts by Rev

• Watch Andrew session at PagerDuty on Tour
• PagerDuty On Tour
• PagerDuty Home Page
• PagerDuty Commons
• Checkout.com
• Episode edited by Mandy Moore
• Transcripts by Rev

• Watch Laura session at PagerDuty on Tour
• PagerDuty On Tour
• PagerDuty Home Page
• PagerDuty Commons
• Backstage
• Episode edited by Mandy Moore
• Transcripts by Rev

• Dark Side of Open Source: The Community Strikes Back
• Open Observability Talks podcast
• What is Copyleft?
• Open Source Summit, a Linux Foundation event
• Vim After Bram: A Core Maintainer on How They’ve Kept it Going
• State of Open Con
• CNCF - The Cloud Native Computing Foundation
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Join the new PagerDuty Commons
• Check out Laundry Duty with Ben Hutchison
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• JNation Conference
• React Summit Conference
• React India Conference
• React Alicante Conference
• React Norway Conference
• Join the new PagerDuty Commons
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• groundcover
• eBPF
• Open Telemetry
• Join PagerDuty’s new community, PagerDuty Commons
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• DevOps Vancouver Meetup October 29, 2024
• Visit the new PagerDuty Commons community site!
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• APM - Application Performance Monitoring at Wikipedia
• Jeff’s article, Cloud Dependencies Need to Stop F-ing Us When They Go Down at The New Stack
• Jeff’s first episode on the podcast, Reliability of Cloud Dependencies, from February 2023
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Get your copy of Confident Devops use code KoganPage20 for 20% off
• Project to Product: How to Survive and Thrive in the Age of Digital Disruption with the Flow Framework by Mik Kersten
• Open Telemetry
• Our episode on Internal Developer Platforms with Dave Bresci
• Agile Conversations by Douglas Squirrel and Jeffrey Fredrick
• Grover’s Algorithm at Wikipedia
• Google’s XPRIZE Quantum for Real-World Impact
• Cybernetics: or the Control and Communication in the Animal and the Machine by Norbert Wiener
• *A Mind at Play: How Claude Shannon Invented the Information Age by Jimmy Soni and Rob Goodman
• Phenomenology at the In Our Time podcast by the BBC
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Grafana Labs. Join the community
• Brandy’s post on monitoring AWS Fargate
• Get started with Grafana and Java
• Prometheus
• Network Monitoring with Grafana and Prometheus
• Open Telemetry
• Raspberry Pi
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Get your copy of Project Hail Mary by Andy Weir
• More on Larry Niven
• The Three-Body Problem by Cixin Liu
• Charles Stross
• Starship Troopers
• PagerDuty Home Page
• Come hang out with us in the Hangops Slack
• Episode edited by Mandy Moore
• Transcripts by Rev

• Cory Doctorow’s craphound.com
• Get your copy of Red Team Blues
• Learn more about Radical Candor by Kim Scott
• Elora mentions The Locked Tomb Series by Tamsyn Muir and The 7 Habits of Highly Effective People by Stephen R. Covey
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Get your copy of Tidy First?
• Join a future episode! https://bit.ly/PageItBookClub
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Techtonica
• Ada Developers Academy
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Learn more, join the newsletter and office hours at https://flowengineering.org/
• Order Flow Engineering
• More on OODA
• If you’re not familiar with the parable of the blind men and the elephant
• Our prior episodes with Helen Beal on Value Stream Management and Dominica DeGrandis on Making Work Visible
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Winglang - A programming language for the cloud
• Winglang Playground
• Wing Cloud
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• A writeup of a talk by Breanne about doing transformative work in sometimes-reluctant orgs.
• A Diana Initiative talk by Breanne about becoming a security partner, and an accompanying Gusto blog post on the subject.
• Reinventing Cybersecurity, edited by Jasmine Henry, is free and has a chapter from Breanne about cultivating a positive and open security culture.
• Read the Fantastic Manual from BSides 2022 talks about knowing what to document and how to keep the right information out there in the right form.
• When it comes to talking about security partnership, this pair of Netflix posts are about as definitive as it gets: one and two.
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Get your copy of Kill it with Fire
• https://www.bellotti.tech/
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

Additional Resources

• Adidas
• Adidas Engineering Blog
• PagerDuty Home Page
• Join the PagerDuty Community
• Episode transcribed by Rev

• Jeli.io
• Our episode on Psychological Safety with Tim Geraghty
• Learning From Incidents
• PagerDuty Home Page
• Learning from Incidents
• Episode edited by Mandy Moore
• Transcripts by Rev

• Want a copy of After the Gold Rush? Send us an email
• Join us on a future episode! bit.ly/PageItBookClub
• After The Gold Rush at Goodreads.
• More from Steve McConnell
• The Gang of Four: Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma, Richard Heim, Ralph Johnson, and John Vlissides
• The Agile Manifesto
• Extreme Programming
• The Phoenix Project by Gene Kim, Kevin Behr, and George Spafford; and The Unicorn Project by Gene Kim
• Kill it with Fire by Marianne Bellotti
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

Additional Resources

• PagerDuty Home Page
• Backstage.io
• PagerDuty plugin for Backstage
• Join the PagerDuty Community
• Episode transcribed by Rev

• ElevateCX
• Support Driven
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Want to be on a bookclub episode? Have a book we should read? Let us know! https://bit.ly/pageitbookclub
• Close to the Machine: publisher’s website, Goodreads
• Mandi mentions The Cathedral and the Bazaar by Eric S. Raymond; and Dreaming in Code by Scott Rosenberg
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Powertools for AWS Lambda
  • Python
  • TypeScript
  • Java
  • .NET
• Powertools for AWS Lambda - Automation
• Powertools for AWS Lambda - Discord
• PagerDuty Home Page
• Working in Public: The Making and Maintenance of Open Source Software by Nadia Eghbal
• Episode edited by Mandy Moore
• Transcripts by Rev

• Syntasso
• The CNCF Platforms Working Group
• CNCF Platforms White Paper and Platform Engineering Maturity Model
• Gregor Hohpe’s Platform Strategy
• Team Topologies by Matthew Skelton and Manuel Pais. Manuel was featured on an earlier episode.
• KubeCon | CloudNativeCon Europe 2024
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Welcome Tiago episode
• Episode 100! Pagey’s Nostalgia Hour
• Sustainable On-Call Culture with Paige Cruz
• Team Topologies with Manuel Pais
• The Incident Response Landscape with CTO Tim Armandpour
• Prioritizing Post-Incident Actions with J. Paul Reed
• Incident Communications with Kat and Mandi
• Some of our events: CodeMash, DevOpsDays Taipei, DevOps Barcelona, Support Driven, AWS re:Invent
• January 2024 Terraform Roundtable
• PagerDuty and Backstage: GitHub, on the Backstage marketplace, and more from [The Unplanned Show]()
• PagerDuty Status Pages
• PagerDuty Blog
• PagerDuty Runbook Automation
• Intro to PagerDuty Copilot and sign up for Early Access
• PagerDuty Analytics Insights, plus videos for Q3 and Q4
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• The Python Software Foundation
• PyPI, the Python Package Index
• Tidelift
• Python on Mars and F´, a NASA project for embedded and spaceflight systems
• Practical Object-Oriented Design by Sandi Metz
• CISA, the Cybersecurity & Infrastructure Security Agency (USA)
• AWS Heroes
• Vint Cerf’s TEDxMidAtlantic talk and a more recent Google Cloud interview about the Interplanetary Internet project
• pdpyras, a Python module for accessing PagerDuty’s APIs
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Dr. Richard Cook’s How Complex Systems Fail
• Papers We Love
• LFI - the Learning From Incidents community.
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Elora’s acapella group, Countermeasure
• Hear all the barbershop ringtones
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• The Goal by Eliyahu M. Goldratt at Goodreads
• Find The Phoenix Project, The Unicorn Project and others at IT Revolution Press
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Read the posts here: Part 1, Part 2, Part 3
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev
• Team Topologies
• Team Topologies Academy

• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev
• Redmonk on AI and developer experience

• PagerDuty’s Incident Response training and the rest of our Ops Guides
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Scott’s first Dev post about ngrok
• Find out more about ngrok
• “Just write business logic” article: The Self Provisioning Runtime
• The folks at J on the Beach are not kidding around
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Visit the Value Stream Management Consortium for the State of VSM Reports and 20% discount on annual Influencer membership using code PAGERDUTY20
• Find Flowtopia sessions and other videos on the VSM YouTube Channel
• The DevOps Institute
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• PagerDuty Home Page
• Guide to assembling a year in review
• Building Greater Operational Efficiency with PagerDuty
• PagerDuty’s Service Ownership operations guide
• The Sustainable Web Manifesto
• Sustainable Web Design by Tom Greenwood
• Ecograder’s “How Green is Your Website?” tool
• Hubspot’s “State of Burnout in Tech” 2022 report
• PagerDuty’s Status Page and Status Update Notification Templates features
• Episode edited by Mandy Moore
• Transcripts by Rev

• AWS Startups
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• PagerDuty Home Page
• Past Episode: Support Career Stories
• PagerDuty’s Incident Response Ops Guide
• PagerDuty’s Customer Support Ops Guide
• Episode edited by Mandy Moore
• Transcripts by Rev

• PagerDuty Home Page
• Metrist
• Watch Jeff’s PagerDuty Summit 2022 session on YouTube
• Episode edited by Mandy Moore
• Transcripts by Rev

• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Jeremy Clark
• Sweetwater
• PagerDuty Home Page
• Episode edited by Mandy Moore
• Transcripts by Rev

• Join Ops.IO and follow PagerDuty!
• Follow LearnAboutOps on Twitter.
• Blink
• Check out our other community focused episodes: Developer Communities with Mary Thengvall and Julie Gunderson and Open Source Communities with benny Vasquez.
• PagerDuty Home Page.
• Episode edited by Mandy Moore.
• Transcripts by Rev.

• Mary’s book, The Business Value of Developer Relations
• DevOpsDays Conference series
• PagerDuty Home Page
• Transcripts by Rev

• PagerDuty Home Page
• Transcripts by Rev

• Resurface
• PagerDuty Home Page
• Transcripts by Rev

• Find Stuart on Cisco DevNet
• The Cisco BGPStream Twitter account
• BGPlay and BGPmon
• Python Libraries: netmiko, NAPALM, pyATS
• RANCID, a true classic
• PagerDuty Home Page
• PagerDuty Community Forums
• Transcripts by Rev

• No, Red Hat 4 is not still available.
• PagerDuty Home Page
• Join the Community Forums
• Transcripts by Rev

• The OWASP Top 10
• PagerDuty’s Security Training
• Hackers. A cautionary tale of password security and the need for multifactor auth.
• PagerDuty Home Page
• Transcripts by Rev

• Buoyant
• Linkerd Project
• Jason’s presentation, Diagnose Application Issues with Linkerd
• PagerDuty Home Page
• Transcripts by Rev

• What is Runbook Automation?
• Learn more about PagerDuty Process Automation
• Check out this post for more on Automated Diagnostics and this one for AWS plugins.
• See Automated Diagnostics in action in this recording from our Twitch channel.
• PagerDuty Home Page
• Join our Community Forums
• Transcripts by Rev

• More information about SBOMs and US federal government regulations from:
  • CISA
  • NTIA
• Scribe Security
• PagerDuty Home Page
• PagerDuty Community Forums

• Check out Pablo’s blog post on how to move from support to software development
• Join the Support Driven community (and consider attending Support Driven Expo in October, where Kat will be a speaker!)
• Transcripts by Rev

This week’s episode may be difficult for some of our regular listeners. We understand and hope you’ll be back with us for the next episode.

Additional Resources

• Get help in your country
• Learn more about the Pomodoro Technique
• Some collected advice from Ask a Manager regarding talking about mental health at work.
• PagerDuty Home Page
• Transcripts by Rev

• Fiberplane
• Best Practices for Oberservability on the Fiberplane blog
• The Single Pane of Glass is a Myth on the Fiberplane blog
• PagerDuty Home Page
• PagerDuty Community
• Transcripts by Rev

• Solidify.dev
• Educative.io
• Superbosses
• Radical Cander
• Feedback Fallacy
• PagerDuty Home Page
• Transcripts by Rev

• Register for Summit Today!
• PagerDuty Home Page
• Transcripts by Rev

• The Tao of Programming by Geoffrey James
• Join us online and in person at PagerDuty Summit
• PagerDuty Home Page
• Transcripts by Rev

• All about AlmaLinux
• The Linux Distro Family tree at Wikipedia
• PagerDuty Home Page
• Transcripts by Rev

• Docs for Developers: An Engineer’s Field Guide to Technical Writing by Jared Bhatti
• Docs Like Code by Anne Gentle
• Docs as Code Philosophy at Write the Docs
• Honeycomb Intro to Observability Workshop
• Honeycomb Advanced Instrumentation Workshop
• PagerDuty Home Page
• Transcripts by Rev

• Check out Nobl9
• Register for SLOConf!
• Alex’s book, Implementing Service Level Objectives
• PagerDuty Home Page
• Transcripts by Rev

Time for a cross over! Today Page it to the Limit host Mandi Walls, DevOps Advocate at PagerDuty and Julie Gunderson, Reliability Advocate at Gremlin host a special two-parter. They are interviewing Kolton Andrus, co-founder of Gremlin and Alex Solomon, co-founder of PagerDuty. Each of them share the origins of their respective companies, both of which began in their respective work at larger organizations. Kolton and Alex reflect on how they identified the space where they could build their respective companies and the shift from larger entities to start ups. Each of them offer up some excellent insight!

• Find the first part of the episode at https://www.gremlin.com/podcast/
• Learn more about Gremlin at https://gremlin.com
• PagerDuty Home Page
• Join the PagerDuty Community Forum!
• Transcripts by Rev

• Security Champions Playbook at Snyk
• Snyk Home Page
• PagerDuty Home Page
• Transcripts by Rev

• PagerDuty Home Page
• Dutchie is hiring!
• Transcripts by Rev

• “Don’t Say That at Work: Lessons Learned from a Lifetime of Mistakes” [ Amazon ] [ Gumroad (25% off) ]
• Receive a 25% discount on any of Michael’s other materials on Gumroad using the following promo code: pager-duty-25
• PagerDuty Home Page
• Transcripts by Rev

• Dominica’s author page at IT Revolution
• Dominica’s immersion course on Making Work Visible
• PagerDuty Home Page
• Transcripts by Rev

• The VOID
• Submit an incident to the VOID
• The VOID Podcast
• Dr. Richard Cook
• PagerDuty Home Page
• Transcripts by Rev

• PagerDuty Home Page
• Transcripts by Rev

• Greg’s article, So You Want to be a Superhero
• Orion Labs
• PagerDuty Home Page
• Transcripts by Rev

• Need recruiting help? Check out Emerald Talent
• Join the PagerDuty Community
• PagerDuty Home Page
• Be a guest on a future episode
• Transcripts by Rev

• The Support Driven community
• Check out our ops guide on Customer Service Operations
• Be a guest on Page it to the Limit! Fill out our form here.
• Follow us on Twitch
• Join the PagerDuty Community
• PagerDuty Home Page
• Transcripts by Rev

Popular myths or misconceptions about on-call

One of the biggest myths is that on-call is just an extra part of a SRE or sysadmin’s job. That it’s not really a big part of their duties. It’s just a thing you do; it hasn’t always been taken seriously, especially the impact of being on-call to the individual.

Remember - on-call isn’t just for ops or SRE. Andrew Clay Shafer used to describe himself as a “conscientious developer”, even prior to the ideas of DevOps. Because he thought about things this way, it caused him to be a better developer, and this heavily contributed to the foundation of the DevOps movement.

Software engineers are often resistent to being on-call because of what they think it means - based on the horror stories they hear from their coworkers and friends who work in Ops.

How has on-call changed?

Jay: “Automation has made so much of the difference”

Well-documented automation makes it easier to track down what might be contributing to issues. Having things watching what is going on through the deployment process and watching what’s going on. We have a greater ability to spin up replacement systems, too.

We are changing from a model of having one team who is on-call for everything inside the business; now it is more about selected domain experts on call for the thing they know really well. Being on-call as a developer, you know you are only being called about things you know about. Additionally, the more people that go on call, it’s much less actual impact to all the folks who are on-call. So the experience is a lot different. “We’ve reduced the individual blast radius by distributing it” - Jay.

“The beautiful thing about going on-call is you get to go off-call. If you aren’t on-call, I have news for you - you’re always on-call” - Matt. It’s very relieving to know you are not on call, so you don’t have to worry that someone will call you. “Trust me - your ops team knows how to find you, and they will” - Matt.

On-call requirements are different

Not every company or service requires 24 hour on-call support. When you are thinking about where you want to work, consider this. That said, if you do work for an organization that provides a service around the clock, on-call is likely a part of that job, and everyone should consider it part of their service ownership. But ultimately, make the decision for the role that works for you. It’s less about the title or role, than it is for the type of company or organization and what they need. As Jay points out, “in the end, we are all just people, and we have basic requirements - like eating, having water, getting enough sleep, and spending time with people we like. On-call should still let you do these things”.

A good question to ask when getting into a role that has a on-call component, is ask “how are incident responders rotated off of an incident?” Responders stop being effective after a couple of hours - understanding things like “what’s the size of the rotation?”, “what are the expectations of a responder during an incident?”, are much more important to know than “how often will I get paged?”

How to avoid having an on-call nightmare

Jay: “It always comes down to tech debt. It’s amazing how much tech debt comes down to a lack of documentation. It becomes one of those scary parts that if it falls down, nobody will know what to do”

Additional Resources

• PagerDuty Home Page
• Episode transcribed by Rev

We’re rerunning some episodes in October. Since the holidays are coming up and retail companies will be getting ready for that important time of their year, we wanted to rerun this episode on hypercare featuring the New York Times Election Reponse Team. We’ll be back in November with new episodes!

Additional Resources

• PagerDuty Home Page
• Hypercare Readiness Checklist
• NYT Career Page - Interested in working for The New York Times? Check out their current listings!
• NYT Subscription Page - Support The New York Times via a digital and/or paper subscription.
• Election Readiness at The New York Times - Want more information about the technology behind the NYT election reporting? Check out their article on The Times Open!
• Join the PagerDuty Community
• Episode transcribed by Rev

Additional Resources

• Check out Teleport
• View our Twitch stream recording featuring Twitch and PagerDuty
• Army Cyber Command
• The NIST Cyber Security Framework
• PagerDuty Home Page
• Transcripts by Rev

• Get a preview of Observability Engineering
• o11ycast Episode 40: Player Experience with Nick Herring of CCP Games
• Honeycomb CEO Christine Yen on Episode 2 of Page it to the Limit
• More on Cribl
• PagerDuty Home Page
• PagerDuty Community
• Transcripts by Rev

• Join the Skycrafters Community
• Brittany’s session The Unofficially Official Guide for Config Management at Scale at ChefConf
• PagerDuty Home Page
• PagerDuty Community
• Transcripts by Rev

• https://psychsafety.co.uk for everything and anything psychological safety, sign up to the newsletter and join the community.
• https://tomgeraghty.co.uk is Tom’s personal site.
• Amy Edmondson’s original 1999 paper
• Google’s project Aristotle, showing psychological safety to be most important in team performance
• The Four Stages of Psychological Safety
• The Fearless Organization by Amy Edmondson
• PagerDuty Home Page
• PagerDuty Community
• Transcripts by Rev

• Learn more about blameless postmortems in our Ops Guide.
• The Infinite Hows by John Allspaw
• Our episode with John Allspaw
• Our episode with J. Paul Reed
• Sidney Dekker talks about accountability in Just Culture
• Tim works at Xero
• PagerDuty Home Page
• Join the PagerDuty Community
• Transcripts by Rev

• Work with us!

  • Head of Community: US or Canada.
  • DevOps Advocate (You get to do episodes of this show!) US or Canada

• Watch the Summit 2021 Sessions on demand at https://summit.pagerduty.com. Login to get access to the PDU classes and the API certification exam.

• Scott talked about his use of Instruqt.

• Looking for more on Idempotency? Here’s a quick definition.

• Join the PagerDuty Community for discussions, help, and challenges to earn swag!

• PagerDuty’s Autoremediation Ops Guide

• Sli.do for realtime audience Q&A, polls, etc.

• Get the PagerDuty Terraform Provider

• Stop by and say hello if you’re at DevOps Days Minneapolis

• Check us out on Twitch at PDCommunity

• The Phoenix Project

• The PagerDuty Foundation

• Rundeck

• PagerDuty Home Page

• Transcripts by Rev

• PagerDuty’s DevSecOps Guide
• Team Topologies book website
• DevOps Topologies website
• DevOps Handbook - co-authored by Gene Kim, Jez Humble, Patrick Debois, and John Willis
• Episode transcribed by Rev

• PagerDuty’s DevSecOps Guide
• OWASP
• OWASP’s DSOMM Framework
• OWASP’s SAMM Framework
• BSIMM Framework
• Episode transcribed by Rev

• For RedMonk’s current posts, research, and other data, please check out their home page!
• PagerDuty Home Page
• Transcripts by Rev

• For RedMonk’s current posts, research, and other data, please check out their home page!
• PagerDuty Home Page
• Transcripts by Rev

• PagerDuty Home Page
• Transcripts by Rev

Additional Resources

• Brad’s personal blog - includes links to his social as well as information about his book.
• The authors mentioned in the episode are Bruce Schneier and Richard Beijtlich, names link to their respective websites.
• PagerDuty’s DevSecOps Guide for broad information about DevSecOps and how to implement.
• The Waking Up App, one of Brad’s preferred mindfulness apps.
• Episode transcribed by Rev

To find a Girls in Tech chapter near you, visit https://girlsintech.org/.

Want to chat more about mentorship, DevOps, cloud operations, or just whatever, join the PagerDuty Community at https://community.pagerduty.com.

Additional Resources

• PagerDuty Home Page
• Transcripts by Rev

• PagerDuty Home Page
• Transcripts by Rev

✅ All tests passed pic.twitter.com/Owx6H0kzuV

Additional Resources

• PagerDuty Home Page

Additional Resources

• PagerDuty Home Page
• Emily’s book, DevOps for Dummies

Additional Resources

• PagerDuty Home Page
• What is Software Understandability? - blog post that accompanies this podcast
• Check out Rookout’s resources at www.rookout.com
• Continue the discussion either with Liran on Twitter or with us in PagerDuty Community
• Episode transcribed by Rev

Additional Resources

• Join the PagerDuty online community at community.pagerduty.com
• Check out the 30 Days of DevOps Activities at https://community.pagerduty.com/forum/c/30-days-of-devops
• For a list of How Tos and Getting Started Guides, visit https://community.pagerduty.com/forum/c/using-pagerduty
• For more on PagerDuty, check out the main PagerDuty site www.pagerduty.com
• Episode transcribed by Rev

Additional Resources

• PagerDuty Home Page
• Building & Scaling SRE Teams With Tammy Bryant
• Approaching Incident Response Compassionately With Connie-Lynne Villani
• Professional & Personal Self-Care With George Miranda
• Thinking About Your Humans With J. Paul Reed Don’t forget to join the Community
• Episode transcribed by Rev

Additional Resources

• PagerDuty Home Page
• Building a Fully Managed Service on Kubernetes - JovianX blog post about launching a new, managed, service on Kubernetes
• Episode transcribed by Rev

Additional Resources

• PagerDuty Home Page
• Hypercare Readiness Checklist
• NYT Career Page - Interested in working for The New York Times? Check out their current listings!
• NYT Subscription Page - Support The New York Times via a digital and/or paper subscription.
• Election Readiness at The New York Times - Want more information about the technology behind the NYT election reporting? Check out their article on The Times Open!
• Join the PagerDuty Community
• Episode transcribed by Rev

Additional Resources

• PagerDuty Home Page
• Verdaccio - open source package as a local npm proxy for enterprises
• Snyk Blog
• MyDevSecOps Community - Vendor-neutral community for developers who care about security - webinars and a slack community
• Episode transcribed by Rev

Additional Resources

• PagerDuty Home Page
• Hacker101
• Community Building Tools From HackerOne’s Head of Global Hacker Community
• Security@

Additional Resources

• PagerDuty Home Page
• PagerDuty Summit

Quintessence Anx: Welcome to Page It to the Limit, a podcast where we explore what it takes to run software and production successfully. We cover leading practices used in the software industry to improve both system reliability and the lives of the people supporting those systems. I’m your host, Quintessence, or QuintessenceAnx on Twitter.

Today we’re talking about Liz’s upcoming talk, Cultivating Production Excellence at PagerDuty Summit at the end of this month. We’re joined by Liz Fong-Jones, principal developer advocate at Honeycomb, where she makes developers, operators and workers on the whole more productive and empowered. Her background is in site reliability engineering with over 16 years of experience and has worked on a few products and services you may have heard of, like Google Cloud’s Load Balancer and Google Flights. Liz, welcome to the show.

Liz Fong-Jones: Thanks for having me.

Quintessence Anx: Getting right to it. We’re looking forward to your talk at Summit. Can you tell us a little bit about what inspired it?

Liz Fong-Jones: Yeah. I spent over 11 years at Google as a site reliability engineer, and over the course of my time there, I realized that there’s this gulf between what Google was saying was best practice, what Google was actually doing. Then what people were interpreting based off of the SRE book, based off of people who had left Google. There’s kind of this spectrum of experiences with production. I really wanted to kind of distill down those 11 years of experience at Google into something that was more concrete and practicable. Four quick things that you can do, or not so quick things, four things to work towards as we develop your culture.That’s kind of where this idea of production excellence and four principles of production excellence came from.

Quintessence Anx: That’s awesome. Related to that, what common myths or misconceptions about production did you find yourself answering from that experience?

Liz Fong-Jones: I think the most common misconception that I run into is people thinking, “Oh. Production has to be a nightmare. Production has to be a mess. This is why we pay ops people,” right? It’s this culture of masochism on the part of ops people or a culture of, almost like abusive behavior or just tolerance of bad behavior where people assume that production has to be bad. Therefore, that production has to be run by people that you pay to sit in the trenches. I think that that misconception was really, really thoroughly debunked for me at Google when I saw that there were teams that were just running their own services.

Liz Fong-Jones: That you didn’t need to have a network operation center. That you could have services that behaved perfectly fine and weren’t generating a lot of noise and that, as long as you had the right setup and you equipped people correctly, that people could actually be on call and not have it ruin their lives. I think that’s the number one misconception, is that production is scary, production is going to ruin your life. I’m here to tell you that no, it’s not going to ruin your life, but you have to clean up your service a little bit.

Quintessence Anx: That all makes a lot of sense. I could definitely see how painful experiences train your brain to think that it’s just going to be more pain. Speaking of pain resolution, what do you feel are the biggest mistakes people make when adopting SRE practices?

Liz Fong-Jones: I think the biggest mistakes people make when adopting SRE practices are, number one, they try to do everything that Google says exactly by the book. That clearly is not going to work, because your company is not Google and therefore different things are going to apply. Secondly, I tend to see that people try to adopt SRE all at once within their organizations. Especially in large enterprises, they have a leader who’s like, “We’re going to do SRE. We’re going to do SRE. Everyone do SRE.” It doesn’t really work that way. You have to start small. You have to expand out from there. You have to build up that model of what does a good SRE team look like in your organization before you try to scale it out.

Quintessence Anx: That makes sense. When people are trying to scale it out, is there anything that you want to expand on for that last?

Liz Fong-Jones: I think with regard to scaling it out, you have to develop what I think of as a community of practice within your SRE org. That doesn’t necessarily mean that everyone has to report to the same manager. Everyone has to be on a dedicated team, but instead, you have to think about how do you get these insights and knowledge shared between your various people who are fulfilling that SRE job role. Even if they’re not necessarily called that. In terms of their job title. How do you make sure good production practices are spread throughout your organization and people have somewhere to start, if they’re just getting started.

Quintessence Anx: That makes sense. It sounds like leaning in a little bit, lots of collaboration, how does this collaboration help improve observability? Basically the end result, right?

Liz Fong-Jones: Yeah. So we haven’t yet touched on what observability is. So let’s quickly define it for our listeners here. Observability is the ability to debug your system. The ability to ask any question about your system and understand what’s happening inside of your system without necessarily having to push new code. So observability is this, not just the tooling, it’s a socio-technical capability. It’s the ability of the people and the existing tooling working together in order to solve these problems. I think that’s kind of one dimension of collaboration, which is something that I’ve definitely heard described by my friend, Jessica Care as this idea of people and systems working together. I think that’s one dimension of collaboration. Then the other dimension of collaboration is collaboration across team boundaries. You shouldn’t have someone else’s service be a complete black box to you. You should have tracing or some kind of other capability to really tie your services together, both in terms of having your RBCs flow to each other, yes, but also being able to debug when those RPCs go wrong. I think that’s really how collaboration improves observability, is erasing some of those boundaries between services that would otherwise be opaque or hard to understand, or unwieldy when it comes to automation and systems working together.

Quintessence Anx: That all makes a lot of sense. Going back to your definition of observability, I know that you spend maybe a little bit of time talking about it, but when people are talking about asking questions of their system, do you have anything off the top of your head that can help people kind of understand what that means? It’s not a human dictated-question, it’s a human-dictated question converted to a machine question somehow, where you can actually get the right data.

Liz Fong-Jones: It’s this stream of consciousness. It’s this dialogue with your system. So when I get paged, the first thing that I try to figure out is how bad is it? Who’s impacted? Then from there, I start going into questions like why is this happening? Is it because this component has failed? Is it because we pushed a new version? Is it because this one user is sending us this really weird combination of traffic? Is it some combination of these things? Those are some of the questions I might be seeking answers to that help me debug and narrow down what’s happening inside of my system, so I can better mitigate and solve the issue.

So I think that when we’re trying to formulate these things, often our systems tend to hard-code us into only being able to analyze it along specific dimensions. Only being able to analyze things by service or only being able to analyze things by machine. It turns out that user behavior is a much more interesting spectrum of behavior. So you have to be able to understand is it because user one, two, three, four started issuing more queries? Can you break down by users? I think that that’s kind of where we start getting into the differences between traditional monitoring-based approaches and observability, where you can actually be in this dialogue with your system and iterate on your queries.

Quintessence Anx: That makes sense. When we’re talking about user behavior and things like that, how do you make sure that the questions are appropriately, I guess, granular? You can think you’re pulling data that helps you, only to find out maybe it’s too broad, too specific. You’re not getting out of it what you thought you were going to.

Liz Fong-Jones: Yeah. I think that there’s two answers to this. One of which is that, first of all privacy has to be paramount. You cannot have an audited access to production. People should be able to make sure that you’re only using your access for appropriate purposes. That means not inappropriately digging into user’s data. I think that’s one dimension. I think the other interesting dimension here, with regards to making sure you’re getting the right data, is always being able to check and verify your assumptions. Can you cross check your hypothesis and make sure that it holds up from a different angle?

For instance, if you think that it is only one user that’s having a problem, then you might cross check to make sure is the increase in error rate actually correlated with this user sending more traffic? Can you actually specifically, not just correlate it in time, but specifically see that user’s error rate spike? Can you make sure that it’s happening across multiple data centers and that’s not just an artifact of, “Oh, this user’s traffic happens to be going to this one data center that’s having a problem”? You have to approach it from multiple different angles to really get that kind of three dimensional shape.

Quintessence Anx: That makes sense. Getting into how people can design around that, people often treat their alerts around all these things as a safety blanket. How do you convince people it’s okay to remove something that’s too noisy, speaking of scope and things like that?

Liz Fong-Jones: Yeah. People tend to assume that just because a dashboard or an alert caught something the last time that it’s going to be good indefinitely into the future. That’s not necessarily true. Alerts and dashboards definitely have a shelf life. They’re good for catching things that are specific to one thing that bit you in the past, but your next outage is not going to look like your past outage necessarily. As these things accumulate and accumulate, eventually you wind up overpaging, where people get woken up for things that aren’t actually impacting system behavior. So you kind of have to pull the analytics out of PagerDuty. You have to say, like, “We got paged a hundred times over the past month.

That’s three times per day, and of those, 50 were not actionable. We didn’t do anything. We just turned it off and went back to sleep.” That’s not necessarily healthy. There’s a limited number of pages that you can cognitively interpret before your brain starts becoming overloaded and you just start ignoring or doing a bad job on everything. I think that that’s the lesson to learn. Is that over-alerting is equivalent to not alerting at all because you start treating all the alerts as noise. That’s kind of how I encourage people to approach it, is this idea that observability helps you debug from first principles. You kind of only need a few basic alerts and you don’t need a granular alert for everything in the system. That enables you to have that cognitive bandwidth.

Quintessence Anx: That makes a lot of sense. I know that some of the things that we talk about at PagerDuty, of course, is alert fatigue. To that same point, where you have too much of a good thing and now it’s not a good thing anymore. Something that comes to mind is how frequently, or not, to flush out those alerts. Not necessarily outright delete them, but how to iteratively edit them, and sometimes it depends what the setup is. Does it make sense to review it after an incidents resolved and say, “Hey. This helped me or not, or maybe on a sprint cadence or something”? Do you have any thoughts or recommendations on how often to edit, entropy balance your alerts?

Liz Fong-Jones: I think every two to four weeks as appropriate, because past that you lose the context of, “Hey. That alert was super spammy, because the person who is on call doesn’t even remember being paged for it.” It has to be relatively fresh. I think that sprint cadence is entirely appropriate, but I think along that same line, as you were saying, if you have a major incident, you might have a retrospective. Your retrospective might say we think we should add an alert for X. Revisit in six to eight weeks, make sure you still feel that way.

Quintessence Anx: That makes sense. Pivoting a little bit, but relevantly, is there anything you like or dislike about alerting platforms as they are now in general?

Liz Fong-Jones: One of the favorite things that I had access to when I was at Google was the alert manager system at Google, which was very much infrastructure as code. As the primary way of interacting with it. So you could kind of see who is going to be on call at a given time. You could edit the text file and submit a pull request to put someone else on call. I think that programmatic access is something that I wish were a lot easier outside of Google. Overall, I think that having people start dispatching alerts, having people have easy ways of escalating is an enormous change that has enabled a lot of more legacy enterprises to move towards a more DevOps-y model. I’m really excited about it.

Quintessence Anx: That makes sense. Okay. Talking a little bit about Honeycomb. How is Honeycomb adapting to COVID? Since that’s on everyone’s mind.

Liz Fong-Jones: One of the funny stories is that last year we started doing these disaster drills where we would all pick a week where everyone worked from home and the company would be full remote. This was in July, August, September of last year. When COVID came and we all had to work remote, everyone had at least some exposure to working remotely, which was really, really interesting and made our lives a lot easier, because all of the remote access challenges that people were struggling with outside of Honeycomb were not really giant issues for us. It meant that we knew how to talk to each other in Slack.

Liz Fong-Jones: We knew how to debug incidents without sitting in the same room together. Definitely I think as this has wound longer and longer and longer, I am definitely starting to miss seeing my colleagues in person. I’m starting to miss physically sitting over the same whiteboard and creating something with them. I think that’s been the main challenge for us, but it’s definitely been an exciting time for us at Honeycomb, given that we really have had customers that are doing critical work related to either COVID research, including the Chan Zuckerberg Initiative, as well as productivity apps like Slack that are using Honeycomb. It’s been very, very important, even more important for us to be available to help people debug their systems so that the whole world can stay productive during this.

Quintessence Anx: That makes sense. You mentioned about having a good short term plan for how to, to your point about disaster scenarios and practice, but since this became a longterm disaster, rather than a short term one, is there anything that you’d like to talk about with how you manage things that changed beyond a week or a day?

Liz Fong-Jones: Yeah. The scenario we were all imagining in August and September originally was like, “Hey. We might lose the lease on the office because we didn’t necessarily have enough money to pay a new security deposit when our lease ran out.” Then it was like, “Hey. This might be helpful if there was an earthquake.” Or the Bay Area gets taken out of commission, but instead it’s everyone that’s working from home. I think that, as you were saying this, this has gone on longer and longer, work/life balance is increasingly hard. It’s hard to actually peel myself away from the computer when I’m just working, working. Times zones also. When you work three hours apart from the rest of your colleagues, the temptation is there to just sit and work and work and work. That’s not necessarily the best idea in the world. Yet it’s a thing that, without clear boundaries between work and not, it’s kind of hard to navigate. Then I think the other thing is, personally, I’m in the middle of moving in a pandemic, which is also fun and hilarious.

Quintessence Anx: Yeah. I actually recently did my own move and I think mine was a bit shorter in terms of distance than yours, but it’s definitely a logistical thing.

Liz Fong-Jones: Our training as engineers who focus on operations is really, really helpful for that. In that it allows you to break things down into pieces. Delegate tasks, have an incident commander. My spouse and I were planning to move and we were like, okay, this is how we’re parceling out the roles. This is who’s on call for doing what. That’s made life a whole lot easier. So that bleed over between work skills and personal skills is really, really helpful to have.

Quintessence Anx: Yeah. It is amazing how we can tie in our work life into our home life a little bit. Getting back to the boundaries, I guess, of work/life balance. It can be hard. I mean, I know some people when I see the Twitter and Doom scroll a little bit, people talk about, “Oh. Well, I have a home office or I don’t have a home office,” but some people, they’re in a smaller living space. They have multiple people to that living space. They have small kids in that living space or whatever, and it interferes with that boundary. Is there anything in your experience that maybe you do that can help people separate work from home when work is home?

Liz Fong-Jones: Yeah. I think that the number one thing that I’ve done there is that for the computer that I use for my gaming and personal stuff, I have taken work Slack off of my Slack client there so that when I go to play computer games, when I go to chat with friends, that I’m not seeing that white notification light on the Slack. If I really need to access work Slack from that computer, I’ll open it up in a web browser. I think that’s the number one life hack there. Similarly, we all are pretty much at our desks, or sorry, we’re pretty much all at home anyways. We’re not on the go.

So why is it that you have your work Slack installed on your phone at this point? If you’re working, you’re going to be at your desk. If you’re not working, if you’re, as you were saying, Doom scrolling on Twitter, you don’t want that work notification to pop up. Seriously, take work Slack off of your phone. You do not need it right now. If there’s something important, you’ll get a PagerDuty notification because someone needs you. So having that trust that someone can get hold of you when they really need to will enable you to decouple and not pay attention when you want to go do something else.

Quintessence Anx: Awesome. Real quick, before we wind down, speaking of people getting a hold of you, establishing that trust, is there just a bit of advice you have for people for next steps or steps they can take to build that trust if they’re in a situation where they’re cleaning up their alerts or they’re in the process of making change, but they’re not there yet, they’re just walking towards it, that can help them get there faster?

Liz Fong-Jones: I think that as you’re working towards cleaning up alerts, you have to apply to the Marie Kondo thing. Does this bring me joy? If it doesn’t bring me or my users joy, you should get rid of it and setting those top level alerts to know when real users are having pain. Like high error rate alerts are super helpful at top level, because then you will know, even if you eliminate some lower level alert, that you would get alerted if there were a serious problem affecting your end users.

Quintessence Anx: Okay. Was there anything else you really wanted to highlight from your talk, since everyone’s super excited to learn from you in a couple of weeks?

Liz Fong-Jones: No. I think that we teased elements of the talk quite well. You’ll hear more from me about the subjects of observability, service level objectives and collaboration during my talk.

Quintessence Anx: All right. Well, that’s awesome. For everyone listening in, please make sure to catch Liz’s talk at PagerDuty Summit. It’s going to be at 9:50 AM Pacific Time on the 23rd of September. Also make sure to check out Liz’s office hours, and we’re going to be posting links to these in our show notes, as well as her blog post relevant to her talk, which is Sustainable Operations and Complex Systems with Production Excellence at InfoQueue. Before we head off, we have two questions that we ask every guest. Are you ready?

Liz Fong-Jones: Sure. Hit me.

Quintessence Anx: All right. What is one thing that you wish you would’ve known sooner when it comes to running software in production?

Liz Fong-Jones: Making mistakes is normal. Having your first outage is completely normal. The sooner you get your first, “I was responsible for that outage,” out of the way, the more comfort you’ll get in production, so don’t be afraid to make mistakes.

Quintessence Anx: All right. Awesome. Is there anything you’re glad that we did not ask you today?

Liz Fong-Jones: I am glad you didn’t ask me about what specifically Google says it does that it doesn’t do. I think that that’s a fascinating topic with lots of skeletons in that closet.

Quintessence Anx: Fair enough. Liz, thanks again for joining us.

Liz Fong-Jones: It was a pleasure. Looking forward to speaking at PagerDuty Summit.

Quintessence Anx: All right. Awesome. We’re looking forward to hearing from you. This is Quintessence wishing you an uneventful day. That does it for another installment of Page It to the Limit. We’d like to thank our sponsor PagerDuty for making this podcast possible. Remember to subscribe to this podcast if you like what you’ve heard. You can find our show notes at pageittothelimit.com and you can reach us on Twitter @Pageit2Limit, using the number two. That’s Pageit2Limit with the number two. Let us know what you think of the show. Thank you so much for joining us and remember, uneventful days are beautiful days.

Additional Resources

• PagerDuty Home Page
• PagerDuty Summit
• Liz’s Office Hours
• Sustainable Operations in Complex Systems with Production Excellence on InfoQ’s Blog
• Episode transcribed by Rev

Connie-Lynne shares her excitement about speaking at PagerDuty Summit, and where her talk stemmed from. She talks about the notion of how we can be both being effective and still acknowledge the humanity within ourselves and our colleagues.

Connie-Lynne: “I’m so excited about this topic of compassion and being kind to people, and actually taking this approach with integrity to work and to incidents.”

Approaching Incident response Compassionately

As a continued topic on the podcast, Connie-Lynne and Julie discuss why it is important to have compassion during the incident response process. Connie-Lynne discusses the stresses on people when an incident happens, and how it can affect thinking.

Connie-Lynne “To make yourself feel ok is really important, because that way you are open-minded when you are coming to solve a problem.”

How Do We Make Things Blameless

Connie-Lynne talks to us about how there are contributing factors to incidents, and how avoiding blame is important, she shares with our listeners words she uses on her team at Fastly.

Connie-Lynne: “Root Cause, particularly when you are talking about complex systems is an outdated concept.”

She continues to talk about the language used in incidents, and using the words trigger and contributing factors instead of terms that point blame.

Talking about the Clown

Connie-Lynne talks to us about how there may be mistakes, but they can also be “happy little trees”. She continues to talk about how methodologies are tools to use, and how the goal is to make things better and learn from mistakes.

Connie-Lynne: “That’s what we want to look at, how do we make things better… and that is where compassion comes in, you are looking at the positives. You aren’t focused on the negatives, the negatives happen, you acknowledge that the negatives happen.”

To see Connie’s full talk on 9/23 register free for PagerDuty Summit at the link below.

Additional Resources

• PagerDuty Home Page
• PagerDuty Summit
• PagerDuty University
• Episode transcribed by Rev

Myth About Self-Care

The biggest myth about self-care is that it’s not entirely about you. It starts with you; you have to help yourself before you help others. You should make sure that you are your own top priority, but those priorities need to be balanced.

Why This Topic Now

George talks about how he’s going through the worst emotional period of his adult life. He describes that we all are experiencing incidents in our own lives, and it’s our instinct to try and resolve them. One way we do this, at least temporarily, is to compartmentalize our incidents to give us time to deal with other things before addressing the problems. But, at some point, we run out of compartments.

Lessons That Apply Both Personally and Professionally

Everyone is struggling to cope with these ambiguous losses. We have all experienced loss of many things that kept us going. And that grief is hard to deal with because there is no end in sight. “Everyone, and I mean everyone, is going though this. Live is hard right now no matter how you slice it.”

George talks about how we’ve been at this for a little over six months now and are now hitting a wall where we are all running at diminished capacities. And he recommends we need to walk with empathy for our fellow humans.

How To Not Trigger Personal Incidents

I think we need to have each others backs–which isn’t all that different than dealing with things professionally. We need to be gentle with each others psychological safety. We need to be gentle with each other

We need to go above and beyond when we communicate with those we call friends. That’s how we build and maintain trust.

Mindful of Professional Self-Care

We have to be ruthless about prioritizing what we’re going to work on. If we’re lucky, we have really only about 3-4 good focus hours in a day. Then you have to decide where is it going to be okay to under deliver? Things are going to fall.

It falls on the leadership to set the pace, but it also falls the the team to return in kind.

Handling Incidents From Perspective of Self-Care

Scott and George talk about how the number of incidents appears to be increasing during the COVID era. Scott shares that PagerDuty has seen the volume of incidents handled by its customers remained fairly flat for several leading up to March 2020. Since that time incident volume has steadily increased by 7% each month.

George refers to a Honeycomb meta-incident writeup about a couple of back-to-back incidents Honeycomb experienced recently.

Additional Resources

• PagerDuty Home Page

• Honeycomb meta-incident writeup
• Pauline Boss, Navigating Loss Without Closure
• Blameless Culture
• Folks can ping George via Twitter. If they’d like to talk, He’d like to listen - @gmiranda23
• Episode transcribed by Rev

Lilia describes her favorite aspects of PagerDuty Summit.

Lilla: “The product keynote! It includes a poem from Rachel [Rachel Obstler, VP of Product Management] as well as a demo that she has been helping with for the past several years. It’s incredible to see all of the new exciting stuff that we’ve been working on all year make it to the big, now-virtual, screen.”

What Listeners Should Tune In For

Lilia discusses recommended talks and content at PagerDuty Summit - hint there’s a lot!

Lilia: “Don’t miss the product keynote! There are also some super exciting product break out sessions. I would strongly recommend going to Paul Rechsteiner’s session “PagerDuty Like a Pro: New Features for Responders” and Erick Dean’s session “Zen your life with Event Intelligence” – there are so many new things these two will be announcing that I don’t even know them all.”

First Time Summit Attendees

Lilia talks about how to navigate Summit as a first timer.

Lilia: Don’t miss the fireside chat with Eric Yuan from Zoom – he’s an incredibly authentic speaker, and I always feel like I’ve learned from his perspective when I hear him think out loud. Take advantage of the PDU classes – the Incident Commander training changed my life several years ago – and now not only am I an Incident Commander, but I’ve also trained a whole group of new folks at PD to be Incident Commanders, and I am so much more decisive in my everyday life.”

What Do You Wish You’d Known Earlier in Your Career?

Lilia discusses how she worked at companies who were PagerDuty customers for years before starting at PagerDuty. Something she really appreciated is the Incident Command structure. At those prior companies, she said she felt mostly that whenever there was a serious issue, she mainly felt unable to help take corrective action as that was left to the developers and operations teams. With PagerDuty’s Incident Command structure, non-engineering teams help resolve the process by taking off load that corresponds to their skillset, e.g. by sending communications, writing up reports, etc.

What Are You Glad We Didn’t Ask?

Lilia explains that her first event intelligence product release was very anxiety inducing. Ultimately successful, but the scope of potential impact for the two weeks leading to release were nerve wracking.

Additional Resources

• PagerDuty Home Page
• PagerDuty Summit
• PagerDuty University
• Episode transcribed by Rev

Introduction

John gives us an overview of what he and the other folks at Adaptive Capacity Labs are working on.

State of the Industry

John talks about the state of the industry around incident response. Learning from incidents is happening; but are organizations supporting it? Are people finding it helpful? Expertise is coming from inside the house, in that software practitioners are getting better at coping with the complexity of the systems. Where there is still work to do is around how teams learn from their incidents and postmortems. Are the artifacts generated by these exercises used after they are created, or do they just become a museum to the incidents?

Who are the Incident Nerds

There is an emerging community of folks who are really enthusiastic about learning from other organizations’ incidents, in software and across different kinds of industries. But many incident reports are still written to be filed rather than written to be read, and leave out some of the other aspects of an incident that are important. In addition to just whatever triggered the incident, there are other aspects to be learned from, like weighing potential fixes, or finding information.

Thinking about incident reports as a story, what elements make it a good story? What did the team struggle with? What was hard about it?

What are We Learning from Other Industries?

A number of “safety critical” domains have a longer history than software development with respect to dealing with incidents. Some domains have different constraints, challenges for gathering data, legal ramifications. What will the future look like, and can software development avoid some of those constraints.

How do all of these potential incidents impact not just the employees on the teams managing the incidents, but also the public, consumers, and how are they impacted by an outage? Are consumers able to make informed decisions about a company based on how incidents are handled?

As a learning exercise, do your new employees take the opportunity to read past incident reports and then ask questions?

Debunking a Myth

John deflects answering about Root Cause. You’ll just have to check Twitter.

An existing belief that leads people to a potentially incorrect outcome, is that an incident is seen by different people, with different perspectives, as different. There is not one true universal story that everyone will get from reading an incident analysis.

The goal shouldn’t be for just the person doing the analysis to understand what happened, but to also make future readers understand what happened.

What Do You Wish You’d Known Earlier in Your Career?

John talks about the practice of software engineering, and the certainty that things has changed and will change. Everything should be up for questioning of assumptions about what is the best way for something to be done.

What Are You Glad We Didn’t Ask?

John talks a bit about incident command frameworks and refers to Laura Maguire’s research on the costs of coordination and how the costs associated with robust incident response are easy to forget. Laura’s talk is linked in the Additional Resources below.

Additional Resources

• PagerDuty Home Page
• PagerDuty Summit
• John’s Talk from PagerDuty Summit 2018
• Read some incident reviews collected by the folks at Incident Labs
• Learning From Incidents
• Laura Maguire’s talk on the Costs of Coordination from QCon London 2020
• Episode transcribed by Rev

Matty kicks us off sharing a bit about his new role at RedHat. Working in the “NAPS” area of Digital Transformation, Matty discusses what digital transformation is.

Matty: “My main focus is I help government agencies; especially state, local, and education agencies in North America as they’re trying to transform how they work and basically do all this DevOps stuff that we keep talking about, and how do they kind of serve their mission better with technology.”

Digital Transformation

Matty talks to us about how there really is no right answer to what digital transformation is; it’s more about how you want it to be.

Matty: “This idea of transforming in a digital way, what does that really mean? Does that mean that to some people that means we’re moving in some kind of a more cloud native approach. Some people that just means they’re paying attention to tech, and technology is moving from being a cost center to being part of the business. And to some it just means we needed another C suite person so we can have a Chief Digital Officer, so we can put somebody else as an executive… So that’s why we had digital transformation. Some people think it means working remotely and Covid drove us to digital transformation.”

The Reality

Matty continues to talk to us about the reality of digital transformation being about transforming the way you work and how he likes to think about transformation over modernization. He also talks about the socio-technical system and how you need to think about the people in tech as well. Matty draws parallels between the government sector and the private sector and how you need to have adaptive capacity.

Matty: “the reality is, this is exactly why you need to work on it [digital transformation] in parallel with this adapting that you need to do. Because you have this brittleness that caused you to not have adaptive capacity and you need to have it so you can respond… This is not a thing you do for your entire organization at once.”

Matty talks about transformation as an interactive project instead of trying to tackle everything all at once.

The Five Elements of Transformation

Matty talks about the things to think about when you are talking about transformation and thinking beyond the squad level and scaling to a larger organization.

Matty: “We need to bring that conversation higher than just software development. And we talked about having five elements of transformation, and those five elements are: leadership, product, development, architecture, and then operations. So development is part of it, but usually a lot of times in IT when we’re thinking about transforming, we’re really hyper focused on just that one element of development. We’re thinking about agile, we’re thinking about how we do our software engineering. Each of these elements is really key across all of them together.”

The Sharp End Versus the Blunt End

Matty talks to us about driving an organization to be more open and driving decisions to the “sharp end” which is the folks who are actually doing the work.

Matty: “Think more product based versus project based. We think about having a culture around measurement so that we’re continually having feedback loops; projects end, products don’t… moving towards a product centric mentality is really key.”

Matty continues to talk about how to think about establishing a foundation of resilience, and to think about what the transformation actually means for the business and its teams and people.

The Necessity to have Resilience

Matty talks about what happened with the Texas Workforce Commission in relation to COVID how the systems weren’t built to be ready for COVID.

Matty: “These are systems that weren’t built to manage this flow of information and this strain… these things can happen and it won’t be COVID next time, it will be something else, something you don’t see coming, you can’t predict it. That’s why we need to have adaptive capacity so that we can flex and rebound.” Don’t be Afraid to Make the Changes that Need to be Made Matty and Julie talk about how the current situation is not going to end, so don’t treat it like the “big game” and how to deconstruct the business logic to make changes now.

Matty: “Your refactoring does not mean rip out the entire thing, we build bridges and it’s okay to have disposable systems that just give us along, right, we’re building a shim that will get us to that next place.”

Plans and Structure to Enable the People

Matty talks about how getting the people who are closest to the work are the ones you need to have a plan and structure around because they are the ones who are able to respond to the things you can’t predict. He also talks about aligning folks to the mission to empower folks to do things better.

Matty: “The goal, the thing we’re trying to accomplish to service our customer, to service our user, to service our constituents, and where that connects to, and they will do this if they’re empowered to do it.” Parting Advice Matty shares about empowering folks, and trusting your folks to deliver. This will give the outcomes and the agility to get things done.

Additional Resources:

• PagerDuty Home Page
• PagerDuty Summit
• Devopsdays Chicago
• Texas Workforce Commission Study
• Digital Transformation wrong answers with Matty
• Episode transcribed by Rev

J. Paul talks to us about being on the critical operations team at Netflix, what that has been like during the quarantine, and the pressure they all felt at Netflix to make sure the service is stable for their customers.

J. Paul: “What we were looking at real early on, is are we able to serve streams to our customers, are we able to provide those moments of joy?”

J. Paul continues to discuss the impact and coordination required to overcome technical challenges, and how looking at COVID as an incident helped with planning exercises.

J. Paul: “We started to shift our perspective, less around technology and systems and making sure they’re stable and all that, because we had some good evidence that things were going to be fine right at that point and we started looking more at the people impact.”

Socio-Technical Systems:

The conversation shifts to the impact of people being on-call and being required to work from home.

J. Paul: “We started to really look at, from the operations perspective, if that operations team was understaffed and underwater when COVID happened, now you’ve got a whole other set of problems to think about with that.”

He continues to talk about socio-technical thinking - how the socio part is really about the people in the system who are responsible for getting systems up and running and operating them.

The Impact of Operating Systems

J. Paul brings up the levels of impact to the people, beyond just the surface-level impacts of being at home.

J. Paul: “If you’re on an engineering team you are likely going to be on an on-call rotation for your team. So the core team will page you into an incident, where we use PagerDuty for that. And so one of the interesting things is that means we have a really large data set of what people are experiencing or what we’re seeing with paging rotations and that sort of thing. So we have a we’ve been starting to parse through that. We actually have a monthly kind of socio-technical systemic risk meeting, so we’ve started actually talking about the impacts of working from home.”

Time has no Meaning?

J. Paul moves on to discuss the difference between capacity and availability, and how people are the same as systems.

J. Paul: “We may be highly available or as available as people expect us to be, so we might be eight hours, you know, online in our home office or whatever the case may be. But people’s capacity is reduced during this because of the stress of COVID.”

The conversation around availability vs. capacity continues and J. Paul encourages us to give our team members more grace.

Changing the Way Teams are Built

Mandi and J. Paul talk about the biggest changes we see with remote conversations and the need to be onsite, as well as the value of being remote.

He then mentions ways distributed teams can increase the cost of managing incidents and how they combat this at Netflix by practicing and doing incident management on Slack. J. Paul continues to discuss the ways folks are changing the way they work due to a lack of in-person meetings.

Thinking with Stories

The conversation moves to a discussion around how humans think through stories and why stress levels are higher during incidents.

J. Paul introduces us to Jabe Bloom’s (@cyetain) research at IBM’s RedHat Global Transformation Office, and how humans process through stories that make sense. He explains how incidents that don’t follow the “stories” have broken down.

J. Paul: “And the reason that it’s stressful is because all of the inferences that we made about the future and the stories that basically reduce the cognitive load for us are not true, which means we have to pay attention in the moment. And the bandwidth to do that on our brain is incredibly high. We have to pay attention to every little detail because we can’t rely on the stories that were told to us about these systems anymore.”

Exit Criteria

J. Paul explains that Netflix couldn’t keep the COVID incident open forever, and how they needed to learn and become increasingly adaptive in the new environment.

J. Paul: “The requirement for that adaptive capacity has actually gone down right, because they figured out, but for other team they’re still having to be adaptive and innovative in the way that they do work, but they know that now, so they know what they need to do to keep that adaptive capacity level.”

Season 2

Just a reminder, if there is a series you want to beg Netflix to bring back J. Paul offered you the ability to tweet him @jpaulreed with your requests.

Additional Resources

• PagerDuty Home Page
• Episode transcribed by Rev

Brian Weber kicks the conversation off with an overview of Site Reliability Engineering (SRE).

Brian Weber: “I look for things that are outliers and efficiency that are health problems, not so much product health, but developmental health and try to impose those “standards” on my team. I find myself talking to other SREs, both in the company and out of the company to try and get an idea of what those kinds of things look like.”

Brian Rutkin gives us the wrong answer, discussing how SRE is more than just a cool title and how SRE is not DevOps.

Brian Rutkin: “Taking your engineers that are DevOps and suddenly rebranding them as SRE is not necessarily the right thing either. SRE kind of falls in toward the middle of those terms of how you would use them.”

Delving into DevOps and SRE

The Brians talk to us about how DevOps and SRE work together.

Brian Rutkin: “SRE is an implementation of DevOps…. SRE is the understanding that operational work is required, and the goal should be to remove absolutely as much of it as you possibly can by a human.”

Brian Weber counters with the misconception of how much software development someone with an SRE title should be doing. He continues to talk to us about applied implementation and researching components of SRE.

Metrics: SLOs and SLAs

We talk about setting and publishing Service Level Objectives (SLOs) and Service Level Agreements (SLAs), and best practices around setting and accomplishing these.

Brian Rutkin: “This is really going to vary for every organization and every service a least by a little bit. I think that most people would agree that you want to focus on a very few number of SLO’s to drive and accomplish your SLAs.”

Rutkin continues to talk to us about setting the metrics and what you want to know from your service; success rate, latency, and accuracy.

Brian Weber discusses the misconceptions about what a customer is, what a customer should be, and what people should be paying attention to with SLAs.

Weber: “Your SLA is the amount of uptime and availability… it has everything to do with what is your end state.”

Tuning Alerts

The conversation turns to creating and tuning alerts and what the role of SRE is within that area. Brian Weber talks about how to make noise levels appropriate for new products and how tuning alerts is an ongoing process.

Weber: “Tuning alerts ends up being an ongoing process, that’s why it’s tuning alerts and not setting alerts.”

Brain Rutkin walks us through antipatterns with alert tuning and how going to extremes is a common mistake that is made. He also talks about how alerts should be used for two purposes; immediate problems and to watch trends over time.

Sharing Learnings

The Brians discuss how to share learnings across the organization.

Rutkin goes over how you can determine the correct ways to communicate within your organization and define thresholds for when a postmortems is required.

Brain Weber illuminates where many postmortems go wrong, “Blame, blame, blame.” He continues to discuss blame as a main reason postmortems go wrong. Weber continues to talk about using the right words in the postmortems and how it’s rarely a single person that is the cause of a problem.

Both Brians discuss the “5 Whys” of how you can prevent future outages through systems and culture changes.

Additional Resources

• PagerDuty Home Page
• PagerDuty’s Postmortem Resources
• Episode transcribed by Rev

Starting the Zine

Emil and Jaime met while working at Shopify and bonded over rock climbing. And while reading a climbing accident book they were inspired by the lessons shared and instantly made the connection to how those accidents related to software incidents.

“When we think of post-incidents in companies…you always think of this very large document,” Emil says. “You think of who was involved, timelines, descriptions, stakeholders, etc. With the book, it ran the whole range. If it was a report from park rangers, they could go into very detailed minute-by-minute exactly what happened. And then, if it was a self-submitted report it could be something like one paragraph, two people were climbing, a rock fell loose, and they fell to the ground. That was the whole description.”

In addition to these accounts the book would also have best practices and considerations to think about, or first aid techniques to keep in mind for similar situations.

Emil talks about how these reports were a catalyst for their idea to catalog and share the post-incidents that were incident to them. Many have to do with running software in production, but some don’t.

“You might think that is a bit grim,” Jaime says, “because you’re reading this book about how this thing happened to this person. And this feels weird to be so excited about all these bad things that happen. But, it’s important to have that learning. Someone already had to learn this lesson. If you don’t share that across the community, then there’s a real possibility that someone else will fall into this same trap.”

Myths about Post-Incident Reviews

Jaime talks about the myth that if you write a post-incident review that people will read it because it’s good for them. He says, “I’m right and there’s a lot of good lessons there, so when I put it out there, obviously people will take the time to read it…That’s just not the way it works. We’re all so busy. We’re all so overwhelmed. You can’t just write the post mortem. You have to think about how the post mortem is going to be used.”

“That’s why when we printed the zine. We were saying, ‘hey, we put in a lot of effort, now just carry this with you! When you have time it’s right here for you! We made it easy for you to read and absorb and enjoy, so why not?!’”

The key point, that Emil says, is that you need to have a culture of reading the reports. But, also you need to be writing reports that are fun to read.

Goals for the Zine

This really started as a passion project as Jaime talks about how they just wanted people to talk about these topics and more companies to release these post-incident reports.

“Unfortunately we don’t live in a culture where admitting mistakes, or errors, actually makes us look stronger…But, you can’t learn if you’re always pretending to be perfect.”

After they printed their first issue they took 200 copies to a conference to distribute.

“The reception was so lovely! We started having people come find us and were like, ‘I heard you have a zine and I really, really want a copy’,” Jaime says.

Emil describes the reaction of those who saw their incidents in the zine. “The sheer joy of seeing some of the people at the conference–we put their incidents in the first issue–seeing their sheer joy of seeing their blood, sweat and tears on paper, that was the best feeling ever!”

Common Threads in Incidents

Jaime: “The biggest one, while we’re reading them, it’s clear that different authors have different intentions. The ones that are most interesting are the ones who have built a story.”

“Incidents aren’t solved just by machines. They’re solved by the people working on the machines.”

“Include the people in the narratives. Interview them. Really understand how things worked.”

Common Keys to Success in Resolving Incidents

Emil talks about their favorite incident which is the story of Apollo 12. On it’s way to the moon, the rocket was struck by lightning 30 seconds after lift-off.

“When that happened all the controls inside the rocket went haywire…they had every single alarm going off. They’re trying to decide in this moment if they need to abort the mission. And, at one point, one of the controllers in Houston said to switch SCE to Aux…and, by chance, one of the astronauts knew which switch he was talking about. And, when he switched it everything went back online.

“Why both Jaime and I love this story is because the controller at NASA…”

“John Aaron,” says Jaime.

“John Aaron! John Aaron knew to make that call because a year earlier in the simulator he was playing around and out of curiosity he got the simulator in a really weird state. And, he tried to get out of it, rather than resetting the simulator. And that curiosity led him to learn that if he turned SCE to Aux it would return the system to a normal state. As a consequence of knowing off the bat how to fix that he earned the name, ‘steely-eyed missile man’.”

Other projects

Jaime explains, “we’ve been working on a product (Ovvy) that will help teams remove bad alerts…because then that frees up your time to work on the things you actually want to do.”

“It frees your time up to read Post-Incident Review zines!!” Emil adds.

Recurring Questions

Emil says, “[the] one thing I would have liked to have learned sooner [about running software in production] is that it’s about the people. Very really is it the technology that will tell you whether or not you’ll have a reliable system.”

Jaime is glad that we didn’t ask, “show we do what Google does for SRE?” To which he says, do I have to be Beyonce to be successful?

Additional Resources

• PagerDuty Home Page

• https://zine.incidentlabs.io/
• https://incidentlabs.io/
• https://ovvy.io/
• https://americanalpineclub.myshopify.com/collections/aac-publications
• http://publications.americanalpineclub.org/
• Episode transcribed by Rev

Application Monitoring:

Erik talks about his high-level strategy for monitoring for folks that are new to the practice. He talks about things you may want to think about such as: How long it takes to load a webpage, breaking down website requests into different categories, measurements, and SLA’s.

Erik: “Another aspect might also be monitoring things like database contention. Maybe you’ve written some really amazing application code and it’s just hammering your database. So you have contention on your database or you need to have read replicas or all kinds of other things to help cash your responses so you can have fast response times but a lot of the things that are monitoring in general are going to be around web performance and kind of the full stack between someone that can request and response back.”

Debunking a Myth

Erik debunks the myth that when big companies started they had everything at their fingertips, when in reality startups start with very minimal footprints and are doing as much as they can from a single server.

Erk: “For instance, you know, we’d have a single server, we’d add a bunch of users. At some point we’d start getting to a point where your databases are overloaded, there’s too much contention, your app servers are falling over, you’re up all night and weekend kind of doing all that provisioning and scalability. But initially you think oh, like the startups just have instant scalability and they’re never going to fail, but the reality is that angel investors only give you a couple hundred thousand dollars”

Making things easier in the “wild west”

Erik chats about how to make things easier in startups and environments with limited resources, and choosing the right resources.

Erik: “Choosing the right technology and the right software and programming languages and scaling technology can really go a long way to reducing how much pain and anguish you go through while you are scaling your company.”

Erik goes on to share ways to invest in your company learnings, by investing in the right people and reinvesting profits back into the people. He talks to us about how everyone in a startup is operations, how teams work together, and how working together with the development teams leads to ultimate success.

Monitoring and Observability

Erik shares how he thinks about observability and what that looks like coast-to-coast. With a background in gaming, he discusses the need to have monitoring in place and how standard deviation and observability in your services are hard things to solve.

Erik: “It’s pretty important to have these tools, and I think that where we’re at currently, and where we’re headed in the future in terms of observability monitoring with various tools that are out there right now that tie into PagerDuty are allowing us to have much more sleep at night than they ever were before.”

Establishing Monitoring Based on Standard Deviation

Erik talks about how standard deviation is the main concept behind monitoring and how to differentiate between normal deviations because of expected work that occurs in the middle of the night.

Erik: “So you’ll end up saying, hey, there’s no traffic between 12 and one, but at one we run all these cron jobs and we do a lot of DB queries or something like that. Or, you know, try to purge records or do some kind of background task asynchronously, and so that’s a normal deviation, don’t worry about that. But then once they were doing those cron jobs and then somebody, a hacker from New Zealand comes in, spams our API’s with a DDOS attack. We still want to be alerted for that even though we expect it to be high, it’s way abnormally high from what we expected. So that’s the main concept behind any monitoring and standard deviation is figuring out, it’s okay to have a huge spike in the middle of the night because we expect it because it’s self-induced.”

Changing How We Think

Erik discusses how being an engineering manager changes the way he thinks, like using code freezes during certain windows, like Black Fridays, and building in robustness when you are planning your software deploys.

Erik: “You typically want to have a plan to either rollback your code changes or fix it forward, which is a really bad thing to do unless you have to do something like that. So imagine you deploy some really bad code and it was because you changed an if-then statement to have a greater-than versus a less-than symbol. So what do you do, do you really back your t deploy or do you change the greater-than or less-than symbol and deploy that as a fixed forward? You have to make those calls all the time”.

Driving a Culture of Accountability

Learning from mistakes is a major driver of a culture of accountability, Erik talks about empowering folks to learn from their mistakes and using postmortems to learn from failures.

Erik: “We have to mitigate risk as best as possible. And there’s only one way to really learn, and it’s by messing up”.

Additionally, Erik chats about how good software design and forethought go extremely far and how thinking through what you are supposed to do and communicating can go a long way.

Erik: “You’ve got speed, quality, or stability, you can have two but not three.”

He goes on to share how to think through those trade-offs and make business decisions with the least amount of risk.

Additional Resources

• PagerDuty Home Page

• Bleacher Report Ruby vs Elixir
• Extreme Programming Explained
• Episode transcribed by Rev

Austin uses the example of a platform team managing a large portfolio of applications that all need to share a common layer, like SSO. Sounds simple, but complexity only gets shifted around depending on your approach.

“The role of this platform team is to impose some simplicity onto this complex design that has been foisted upon them. We started coming up with some very complicated solutions to how to do different tasks for different systems. Until we stepped back and instead focused on what these systems had in common. Once we reframed into looking at the similarities instead of the differences, we were able to make a lot of progress very quickly.”

Simple and easy are not the same thing

Simplicity is being able to reason about what is happening, even if what is happening is a very difficult and involved process. Just because something is easy, that doesn’t mean it’s simple. Just because something is simple, that doesn’t mean it’s not difficult to do. Simplicity can be very hard to implement when you see it that way.

“Breathing sounds very simple, but it’s not very easy if you think about what is happening in your body to make you breathe. It’s an incredibly complex series of physical biological systems: I’m not a biologist or doctor, so I don’t even understand them all.”

Are abstractions actually making our lives better?

The microservices vs. monolith debate keeps popping up. Austin argues that microservices were never about managing technical complexity, they were about managing organizational complexity. The abstractions for microservice platforms are managing our organizational problems but at a high technical cost. It’s how we try overcoming that cost that might be making lives more difficult.

“On the technical side, that now means you have to be able to have a deeper understanding of your software: you have to be able to introspect the behavior of your application at every single point that it is used … At some point, we’ve covered it with thousands of probes … so much so that no one can even step back to see the application anymore, because we’ve lost sight of what it actually is.”

Finding the signal in the noise

People are at the heart of these problems because they’re the ones who need to constantly re-evaluate system state. Austin makes the point that people usually fail to do that hard part of system design because it’s seen as a Day 2 problem.

“We make decisions about system normalcy at the tail end of design, using what we’ve known up to that point. Most people aren’t thinking about how they write logging statements to understand application performance at that point.”

It always comes back to Sydney Dekker

We talk about “The Field Guide to Understanding Human Error” for the first time on this show! How did that take so long??? When it comes to understanding systems, it’s better to be clear than to be concise.

“The designers of an airplane cockpit changed the airspeed indicator from a round dial to a linear tape, figuring that it would be easier to quickly read airspeed if your target was in the middle of the tape… What they found was that it actually made it harder to understand and it took cycles away from pilots and added more response time whenever things went wrong.”

Simplicity is hard

Experience is thinking about how humans interpret signals. It’s important to know when to be clever and complex and when it’s better to optimize for reasonability. Austin gives some practical tips to simplify choices when it comes to designing simpler systems.

“While simplicity and easy aren’t necessarily connected, simple and good kind of are.”

Recurring Questions

Austin wishes he would have known how to write good logs sooner. Very rarely do people stop to consider how to make metrics useful to other people. It’s less about how than why.

Austin is also glad we didn’t ask him about running production software on Windows based systems. The problem wasn’t Windows, it was dealing with heterogeneity.

References

• PagerDuty Home Page
• The Field Guide to Understanding Human Error - Sydney Dekker
• The Open Telemetry Project
• Lightstep Sandbox
• Your System is Deeper Than You Think - Austin Parker
• Episode transcribed by Rev

Sumbry, Head of Reliability Engineering at AirBnb, and George discuss tips for the current challenge many of us in the software industry are facing: how we communicate with our coworkers remotely. We consider lessons learned from managing globally distributed teams and compare those to what we’re learning now due to the COVID-19 pandemic.

Lessons from managing globally distributed teams

Having managed engineering teams at Twilio, Uber, and now AirBnb, Sumbry shares a bit about the timezone madness from being globally distributed. Some teams orient their schedules around the timezone at their “default” location. That doesn’t do the team in the default location any favors when it comes to thinking about their remote peers.

“When most of the people on the same video conference are in the same room, there’s an ebb and flow to the conversation. Add in a couple of remote folks and they feel like they’re interrupting the conversation when they interject.”

Remote work vs. working from home

George overlooks Sumbry’s past experience being a solo entrepreneur working from his living room. They discuss the differences between “working from home” and being a distributed team that needs to communicate, aka “remote work.”

“When there’s a center of gravity around a particular physical location, it’s very easy to be excluded if you’re not physically there.”

Debunking a myth about working remotely

You gotta wear pants, people. Or rather, it’s tough to work from home in your pajamas. Working from home doesn’t mean ignoring your usual routine.

“Staying in my pajamas all day was the key signal that my work days and my overall well-being were suffering.”

Building relationships with remote teams when you’re new

Sumby just started his new role at AirBnb. Many companies with distributed teams fly new teammates into a central office for onboarding because they think it’s easier to build relationships in person than it is over video conferencing. But not meeting someone can have unexpected results.

“We started our interactions over video conferencing and got to know each other. When we finally met in person, I realized they were about 7 feet tall. I didn’t notice until then how my perceptions and how my interactions might have changed if we’d started out getting to know each other in person.”

Making clear rules of engagement

Remote work means turning communication into documentation. It helps to document team “norms” so that anyone can easily get up to speed. Make it clear how much communication over which mediums are expected because everything needs to be more explicit when you’re distributed.

“Correctly set up expectations, develop this routine and cadence, and it helps a lot.”

Distractions when working from home

The reality is that when working from home, our personal lives can sometimes work their way into our workdays and distractions can appear unprofessional. But should they?

“You need those rules of engagement not just with your teams, but within your household as well.”

Productivity for remote teams

People often report being able to get more done from home, but that can also have unexpected consequences.

“After we saw a 30% increase in deployments, we sent out a survey to our engineers. 70% of them reported feeling like they were able to get more done, but those same 70% also reported lower energy levels.”

Tips for creating social connection remotely

Sumbry and George discuss tips & tricks that have worked for their teams during the COVID-19 pandemic.

“We did our own version of MTV Cribs. People would take their laptops and show off their homes and working areas. Seeing people’s homes instantly changed our levels of interaction and familiarity.”

Recurring Questions

Sumbry cleverly calls out the need for simpler production systems and wishes he’d learned to appreciate a lack of complexity sooner in his career.

He also becomes the first guest to plead the 5th amendment.

References

• PagerDuty Home Page

• Remote engagement recommendations

  • Sticking around after a meeting ends to just chat
  • Virtual talent shows (Sumbry swears by that)
  • Virtual tours of your home and workspace

• An example of documenting rules of engagement.

• Sumby on Twitter - https://twitter.com/sumbry

• Episode transcribed by Rev

Alex Solomon (CTO and Co-Founder of PagerDuty) kicks us off with a definition of real-time operations and why it matters.

Alex: “Real-time operations to me, what that means is, it’s about dealing with problems and incidents and alerts in real-time. Making sure that the right people are pulled in whenever you have an issue with your production software, and only the right people. Those teams and individuals are looped in quickly, looped in via multiple channels to make sure they get there fast. Then once they are paged and looped in it’s about collaborations, it’s about communication, it’s about coordinations, it’s about defining clear roles for all the individuals and making sure they can collaborate and communicate effectively to make decisions quickly and resolve the underlying problems with those systems.”

Matt hops in to discuss that real-time operations also encompasses how we learn about incidents and how we continue to learn from them.

George talks about how real-time operations extends to every facet of online operations that might impact our team, whether it’s web services or code we write and how it operates in production, and how the definition of real-time operations is very broad.

The Myth of Real-Time Operations

Alex talks about the main myth he sees with real-time operations.

Alex: “The myth that you can buy a software platform like a PagerDuty or a DataDog or a New Relic or any of these toolboxes that we all have when running digital systems, and that buying the platforms will solve all your problems and be a silver bullet. In my experience what I see over and over is that yes you can buy the platform but the hard part is changing culture and transforming culture and transforming the way people work, and that comes down to people and process.”

Alex goes on to mention that it’s about the people supporting the services and full-service ownership.

Matt talks about the myth that we can prevent failure.

Matt: “The reality is we can do a lot to kind of steady ourselves and be ready to respond and take information we’ve already had, but our systems are so complex there’s no way to be fully predictive, and we need to understand how to make our systems - our socio-technical systems - more resilient rather than thinking if we just build in enough failover, enough automation, or write the best runbook ever, will be able to prevent failure.”

The discussion moves towards how systems are designed for failure, and that we have ways to detect problems and rectify them quickly so we can detect and resolve problems quickly.

Sharing What We Have Learned at PagerDuty

The conversation moves to what we have each learned during our collective time at PagerDuty, whether it is the incident response process or postmortems.

Scott talks about how his time at PagerDuty has been entirely remote and how to be successful as a remote worker by being vocal about your wins, taking time for yourself and helping others learn about what you are doing by being an internal advocate.

George mentions that advocating internally and externally is about how you communicate with different folks that are distributed.

Julie discusses her experience with this being her first remote job and how the PagerDuty culture of having video on all the time makes being remote much easier by helping to build a great team relationship.

The Shift to Remote Work

The conversation shifts to how real-time operations are impacted by the shift to remote work.

Alex discusses how in the last 20-30 years it was about data centers and folks being on-site, but with remote tools companies have the ability to move to remote easier. However, the challenge and gap can be the culture of remote work if teams and companies aren’t used to that experience.

Julie talks about what it is like to work remotely with families in our homes while we work. She mentions how she packs her son a lunch like she would have if he was physically going to school.

Matt offers his story of how he has trained his kids to understand that he is working when he is at home.

Matt: “What I used to do is I used to wear a special baseball hat if I was going to be in the main room and it was like, if daddy had that hat on he was working, and for all practical purposes he was invisible, and that worked about half the time.”

Matt continues to talk about how we can be empathetic towards our co-workers and get to know them a little better.

Julie shares the biggest learning for her at PagerDuty is that:

Julie: “Every organization feels they have a very unique story to tell, but it’s not as unique as they may think. A lot of these organizations, they may have a different journey but they are still on kind of the same level as to what they deal with.”

Julie goes on to talk about how organizations are dealing with a lot of HybridOps situations.

George hops in to discuss how his background as a first responder applies to managing real-time operations:

George: “A lot of that comes down to preparedness, to having a plan, to knowing what you are going to do when those unexpected surprises come up.”

George continues to say you cannot plan for everything, such as COVID-19, but you can have repetition and practice around when a type of crisis occurs.

George: “Having a plan is not about following that plan to the letter, because we never know what we are going to expect. Real-time operations is completely unpredictable, but what is important is just knowing how you might approach a situation like something we can reasonably infer.”

The hosts talk about how practicing everything helps with times of uncertainty.

HybridOps

Alex shifts to discussing how HybridOps has been a big learning over his 11 years of building PagerDuty. He talks about how early on a lot of the customers were digital natives and cloud-first, and how they helped us in developing our product and vision early on. Alex mentions how HybridOps comes into play as some of these organizations have both legacy systems and newer digital systems, they also have central operations and teams that are DevOps oriented that build and run and maintain their own systems.

Alex: “That’s what HybridOps is all about, it is the situation that these companies are in, that they need to operate in both modes at the same time, while working on modernizing their older applications.”

Wrap Up

The episode wraps up by asking Matt the final two questions on his last Page it to the Limit episode.

Matt talks about how for the majority of his career he felt like his job was to defend production from DevOps but how that changed when he got into the DevOps mindset and changed his perception.

Matt closes by pointing out that he is really happy that in all the time he has worked for Alex he has never been asked to do anything with regular expressions.

Additional Resources

• PagerDuty Home Page
• Episode transcribed by Rev

Leandro and George discuss a few of the practical considerations that teams need to tackle when moving to a model where teams own the code they ship to production.

Building Ownership Teams

We discuss the transition from just writing software code to empowering teams to own more of that lifecycle. Leandro tells us a bit about how his company, MessageBird, has approached this problem.

“It’s giving ownership to both functional and non-functional aspects of an application to a single team: functional aspects being something like delivering product features to a customer, and non-functional being aspects like availability, performance, and security.”

But developers don’t want to be on-call

Leandro debunks two popular myths about going on-call.

“You hear a lot of people saying that developers don’t want to be on-call. You’d be surprised how many people step up to and take that responsibility when you actually empower them to fully own their services. People will take that responsibility and go the extra mile.”

Practical ways of removing roadblocks

As teams shift to a full-service ownership model, one of the biggest challenges is shifting roles and responsibilities in a way where things don’t fall through the cracks. How exactly is that done when people have a finite amount of bandwidth and knowledge? Do you hire more people? Do priorities just change?

“We still have a platform team that is more focused on [infrastructure]. But we also invested a lot in training. So instead of having a skills gap, our teams have an intersection with that [platform] knowledge.

Unexpected lessons learned

We talk about both the challenges and the things that became much easier when MessageBird transitioned to a full-service ownership model.

“What makes it easier is the closeness you have to [customers] and how easy it is to get to the bottom of issues with your service. It brings people closer to the punchline of solving problems.”

Incident alerting

We discuss how this shift in thinking is being driven and reflected by how MessageBird manages their approach to incident alerting.

“In the past we structured ‘services’ around escalation policies and team structures… Now our services are at the center. Now our on-call rotations and escalations are set up around who actually owns that service, instead of reflecting our organizational structures.”

Ops in my Dev or Dev in my Ops

We talk about the fear most teams have that software engineers can’t own running code in production. The first wave of DevOps seemed more focused on having Ops engineers learn to code. The second wave we’re in now, is about Devs learning how to Ops.

“What I always say is, honestly, software engineers have a great foundation for what they need in order to be successful when running software in production.”

Recurring Questions

George and Leandro both get real about the mistakes we made early in our careers.

References

• PagerDuty Home Page
• Full-Service Ownership Blog
• Let’s Talk Full-Service Ownership
• MessageBird’s Slack Integration Commands for Incident Workflow - https://github.com/messagebird/slackincident
• Full-service Ownership Ops Guide - https://ownership.pagerduty.com
• Leandro Pinto on Twitter - https://twitter.com/leandroshp
• Episode transcribed by Rev

Julie and Jason talk about how COVID-19 is affecting the conference industry and how we are adapting our work.

Jason Yee: “There are other implications of how do you operate as a company when you are impacted by outside forces such as viruses and outbreaks, and what does mean for things like disaster recovery and resiliency and not just for your systems but for your people systems.”

The conversation turns to how things are moving to online options and remote work.

Failure of Imagination

Jason and Julie talk about how you imagine what failure looks like.

Jason Yee: “Failure of imagination means that we often fail to think about ways that things can break, and in hindsight they look fairly obvious.”

Jason goes on to talk about how do you think about failure and how do you imagine what failure states look like. Jason and Julie talk about how PagerDuty tests for failure through Failure Fridays, and how we can translate learnings into less technical learnings. Chaos Engineering with People

Jason talks about the practice of chaos engineering with people, and ensuring that knowledge is distributed.

Jason Yee: “If we actually spend time and imagine what our processes would look like by sort of messing with the people [vacation and schedules] in it we could probably come up with some more interesting ones as well.”

Julie and Jason talk about practicing for failures and disasters and how practicing leads to comfort and the reduction of chaos in actual emergencies and incidents.

Early Signals

Jason and Julie talk about what we can learn from our systems and the past and how we can learn from those things moving forward. Jason talks about the three categories: Work Metrics, Resource Metrics, and Events and how early indicators feed into larger objectives.

Jason Yee: “What are those early things that I can monitor and take a look at that contribute to the overall objectives, and if I can monitor those indicators and get advanced warning on those to see if something is potentially wrong, then I could potentially head off issues before I violate my objective or agreements.”

The Right Way to Wash Your Hands

Jason and Julie talk about how proper handwashing is 20 seconds and the songs you can sing while doing this, but more so how it’s about practicing doing things the right way so that when you are in an emergency situation you don’t have to retrain bad behaviours.

Jason Yee: “Practicing correctly should be the same process as what you do in real life, it shouldn’t just be a response to like “oh now we’re going to do a different process because it is a real virus” or “now were going to do a different process because our critical systems are really down” versus what we are doing when we practice chaos engineering.”

The Right Methodologies

Jason talks about the methodologies behind chaos engineering.

Jason Yee: “In terms of the systems we build the methodologies really come down to, when it comes to chaos engineering; make that practice rigorous, come up with a good hypothesis, be rigorous about how you test that.”

Jason continues talk about how you need to test in a scientific and repeatable way, and how you need to do things in the same ways to have the same effect when you are testing.

Chaos Engineering Doesn’t Have to be Scary

Julie and Jason talk about concerns organizations can have around chaos engineering, and how chaos engineering doesn’t have to be scary when you implement the right methodologies.

Jason Yee: “With Failure Fridays and chaos engineering, you want to start small. So you want to start in your development environment, and with little bits of your components.”

Jason gives us advice on how to build up to staging environments and to production with chaos engineering.

Additional Resources

• PagerDuty Home Page
• U.S.: US Department of State - Travel Advisories
• U.K.: Gov.UK - Foreign Travel Advice
• Australia: Smartraveller.gov - Destinations
• Canada: Government of Canada - Travel Advice & Advisories
• Episode transcribed by Rev

Corey discusses how he became a cloud influencer, what that means for his credibility, and how people now listen to him.

Corey: “I have been a fan of PagerDuty for a long time, largely because I remember a time working on systems before PagerDuty existed.”

Corey discusses why waking the right people up at the right time makes it much less of a “Call of Duty” situation.

Business Hours and Business Outcomes

Corey wanted a problem that was strictly business hours only, Julie and Corey discuss how it all ties together with being on call.

Corey: “One of the interesting pieces that we see around all of those different business functions is that in most companies, engineering is usually the only group that needs to be awake at two in the morning that needs to deal with an emergent event.”

Corey goes on to explain that there are exceptions to this rule, and how businesses should staff accordingly if it is a 24 / 7 critical system, and how empowering folks to fix things that are broken is the start of cultural change.

The Cost

Corey: “Everytime you wake someone up, you can contextualize it as pushing someone one step closer to rage quitting.”

Corey talks about looking back and removing the noise by understanding and analyzing what is alerting, and knowing when to wake people up.

Debunking a Myth

Corey debunks the myth that is the “central conceit of AWS billing”.

Corey: “People talk about how the bill is too expensive… The pain as articulated by the business, is that someone wound up receiving a bill from Amazon that was 20% higher this month than last month… contextualizing what the Amazon bill is for someone that does not work with technology is not exactly a straight shot.”

Corey goes on to tell us what kinds of questions to ask when the bill is 20% higher, and how tying the AWS bill into PagerDuty is a “DON’T DO THAT” situation, due to how billing works at AWS, as the bill is always a trailing function which requires more analysis than waking someone up at 3am.

Advice from Corey: “I would be fine with that going to a slack channel during business hours.”

Don’t Train People to Ignore Alerts

Corey and Julie talk about how people get trained to ignore alerts, either through distribution lists or API quota limits.

Corey: “If something blows up and this is hitting your email, surprise suddenly you’re not getting actual important emails for a while until everything stabilizes.”

Corey talks about how alerts that wake people up should be emergencies, things that are actually damaging to the business if they don’t get fixed… with exceptions. Corey goes on to talk about how peoples patience can get eroded and terrible patterns can emerge.

Do Something About it, Owning your Code and Services

Corey talks about the nuanced story behind why things don’t get fixed, and why it becomes a challenge to tune services.

Corey: “What made it terrible [the on-call rotation] was that the VP in charge of engineering was a big fan of a software package called Big Brother… this VP was also a fan of AWS… the answer that the braintrust came up with was to have three of them running simultaneously, so when you get paged you don’t have to worry unless you get paged by all three of then and then you know it’s real.”

Corey about how driving cultural transformation does require executive sponsorship and that buy-in will get you everywhere.

Dogmas

Julie and Corey discuss ingrained dogmas and Corey talks about how we try to reinvent everything, and how that isn’t necessarily.

Corey: “There’s this ongoing narrative within this space of if you are not reinventing everything you are doing then there’s something awful that’s happening.”

Corey goes on to discuss how you don’t have to throw everything out you have done to be successful.

The Greatest Cloud Influencer in the World

From suits to personality, Corey talks about what he wishes he knew years ago. Corey talks about owning the “thing that makes you special”, and finding your niche.

Additional Resources

• PagerDuty Home Page
• Episode transcribed by Rev

George introduces PagerDuty’s Ops Guides: process (not product) manuals that help you figure out how to introduce leading-edge operational models to your teams.

“Ops Guides are something I think are fairly unique for our industry. They’re sorta like a whitepaper, but they’re actually useful. They’re a step-by-step process guide. They’re vendor agnostic. And they’re free open-source frameworks designed to walk you through how to accomplish newer–more modern–types of operational tasks.”

What is a Full-Service Ownership model?

Scott proves he read the guide by describing the concept behind the name.

“It’s when those who build something are responsible for supporting it all through every single stage of its lifecycle. It brings developers and product closer to their customers and it leads you to writing better software.”

Myth Debunking

George mimics an old man yelling at a cloud: this won’t work in my org! Scott takes on software engineers who think being on the hook to support production software means that they’re responsible for fixing everything. George then references an XKCD post to pick on his favorite pet peeve: thinking that running software in production won’t make you a better developer.

“Take this Dev and shove it up your Ops!”

How an Ops Guide works

George discusses how a process guide works. It’s not a mechanical set of steps to follow: it’s a primer that frames the problem and the goal in a specific way so that you can then uncover your specific challenges and make a plan that’s right for you.

“How do we take this audacious marathon goal and figure out how to break it down into steps? When it comes to doing that, the idea isn’t to follow a flow chart—step A, then B, then C—it’s more that you have to think about your organization and your stack in certain ways, and in these vague orders, and that’s what helps you make a real plan for your team vs. this generic thing we’ve written,”

Functional Team Roles

Scott cracks open a discussion on how the framework approach to defining this new pattern might pan out when going between organizations.

“If you use titles with so many presumptions baked into them, you don’t entirely step back to really examine where a function really lives. We shouldn’t put job titles next to that. We should tell you this is what the function should be. We’re just going to describe all the things and then you can decide how to assemble those cross-functional teams inside your own org.”

How to make an Ops Guide your own

We cover how to plug into discussions about the content via Github and where to find the web version.

Brownfield takes on greenfield

A discussion unfolds around why a step-by-step guide to restructuring how teams own their services in production is better for brownfield teams.

“If you have a big monolith—not just a monolithic code base, but a way of operating that’s monolithic in nature—and you want to start breaking that down into smaller components, then this guide is particularly useful.”

Pagey’s Picks

We debut a new show segment: our picks for stuff we’re all about right now.

George

• The Good Place on NBC just wrapped up. Go binge that show now.
• The secret off-menu Verbena cocktail at the Chandelier Bar in Las Vegas

Scott

• The Nickel Boys by Corson Whitehead, a fantastic book with a twist at the end
• Disc Golf, Scott’s new therapeutic hobby. Go try it today!
• Inventions a rediscovered collaboration album between Matthew Cooper of Eluvium and Mark T. Smith of Explosions in the Sky

Show wrap up

Additional Resources

• PagerDuty Home Page
• Episode transcribed by Rev

Sarai talks to us about how security is about assessing risk and deciding what you care about in your systems, what you want to protect, and what you want to do about these risks. Bea hops in to describe security as being about managing risk, and harm reduction as you can’t defend everything equally.

“You can’t secure everything, not everything in your company is the most important thing”

Debunking a Security Myth

In debunking a common myth in security, Bea talks to us about how there isn’t always one solution for everyone:

“That there’s one solution that works for everyone, that what is a big risk for me for my product, for my software is going to be a risk for every software, the threats that I have to care about the things that I want to protect are completely different than the threats that you may want to protect and the things that you care about.”

Values of the PagerDuty Security Team

Bea and Sarai talk about the values on the PagerDuty security team and why the team has those values.

Sarai talks about how at previous companies, there was naming and shaming of folks who failed things like phishing tests, and how at PagerDuty “we try to make it easy to bring in other people, never hesitate to escalate”.

You Can’t Prevent Phishing, But…

Sarai talks to us about how while you can’t prevent phishing, there are ways to help people use their emails in a secure way without trying to trick employees so you can catch them and call them out.

Instead of creating a negative experience Sarai tells us that it’s ok to fail.

Bea talks to us about how it’s not the fault of specific people to accept risk instead: “Present this information in a secure way, and not chastise them for doing their job”, instead as a security team you need to update what you’re doing to enable folks to do their jobs.

How do you Impart your Values on the Rest of the Company

Sarai and Bea talk about how it is important to make security training fun for the participants, and how through the use of lockpicking you can “find things that people can use so they understand that security is about defense and depth”.

They go on to share with us how Security has a collective responsibility to be collaborative.

Bea expands on why lockpicking is a great example of the vulnerability of security but is part of a “layer of security”.

How the Security Team Tackles Incident Response

We talk about how PagerDuty’s Security team has a little bit of a different take on incident response because it is a mixture of traditional security response which is responding to attacker or breaches, and how tuning alerting also comes into play.

Sarai: “It’s this cycle of tuning our alerts as we add more and more monitors”. Julie: “…and tuning is so important because eliminating the noise, reducing the noise; helps with alert fatigue, it makes life better for your engineers”.

Reducing the Noise, Getting Rid of Stuff

Bea talks to us about how to reduce the noise, and how if you can get rid of unnecessary services you have less to secure.

Bea: “Step one, in buying an IDS, don’t; if you do by mistake, delete all the rules and then slowly add them in because every IDS rule set has way too many things to actually be useful”. Bea goes on to say how alerts should have outcomes and be high quality.

Quick Wins

Sarai discusses how quick wins are great and how there are so many risks out there, so let’s look at one particular process. “Start with basic things and the easiest things to implement”. By starting with something quick and easy we get that quick wins.

Bea talks about how having a security win not be perfect is better than pretending that you can get to 100%.

What is the one thing you wish you would have known sooner

Sarai tells us how she wished she realized that she has a lower risk tolerance than other security and software people in general and that she should: “Just go for it, go try something, see if it works, see what happens if I break something. As long as I have a plan to go backwards then might as well try it”.

Bea brings us the closing wisdom of “stop worrying about your crypto because that’s probably not where you are going to get owned in a security sense”, and Sarai says: “Cryptography problems are actually social problems… it depends on your use case”.

Additional Resources

• PagerDuty Home Page
• Handmade security at Etsy
• PagerDuty’s Security Training
• Episode transcribed by Rev

What is Terraform

Robbie Th’ng and Paul Hinze discuss how Terraform provisions infrastructure as well as external development tools.

“To most people, Terraform is the best way to provision infrastructure. That was true for a fairly long time, but I think it’s now the best way to provision external development tools as well as the infrastructure.” - Robbie

Common Myths and Misconceptions Surrounding Terraform

Terraform is a tool that addresses an abstract problem that usually requires trying it out before really “getting it”.

“A lot of those common misconceptions are not having it all click in your brain. For those folks–trying it out–you really get that ‘ah ha’ moment.” - Paul

Robbie talks about how Terraform isn’t always the best tool to use. He said, “people often think that we only think Terraform is the way to do things, and that is definitely not true.”

Both Paul and Robbie talk about the HashiCorp philosophy of building tools.

Common Stumbling Blocks While Learning Terraform

The adoption of Terraform usually travels along the same progression. First, you digest the concept of Infrastructure as Code (IaC). Then, there’s the HashiCorp Configuration Language (HCL), which provides an opinionated approach to IaC. And, finally as your operation scales you need to figure out how to organize and share our code configuration.

Paul said: “Helping people link their brains back to the ways in which they solved these problems for application code [helps] because a lot of those solutions are going to be same.”

Most Unique Use Cases

There is a Terraform provider that allows you to configure and order pizza. No Joke.

“The number of providers is staggering.” - Robbie

Upcoming Trends Surrounding Terraform

Industry has followed Terraform as configuration solutions have become more necessary with the rising complexity of infrastructures and the sheer scale of operations.

“There is no world where there are fewer services. There is no world where there is less software.” - Paul

Building Providers

We discuss the provider development ecosystem in general, and the PagerDuty Provider specifically. We talk about the provider’s origin, being developed by an engineer in Sweden named Alexander Hellbom. He was impressed by the positive and supportive nature of the Terraform development community.

“The providers make Terraform.” - Paul

What’s Next for Terraform

“What’s next for Terraform is trying to give more back to the community, in all honesty.” - Robbie

The team at HashiCorp continue to develop on the Terraform offering. They launched Terraform Cloud last year to help facilitate collaboration.

Additional Resources

• PagerDuty Home Page
• Episode transcribed by Rev

Popular myths or misconceptions about on-call

One of the biggest myths is that on-call is just an extra part of a SRE or sysadmin’s job. That it’s not really a big part of their duties. It’s just a thing you do; it hasn’t always been taken seriously, especially the impact of being on-call to the individual.

Remember - on-call isn’t just for ops or SRE. Andrew Clay Shafer used to describe himself as a “conscientious developer”, even prior to the ideas of DevOps. Because he thought about things this way, it caused him to be a better developer, and this heavily contributed to the foundation of the DevOps movement.

Software engineers are often resistent to being on-call because of what they think it means - based on the horror stories they hear from their coworkers and friends who work in Ops.

How has on-call changed?

Jay: “Automation has made so much of the difference”

Well-documented automation makes it easier to track down what might be contributing to issues. Having things watching what is going on through the deployment process and watching what’s going on. We have a greater ability to spin up replacement systems, too.

We are changing from a model of having one team who is on-call for everything inside the business; now it is more about selected domain experts on call for the thing they know really well. Being on-call as a developer, you know you are only being called about things you know about. Additionally, the more people that go on call, it’s much less actual impact to all the folks who are on-call. So the experience is a lot different. “We’ve reduced the individual blast radius by distributing it” - Jay.

“The beautiful thing about going on-call is you get to go off-call. If you aren’t on-call, I have news for you - you’re always on-call” - Matt. It’s very relieving to know you are not on call, so you don’t have to worry that someone will call you. “Trust me - your ops team knows how to find you, and they will” - Matt.

On-call requirements are different

Not every company or service requires 24 hour on-call support. When you are thinking about where you want to work, consider this. That said, if you do work for an organization that provides a service around the clock, on-call is likely a part of that job, and everyone should consider it part of their service ownership. But ultimately, make the decision for the role that works for you. It’s less about the title or role, than it is for the type of company or organization and what they need. As Jay points out, “in the end, we are all just people, and we have basic requirements - like eating, having water, getting enough sleep, and spending time with people we like. On-call should still let you do these things”.

A good question to ask when getting into a role that has a on-call component, is ask “how are incident responders rotated off of an incident?” Responders stop being effective after a couple of hours - understanding things like “what’s the size of the rotation?”, “what are the expectations of a responder during an incident?”, are much more important to know than “how often will I get paged?”

How to avoid having an on-call nightmare

Jay: “It always comes down to tech debt. It’s amazing how much tech debt comes down to a lack of documentation. It becomes one of those scary parts that if it falls down, nobody will know what to do”

Additional Resources

• PagerDuty Home Page
• Episode transcribed by Rev

Bruce tells us about how the term “chaos engineering” came to be and the mindset behind using the term.

“Let’s create a team strategy and vision around [Chaos Monkey and the practices around it] and let’s double down on what we already started. So in that fashion, we wrote a blog post that introduced the term ‘Chaos Engineering’ and introduced the term ‘Chaos Engineer.’”

What does Chaos Engineering really mean?

Bruce breaks down the pragmatic reasons this practice exists and why we should think about adopting it.

“It’s being proactive and getting a chance to validate our resilience design: finding out how well our systems are architected at 3pm instead of 3am.”

Chaos Engineering Thought Exercises

We discuss how tabletop thought exercises serve as a valuable tool to help you flesh out considerations long before touching any production systems.

“We call it ‘zero tech’ tabletops. I don’t want laptops. I don’t want distractions and excuses for why we can’t get started. And so I run these tabletop exercises, with a whiteboard, with a drawing of the architecture and we talk about our detection strategy, resilience, trade offs, and the parts that fail.”

But I’m not ready for Chaos Engineering!

A common response to the suggestion that a team adopts Chaos Engineering is that they’re simply not ready to get started. We discuss some ways to address these concerns.

“If we’re not ready for this, then are we really ready for production? Ready or not, failure is going to happen.”

Identifying big impact components to test

How do you prioritize which components of your stack to test? What are the considerations for figuring out where to start? Bruce gives some practical advice for where in your stack to start and finding opportune moments to seize upon.

“Cloud provider outages… are the best opportunities. They allow us to identify and be introspective about the things in our control that we can do about this.”

When should you start?

No, really. Big outage aside, when should we get started? Here’s where we see George’s managerial background kick in. Can we start today? Bruce provides some great practical wisdom around getting started as early as when new team members are being onboarded.

“When’s the time you want to start writing more resilient software?”

When the real outage happens

It’s important to celebrate wins. The Chaos Engineering wins are when you’re the team relaxing as a failure happens.

“You’re celebrating because this thing failed exactly as we planned! It happens and there’s nothing for us to do. We’re just sitting back and watching the show.”

Capturing what we learn from Chaos Engineering

Building more resilient systems means taking the things we learn from Chaos Engineering exercises and ensuring that resulting action items make it into our work streams. How can teams do that successfully?

“The first time I did this, we did sprint planning and then the chaos engineering exercise. Nope. That’s the wrong order!”

Parting Advice

Bruce wraps up with practical tips for moving your teams in the right direction.

“You don’t need fancy tooling. You need 3 lines of code: if my user, fail this call.”

Additional Resources

• PagerDuty Home Page
• Episode transcribed by Rev

• PagerDuty Home Page
• Episode transcribed by Rev

Yuri discusses the history of alerting and why alerting should be on things that impact the customer experience.

Yuri: “It’s one of the soapboxes that I find myself on when talking to customers. It so often happens that customers will look for help with alerting; ‘How will I know if I am having an issue with my infrastructure, how will I know if I have high memory consumption?’ You should never alert on things in your infrastructure, you should only alert on things that impact your customer experience.”

Yuri and Julie discuss common mistakes customers make dating back to the beginning of alerting and how products like PagerDuty have changed the way alerting should be done.

A little bit about PagerDuty

Yuri and Julie discuss where PagerDuty came from and where it is today.

Yuri: “The fundamental problem that you are trying to solve, which is like hey I get 1,000 alerts an hour and I don’t know which ones are important. That’s not the problem we should be buying PagerDuty to solve, that’s the problem we should be addressing at the root.”

Julie and Yuri continue to discuss the issues with email filtering and essentially “training people to ignore alerts”

Service Level Indicator and Service Level Objective Usage

Yuri talks about how it’s about the service owner and who is ultimately accountable for reliability of that system and what user happiness means in this context. He continues to discuss SLI’s and SLO’s.

Yuri: “We use SLI’s as a proxy for user happiness.”

Yuri and Julie discuss setting up alerting with SLI’s and SLO’s in mind along with alerts needing to be humanly actionable, with a little bit of error budgets sprinkled in.

Yuri: “Things that are not directly contributing to or impacting user happiness, those should be created as tickets in a ticketing system… there is no need to wake someone in the middle of the night.”

Continued discussion around when alerting should wake someone up in the middle of the night.

Julie: “Making sure every alert that wakes a human up is humanly actionable sounds great but isn’t always easy, and it comes down to fine tuning. Do you have recommendations?”

Yuri: “People often feel that if they don’t have an alert for it, it’s not actually happening..”

Decrypting the Metrics

Julie: “Let’s go deeper on how do we really dig deep into what the customer experience means when you are looking at service level indicators and service level objectives”

Yuri: “You have to have a good understanding of what are people actually trying to do, and then some way of quantifying.”

The discussion continues on what metrics we use to quantify customer success and performance.

Yuri: “The closer we are able to collect this information to the customer, the more accurate it is going to be.”

Technical Debt

Shifting topics to how technical debt is expressed as a gap in knowledge, and how people treat their systems as a black box.

Yuri: “You’ll hear the term ‘black box monitoring’ because they don’t actually know how it works.”

Continued discussion on how technical debt manifests itself in monitoring.

Service Ownership

Yuri discusses what service ownership means to him and how it is really “engineer empowerment”, and what that means to reliability.

Additional Resources

• PagerDuty Home Page
• Episode transcribed by Rev

The hosts start out with their pet peeves: which myths about running software in production would we like to debunk?

Scott: “As a developer, I focus on APIs… So the myth I think about is that software engineers will just ‘get’ your API with very little documentation or [code] samples. While some will, to fully support your community you need to have good documentation, good sample code, and good tools to help people get started.”

We also cover “root cause” vs “contributing factors”, the role of psychological safety in comparison to picking the right tools, and the obligation software engineers have to manage their software in production if they want to write “good” code.

Building more resilient systems & better quality of life

The goal of this podcast is to discuss practices that build both more resilient systems and better quality of life for the people supporting them. In order to do that, which topics would we like to cover?

Matt: “I think a key practice is the idea of learning from incidents. And what I mean by that is how can we get better at actually learning and taking incidents as gifts and not just something we react to and create action items and tickets about.”

We also cover collaboration, development tools, and the importance of diverse viewpoints.

Topics of particular interest

What are each of the hosts focused on?

Julie: “How do we work together to own our code in Production? When there’s a resistance from software engineers to owning that code in production, I’d like to know why.”

The hosts each discuss their own interests which span across psychological safety, developer education & experience, open source software communities, emergency response systems, chaos engineering & game days, human factors, and learning from incidents.

Personal Intros

But wait, WHO are these hosts?

George: “I hated being on-call. I mean loathed being on-call. So many on-call disaster stories that just took a toll on my personal life… Ironically, I ended up working at PagerDuty… It turns out there’s a lot we can do to make [on-call] better and there’s this good Venn diagram of interests that come together when we talk about emergency response.”

Everyone does a quick roundtable introduction.

Why are we all now focused on incident management?

The hosts find common ground from their diverse backgrounds.

Julie: “I started [in IT] as a recruiter which means I got to hear why people were leaving their jobs… What made them miserable?… To have the opportunity to help make people’s lives easier was one I couldn’t pass up.”

Everyone shares a brief version of their origin stories. Scott continues to focus on tools. LOL

What’s with the name Page It To The Limit?

Mysteries are revealed.

What’s on the horizon for content?

The hosts talk about the next few episodes and what to expect on the show.

Show Links

• PagerDuty Home Page

• Google Study on Psychological Safety
• Episode transcribed by Rev