Blameless culture in tech - what, why and how?
The team at Pushpay are passionate about innovation, technology and giving back. They've used Summer of Tech for a few years now to give back to the NZ tech community through summer tech internships. Today, they're talking about the importance of blameless culture in the tech world.
Written by Gina Holden and Oliver Jacks
What are Postmortems?
Software is complex, and it's inevitable that things will go wrong: even when people are trying to do their best work. Blameless postmortems are a key mechanism that allow us to learn from our mistakes. You may be wondering how examining a dead body has anything to do with IT. Well, much like in the medical field, where postmortems examine how someone has died, in IT this can be applied in a similar way when something goes wrong in the production environment (a live environment such as a website or app that customers interact with).‘Blameless postmortems’ were first pioneered by Etsy. Once something goes wrong in the Production environment, a postmortem is started. Their main purpose is to facilitate learning: a timeline of actions that led to the incident is built up, the impact of the incident is analysed (did it affect any customers, and how were they impacted?), and mitigations are implemented as a team to prevent the incident from happening in the future. A discussion is usually led by a trained facilitator. This meeting involves all parties that were involved, and the facilitator conducts the meetings in a blameless manner. Many IT companies around the world today, including Pushpay, have implemented this concept into their company’s culture.
Blameless culture brings space to innovate.
Blameless culture is all about not blaming yourself or others when something goes wrong in Production. This may sound odd and you may be thinking, surely someone is always to blame right? With Blameless culture, scapegoating is out! There are many steps to a delivery process that leads to code going live. At Pushpay, code has to be reviewed and approved by at least one Quality Assistant and one Developer before being put into Production, so a mistake can’t be placed solely on one person. Having software engineers who aren’t afraid to try new technologies, and apply new skills, is an asset to Pushpay. It allows us to continuously develop new features. In fact, some of the best ideas/solutions raised from Postmortems have involved improving our systems and processes (the environment people work in).
Mistakes can be caught earlier
Focusing on how an incident happened, over blaming those involved directly, creates a work environment that’s much safer. By not fearing retribution, breaking changes can be caught earlier, preventing a postmortem from happening in the first place. Fun fact: at Pushpay we often celebrate with an office clap when someone writes up their first postmortem, as we believe this is something that everyone should experience.
Continuous Learning & Improvement Are Valued
The large majority of postmortems actually happen because assumptions are made about how things work and what engineers can do. Taking the time to sit down and discuss the events leading to an incident allows people to share their knowledge over what has happened. Postmortems have a focus on learning: what we have learned from the incident, and how we can use this to improve our systems in the future. Everyone at Pushpay has access to these documents and the recordings of these postmortem discussions, to be able to learn from other people’s learnings.
Pushpay & Tech Internships
As a Summer of Tech intern, you’ll be working with new tooling and software, possibly coding in a live environment for the first time, and it’s expected that you’ll make mistakes. Everyone does! What is important is how you deal with them: communicating problems you are facing, and any issues you might find along the way. If you’re an employer and it’s your first time hearing about blameless postmortems then hopefully you can take something away from this too and consider if implementing this at your company could be beneficial.If you’re interested in learning more about Blameless postmortems, check out these links here:How Blameless postmortems and Just Culture works at EtsyPostmortem culture at GoogleMore on what are postmortemsTo register for Summer of Tech and access tech internships with companies like Pushpay, go here.
