Site Reliability Engineering ‘Key Concepts’ – SLO, Error Budget, TOIL and Observability


By: Niladri Choudhuri

“What happens when a software engineer is tasked with what used to be called operations” – Ben Treynor, Google.

Around 2003, much before DevOps came into existence, Google created Site Reliability Engineers. SRE is a discipline where the software engineering principles are applied to the infrastructure and operations problems to make the systems much more stable and reliable and to be able to ultra-scale as per the business needs.

The goals of Site Reliability Engineering are to create ultra-scalable and highly reliable distributed software systems.

SRE’s spend 50% of their time doing “ops” related work such as issue resolution, on-call, and manual interventions and spend 50% of their time on development tasks such as new features, scaling or automation. Monitoring, alerting and automation are a large part of SRE work.

The following are the SRE Principles:

  • Operations is a software problem
  • SRE services are managed with Service Level Objectives
  • SRE practices aim at removing TOIL through automation
  • Automate as much as possible
  • SRE helps in reducing the cost of failure
  • SREs have skillsets of both Dev and Ops and share “Wisdom of Production” to the development team

What is SLO?

SLO or Service Level Objective is the availability criteria for the product and service. It is the expected goal for how well a service should operate. SLOs are very strongly related to the user experience. Once the SLOs are met, customer satisfaction will be high as users will be happy.

SLOs need to be set and monitored regularly as it is a key objective of SRE. There should be various SLOs for Products and Services. SLOs are always from the Customer point of view.

What is TOIL?

“TOIL is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical devoid of enduring value and that scales linearly as a service grows”. – Vivek Rau, Google.

Examples of toil are manual releases, physically connecting to infrastructure to check something, doing regular password resets, testing over and over, acknowledging the same alerts every day, creating users, manual resets, on-call response, extracting data, manual scaling of infrastructure, etc.

TOIL is bad because:

  • It slows down progress
  • Manual Work reduces the quality
  • Career Progression slows down
  • A never-ending list of manual tasks
  • Burnout of resources

According to the Catchpoint SRE Survey Report 2019, the following are the most popular SLOs:

  • Availability – 72%
  • Response Time – 47%
  • Latency – 46%
  • We do not have SLOs – 27%

What is Error Budget?

“100% is the wrong reliability target for basically everything” – Ben Treynor

Error Budget means the amount of Time Budget we have where service can get affected. This is the time that is used to bring in new features or make architectural changes. If we tend to spend more than the budget, there has to be a consequence. One such consequence is to stop new features and get the system stable. So, all the post-mortem related backlogs are prioritized over the new features. SRE encourages to burn the Error Budget to Zero and use it strategically to balance velocity (speed) and availability (stability).

We need to be lean and have smaller batches as big changes can lead to higher risk and thus burning up of the error budget.

For Example:

SLO – 99.9% Availability of the System

Error Budget – 43 minutes per month (0.1%) Within this time all new feature releases, patches, planned and unplanned downtime needs to be fit into these 43 minutes.

Consequence – If the Error Budget is used up, then the release of new features has to stop and user stories from the post-mortem related backlogs need to be prioritized.

What is Observability?

“Observability, as a noun, is a property of a system, it’s a measure of how well internal states of a system can be inferred from knowledge of its external outputs. Therefore, if our IT systems don’t adequately externalize their state, then even the best monitoring can fall short” – Peter Waterhouse, CA

Observability is about having enough data that can be used to answer questions that are not already known. Observability required architecting is such a way so that the system can provide information to be able to help understand the health of the systems.

Observability is important because:

  • Service growth is happening at a very rapid pace
  • Architectures are dynamic in nature
  • The workload of containers is increasing
  • There are service dependencies
  • High level of Customer Experience is very important

There are many other concepts that need to be looked at for delivering the best of services to the customer.

Link to original article

Join our Community for Free

related posts

DevSecOps and ITIL4

DevSecOps and ITIL4

By: Niladri Choudhuri, Hugo Lourenco, Jay Shah and Helen Beal DevSecOps has emerged and established itself as the model to assure cybersecurity is properly considered when transitioning to DevOps ways of working. It demands collaboration between the security, IT...

Progressive Delivery

Progressive Delivery

By: Orit Golowinski October 29, 2020 What is Progressive Delivery? Progressive delivery is the process of pushing changes to a product iteratively, first to a small audience and then to increasingly larger audiences to maintain quality control. Progressive Delivery is...

CI/CD Patterns and Practices

CI/CD Patterns and Practices

By: Tiffany Jachja October 27, 2020 CI/CD Patterns and Practices In August, I shared a talk on enabling CI/CD through patterns and practices. This blog post summarizes the contents of said talk, informing readers how to enable continuous software delivery. What is...

Visit Us
Follow Me