By: Niladri Choudhuri, Hugo Lourenco, Jay Shah and Helen Beal
DevSecOps has emerged and established itself as the model to assure cybersecurity is properly considered when transitioning to DevOps ways of working. It demands collaboration between the security, IT operations and development teams. It drives automation of security testing as early as possible in the software development and delivery lifecycle. It requires that knowledge is shared from security experts to software engineers and vice versa.
DevOps recognizes that every organization is a technology company and the huge increase in use of devices, mobile apps and IoT, etc. Cybersecurity risk is increasing at the same rate or even faster. Information is available at the speed of your connection and data breaches are expensive.
We also find that the vulnerabilities that we currently see are the same ones that we will continue to encounter for quite some time. The OWASP Top 10 has had the same vulnerabilities at the top for the last seven years. Gartner predicts that, through 2020, 99% of vulnerabilities exploited will continue to be the ones known by security and IT professionals for at least one year.
Security has become very important and, as per DevSecOps: “Security is everyone’s responsibility.” Security needs to be brought into the lifecycle from the time of requirements definition.
DevOps and DevSecOps connect through a system of strong values. The diagram below shows the values of DevOps and DevSecOps.
It is not about the name but about the concept. Security must be completely integrated with Development and Operations. This is the time to reinvent security. We need to leverage automation and redefine the roles and practices of Security Engineering, Security Management and Governance, Risk Management and Compliance (GRC). We need to start implementing “Security as a Code” and Security Policies as Code.
GRC and Security SMEs need to come as trusted advisors to the team. Remember the Project to Product Approach where it is one team looking after all the needs of a set of applications from new changes to maintaining it without the need to go out of the squad for anything related to capability, skills and decision making.
DevSecOps is getting security embedded in DevOps. A security team that operates in a DevOps manner. Shifting of security left with Dev and Ops empowering security.
The New Way
DevSecOps is redefining the way of running a DevOps initiative. We need to focus on the following:
The Advice Process
Resilience is about how fast we can get back to normal after an incident or attack. There is nothing that is 100% secure. We need to be able to get back to normal business as quickly as possible after a security attack. To do this, we need the following:
Fail fast, learn fast, recover fast
“The best way to avoid failure is by failing regularly.” (Netflix) – Choose where to fail
Security is important for reliability and hence for resilience
Continuous experimentation and learning is important for security aspects too
We need to be following Lean principles such as:
Minimize overhead and waste
Automate as much as possible – People makes mistakes, scripts do not
Emphasize on measurable results and the outcome effectiveness
Do not make rules for the sake of making rules
Come as a trusted advisor and not as a road-blocker
It is important to understand the context and have a holistic view. Keep these points in mind:
Context is everything
Understanding of the business (domain) is important
Take a holistic approach
The level of tolerance in various service areas (do not spend $200 to save a $5 asset)
Understanding of the entire system (System Design)
How Much Security is Enough?
Security SMEs should come as a cooperative partner and not as an adversary. We need to strike a balance between real versus perceived threats. Threat modelling is one of the most important techniques, that should be made mandatory, to follow throughout the lifecycle from the time of designing. Evaluate assets versus threats versus vulnerabilities. Pick your battle.
Design is a very important aspect and security has to be designed into the system. Understanding of the holistic IT landscape is important. System design approach will help.
Basic Security Hygiene
There needs to be basic security hygiene for everyone in the squad. This can be:
Vulnerability and Patch Management
Automate Scanning and Report Processing
Automate Patch Management
Automate Patch Management
Backups, Replication and Redundancy – Production like data must have Production like Security
Monitoring Size and Utilization
Basic Network Security
IDS/IPS (Intrusion Detection System/Intrusion Prevention System
Logging and Monitoring
Logging and Monitoring for Security is also a very critical aspect. We need to look at the following three areas:
The following are some of the activities related to the above three:
Setting Up Log Management
Queries/Reporting/Alert – One Alert for One Service
Opportunities for Automation
Incident Response and Forensics
Upskill – Cloud Skills
Live Information and Response
Leverage Automation Opportunities
Enrichment activities after Alert
Automate workflow for Forensic Analysis
Have Contingency Plan – (Nothing is 100%)
Call for help
Pre-define Incident Response Plan
Chaos Engineering and Fire Drills
Threat Intel and Information Sharing
ITIL® 4 – High-Velocity IT (HVIT)
Now that we understand security, let us see how ITIL® 4 relates to security and DevSecOps. The HVIT manual states that:
“Digital Technology is increasingly important. Its economic, societal, and political impacts are unprecedented. At the same time, it is increasingly challenging for digital practitioners to design, develop, run and support the systems and services that fulfill this demand. ITIL® 4 High-velocity IT focuses on digital products and services, including digital customer experiences, covering good organizational practices and mental models, all from a practitioner’s perspective.”
The following is the IT Value Stack (From ITIL® 4: High-Velocity IT Manuscript by Axelos):
Objectives of HVIT:
Key Characteristics of HVIT are:
HVIT emphasises that resilience is necessary to high velocity and a system cannot be reliable if it is not secure. These are the different practices and processes that help to incorporate resilience. Here are some excerpts as well as our insights on the guidance.
From ITIL4 HVIT: “Approaches with resilient characteristics are focused on maintaining workable availability and performance, and minimizing the effect of Incidents.” We say that SRE and DevSecOps focus on reliability and a system cannot be reliable without being secure
IFrom ITIL4 HVIT:: “Security Officer’s role shifts from specifying requirements and monitoring performance, to enabling practitioners to address security concerns.” We agree, but go further and say that security people must be involved in the end-to-end value stream, specifying non-functional requirements in the product backlog and working with the team to ensure these requirements are tested at the earliest opportunity
The practice relevant to security is Information Security Management and helps in resilient operations and assured conformance. Our DevSecOps view also builds to value stream thinking to result in continuous compliance
These practices are part of DevSecOps:
Infrastructure as Code: designing and enforcing security policies in virtual infrastructure components
Loosely Coupled Architecture: design and manage information security policies for loosely coupled services and service components
Blameless Retrospective: Obtaining more information about the circumstances related to security breaches and security incidents
CICD – Ensuring compliance with information security policies by reducing manual work. Increasing the scale and scope of automation by leveraging CI/CD tools may help to ensure compliance with information security policies by reducing manual work (Toil) and improving traceability of changes.
Continuous Testing (CT): Two practices are involved: first, Service Validation and Testing – Unit and integration Testing is conducted on an ongoing basis throughout the development lifecycle This includes application unit testing, infrastructure services testing, functional/non-functional testing, canary releases, blue/green testing, and infrastructure security testing. The next practice is Information Security Management – Ensuring compliance with InfoSec policies by reducing manual work. Leveraging automated testing tools may help to ensure compliance with information security policies by reducing manual work and improving the traceability of changes.
Technical Debt: Design, implementation and improvement of information security controls are influenced by existing technical debt. Information security controls can also result in the creation of technical debt, which needs to be acknowledged and communicated to all relevant stakeholders.
Non-functional criteria should specify the quality needed by the people who will operate, maintain, and enhance the system, and by those who will ensure security and regulatory compliance.
Definition of Done: Security Tests such as vulnerability, penetration, or policy compliance should be included in the definition of done for resilient products and services
Version Control: Addressing or closing information security risks by flagging vulnerable versions of service components
Security can contribute significantly to relationship management due to better service management
Assured conformance can be measured by or lack of security breaches, fines by regulators, bad publicity, actions required by internal and external auditors, and the cost of measures to ensure conformance with GRC issues
Audit Defense Toolkit: Information Security Management – Designing and implementing controls in the product lifecycle to provide extensive traceability and joint accountability
Service Configuration Management: Standard configurations to support security and audit requirements
A specific section is provided explaining DevSecOps from the authors’ perspective
ITIL4 processes supporting Information Security Management:
Security Incident Management Process
Risk Management Process
Control Review and Audit Process
Identity and Access Management Process
Procedures for Penetration Testing, Vulnerability Scanning, etc
Procedures for managing security related changes, such as firewall configuration changes
The integrated way of approaching security contributes to assured conformance
DevSecOps is relevant for:
Continual Improvement – Improvements to security controls and policies can be part of the learning and feedback incorporated by development and operations
Information Security Management – Designing and implementing controls in a development lifecycle to provide extensive traceability and joint accountability. Integrating Information security duties into the daily work of practitioners. Information Security Management and Risk Management should be an integral part of the daily work of practitioners
Monitoring and Event Management – Configuring monitoring tools to continually scan for threats and vulnerabilities
Change Enablement – Implementing a preventative control that automatically requires pre-authorization from security management before developers can make certain types of production data edits
Deployment Management: Security Management provides guidance on key credential management, CD pipeline security checks, container security, automated pentesting, and data and performance management.
Knowledge Management: Access to security policies
Risk Management: security is about risk
Service Validation and Testing: Test data management is a key element that helps to ensure continued stability, reliability, availability and security
Strategy Management: Integrating duties to balance regulatory requirements with speed of execution
Workforce and Talent Management: Training and coaching staff and other relevant stakeholders on how to build security into development and operations work
Business Analysis: Understanding the security policies, standards, risks, potential threats and vulnerabilities in internal and external environments and translating them into requirements for development and operations team and including security requirements into product backlog
Infrastructure and Platform Management: Security Management can enhance infrastructure and platform management with guidance on secure standards and training, privacy reviews, threat modelling, credential management and data security, especially with Infrastructure as code
Software Development and Management: Enhancing software development with guidance on secure coding standards and training, privacy reviews, threat modelling, code analysis, source code and credential management and data security
In designing for reliability and security, you must consider different risks
Primary reliability risks are non-malicious in nature like a bad software update or a physical device failure. Security risks come from adversaries who are actively trying to exploit system vulnerabilities
Reliability and Security Tradeoff – Redundancy
Reliability and Security Tradeoff – Incident Management – Special small team with skills to handle security related incidents
Security and reliability are about confidentiality, integrity and availability
Intersection of Reliability and Security – Effects of insiders
Design of Security and Reliability
Design for least privilege
Design for understandability
Design for resilience
Design for recovery
Mitigating Denial of Service (DOS) attacks
We should also touch upon the need for Secure Automation as there is a lot of automation done everywhere, both in Dev and Ops Side including SRE. Automation protects from chance of human error or willful sabotage and provides secure opportunities. The following points are important:
Automated steps in the pipeline can be secure but manual steps cannot be secure
Artifacts used in the pipeline can be validated and checked for compliance through digitally versioning and signing
DevSecOps has introduced security into the build-test-deploy lifecycle
SRE places extra emphasis on Security in Production
Application, Infrastructure and Configuration changes through code analysis tools and check for security issues
Digitally signed artifacts to avoid “fake” code
Secure coding practices to share with everyone
Secure code repositories with proper access control
Open source product with feedback from community
For Infrastructure change in an “immutable” test environments creation guarantees compliance with code repository
Deploy artifacts on test environment securely
Test Data and Test scenarios are built for security
Staging environments are immutable
Same artifact deployed in all environments
Production like data in any environment needs production like security
Dedicated Security Scanning
Proxy Security Requirements to be considered
Immutable Production Environment
Same artifact in Production environment
Regulatory Compliance needs to be evidenced and best done with automation and observability
Failure Testing helps audit compliance
There is a great amount of involvement of both Development and Operations in providing a reliable and secure system. Security needs to be looked at seriously and everyone should take shared accountability.