Success in a DevSecOps enterprise hinges on delivering value to the end user, not simply completing intermediate steps along the way. Organizations and programs often struggle to achieve this due to a variety of factors, such as a lack of clear ownership and accountability for the capability to deliver software, functional siloes as opposed to integrated teams and processes, lack of effective tools for teams to use, and a lack of effective resources for team members to leverage to quickly get up to speed and boost productivity.
An absence of a central driving force can result in siloed units within a given organization or program, fragmented decision making, and an absence of defined key performance metrics. Consequently, organizations may be hindered in their ability to deliver capability at the speed of relevance. A siloed DevSecOps infrastructure, where disjointed environments are intertwined to form a complete pipeline, causes developers to expend significant effort to build an application without the support of documentation and guidance for working within the provided platforms. Teams cannot create repeatable solutions in the absence of an end-to-end integrated application delivery pipeline. Without one, efficiency suffers, and unnecessary practices bog down the entire process.
The first step in achieving the value DevSecOps can bring is to understand what it is: “a socio-technical system made up of a collection of both software tools and processes. It is not a computer-based system to be built or acquired; it is a mindset that relies on defined processes for the rapid development, fielding, and operations of software and software-based systems utilizing automation where feasible to achieve the desired throughput of developing, fielding, and sustaining new product features and capabilities.” DevSecOps is thus a mindset that builds on automation where feasible.
The objective of an effective DevSecOps assessment is to understand the software development process and make recommendations for improvements that will positively impact the value, quality, and speed of delivery of products to the end user in an operationally stable and secure manner. A comprehensive assessment of current capabilities must include both quantitative and qualitative approaches to gathering data and determining precisely where challenges reside in the product delivery process. The scope of an assessment must consider all processes that are required to field and operate a software product as part of the value delivery processes. The aperture through which a DevSecOps assessment team focuses its work is wider than the tools and processes typically thought of as the software development pipeline. The assessment must encompass the broader context of the entire product delivery pipeline, including planning stages, where capability (or value) needs are defined and translated into requirements, as well as post-deployment operational phases. This wider view allows an assessment team to determine how well organizations deliver value.
There are a myriad of overlapping influences that can cause dysfunction within a DevSecOps enterprise. Looking from the outside it can be difficult to peel back the layers and effectively find the major causes. This blog focuses on how to conduct a DevSecOps assessment with an approach that uses four methodologies to analyze an enterprise from the perspective of the practitioner using the tools and processes to build and deliver valuable software. Taking the perspective of the practitioner allows the assessment team to surface the most immediately relevant challenges facing the enterprise.
A 4-Pronged Assessment Methodology
To frame the experience of a practitioner, a comprehensive assessment requires a layered approach. This kind of approach can help assessors gather enough data to understand both the full scope and the specific details of the developers’ experiences, both positive and negative. We take a four-pronged approach:
- Immersion: The assessment team immerses itself into the development process by either developing a small, representative application from scratch, joining an existing development team, or other means of gaining firsthand experience and insight in the process. Avoiding special treatment is important to gather real-world data, so the assessment team should use means to become a “secret shopper” wherever possible. This also allows the assessment team to figure out what the real, not just documented, process is to deliver value.
- Observation: The assessment team directly observes existing application development teams as they work to build, test, deliver, and deploy their applications to the end users. Observations should cover as much of the value-delivery process as practicable, such as user engagement, product design, sprint planning, demos, retrospectives, and software releases.
- Engagement: The assessment team conducts interviews and focused discussion with development teams and other relevant stakeholders to clarify and gather context for their experience and observations. Ask the practitioners to show the assessment team how they work.
- Benchmarking: The assessment team captures available metrics from the enterprise and its processes and compares them with expected outcomes for similar organizations.
To achieve this, an assessment team can use ethnographic research techniques as described in the Luma Institute Innovating for People System. Interviewing, fly-on-the-wall observation, and contextual inquiry allow the assessment team to observe product teams working, conduct follow-up interviews about what they observed, and ask questions about behavior and expectations that they did not observe. By using the walk-a-mile immersion technique, the assessment team can speak firsthand to their experiences using the organization’s current tools and processes.
These methods help ensure that the assessment team understands the process by getting firsthand experience and does not overly rely on documentation or the biases of observation or engagement subjects. They also enable the team to better understand what they are observing or hearing about from other practitioners and identify the aspects of the value delivery process where improvements are more likely to be had.
The 2 Dimensions of Assessing DevSecOps Capabilities
To accurately assess DevSecOps processes, one needs both quantitative data (e.g., metrics) to pinpoint and prioritize challenges based on impact and qualitative data (e.g., experience and feedback) to understand the context and develop targeted solutions. While the assessment methodology discussed above provides a repeatable approach for collecting the necessary quantitative and qualitative data, it is not sufficient because it does not tell the assessor what data is needed, what questions to ask, what DevSecOps capabilities are expected, etc. To address these questions while assessing an organization’s DevSecOps capabilities, the following dimensions should be considered:
- a quantitative assessment of an organization’s performance against academic and industry benchmarks of performance
- a qualitative assessment of an organization’s adherence to established best practices of high-performing DevSecOps organizations
Within each dimension, the assessment team must look at a few critical aspects of the value delivery process:
- Value Definition: How are user needs captured and translated into products and features?
- Developer Experience: Are the tools and processes that developers are expected to use intuitive, and do they reduce toil?
- Platform Engineering: Are the tools and processes well integrated, and are the right aspects automated?
- Software Development Performance: How effective and efficient are the development processes at building and delivering functional software?
Since 2013, Google has published an annual DevSecOps Research and Assessment (DORA) Accelerate State of DevOps Report. These reports assemble data from thousands of practitioners worldwide and compile them into a comprehensive report breaking down four-to-five key metrics to determine the overall state of DevSecOps practices across a wide variety of enterprise types and sectors. An assessment team can use these reports to quickly key in on the metrics and thresholds that research has shown to be important indicators of overall performance. In addition to the DORA metrics, the assessment team can conduct a literature search for other publications that provide metrics related to a specific software architectural pattern, such as real-time resource-constrained cyber-physical systems.
To be able to compare an organization or program to industry benchmarks, such as the DORA metrics or case studies, the assessment team must be able to gather organizationally representative data that can be equated to the metrics found in the given benchmark or case study. This can be done in a combination of ways, including collecting data manually as the assessment team shadows the organization’s developers or stitching together data collected from automated tools and interviews. Once the data is collected, visualizations such as the figure below can be created to show where the given organization or program compares to the benchmark.
From a qualitative perspective, the assessment team can use the SEI’s DevSecOps Platform Independent Model (PIM), which includes more than 200 requirements one would expect to see in a high-performing DevSecOps organization. The PIM allows programs to map their current or proposed capabilities onto the set of capabilities and requirements of the PIM to ensure that the DevSecOps ecosystem under consideration or assessment implements the best practices. For assessments, the PIM provides the capability for programs to find potential gaps by looking across their current ecosystem and processes and mapping them to requirements that express the level of quality of outcomes expected. The figure below shows an example summary output of the qualitative analysis in terms of the 10 DevSecOps capabilities defined within the PIM and overall maturity level of the organization under review. Refer to the DevSecOps Maturity Model for more information regarding the use of the PIM for qualitative analysis.
Charting Your Course to DevSecOps Success
By employing a multi-faceted assessment methodology that combines immersion, observation, engagement, and benchmarking, organizations can gain a holistic view of their DevSecOps capability. Leveraging benchmarks like the DORA metrics and reference architectures like the DevSecOps PIM provides a structured approach to measuring performance against industry standards and identifying specific areas for improvement.
Purposefully taking the perspective of the practitioners tasked with using the tools and processes to deliver value helps the assessor focus their recommendations for improvements on the areas that are likely to have the highest impact on the delivery of value as well as identify those aspects of the process that detract from the delivery of value.
Remember, the journey towards a high-performing DevSecOps environment is iterative, ongoing, and focused on delivering value to the end user. By applying data-driven quantitative and qualitative techniques in performing a two-dimensional DevSecOps assessment, an assessment team is well positioned to identify unbiased observations and make actionable strategic and tactical recommendations. Regular assessments are vital to track progress, adapt to evolving needs, and ensure you’re consistently delivering value to your end users with speed, security, and efficiency.
