Talk to any DevOps vendor today, and they’ll proudly tell you about their AI roadmap. Most vendors have already built something that will tick the checkbox, if that’s among your requirements.
But checkboxes don’t solve problems. A feature that’s hard to use or adds extra manual steps to a developer’s processes doesn’t save you anything — and may end up costing you more than you expect. Just like you, vendors today are at the start of their AI journey. In some cases, the proof of concept gets packaged and shipped. The box is checked, the product goes out the door, and now it’s up to you to figure out if it’s worth using.
Most DevOps AI Tools Are Still Point Solutions
The truth is that nobody’s using one AI solution to address the entire software development lifecycle (SDLC). The vision of AI that takes you from a list of requirements through work items to build to test to, finally, deployment is still nothing more than a vision. In many cases, DevOps tool vendors use AI to build solutions to the problems their customers have. The result is a mixture of point solutions that can solve immediate developer problems. The point solutions may share a framework that’s akin to a large language model but don’t interoperate beyond that.
Sometimes, Some AI Is Worse Than None
I recently evaluated AI in 11 DevOps platforms. One of my questions was “Does this make life easier for the user?” In some cases, the answer was clearly no.
- One vendor had “fully integrated AI” that consisted of a button next to every text widget. The result? The developer turned into a robot whose job was to copy from one text widget to the AI, press a button to generate a result, then copy the result back again. When faced with that job description, it’s no wonder that some developers think AI makes their life worse — it does!
- Another vendor had a chatbot. When I prompted the chatbot to recommend a few good DevOps platforms, I got a confident answer back. There was only one problem: The answer recommended the platform’s competitors.
There Are Some Bright Lights
We’re starting to see some uses of AI that are well integrated and truly a benefit to development and operations. AI features that aren’t yet table stakes but are coming soon include:
- Natural language policy as code. Building Rego assertions is usually not anyone’s favorite task. A few platforms offer tools that translate natural language into Open Policy Agent. This simplifies setting up pipeline governance.
- Reordering of builds and tests. Machine learning is speeding up testing by failing faster. Build steps get reordered automatically so those that are likely to fail happen earlier, which means developers aren’t waiting for the full build to know when they need to fix something. Often, the same system is used to detect flaky tests by muting tests where failure adds no value.
- Remediating pipeline failure. Almost every DevOps platform provider includes some way to ask “What does this error message mean?” when a pipeline fails. Those further ahead are also using AI to create plans that fix problems and suggest changes to correct pipeline issues.
- Monitoring with automatic remediation. Machine learning gradually helps identify the characteristics of a working system and can raise an alert when things go wrong. Depending on the governance, it can spot where a defect was introduced and start a production rollback while also providing potential remediation code to fix the defect.
- Release readiness reports with key themes highlighted. AI summarization can create a simple summary based on the issues fixed in a release, those still open, issue severity, and management overrides to come up with a natural language readiness report suitable for auditors.
Look For Good Integration Today
If you’re choosing a DevOps tool today and want to decide whether the AI is ready for prime time, consider the following:
- Copy-and-paste is a failure. When you get code snippets back, do you have to “paste at cursor,” or does the AI update the file for you? When you’re diagnosing a problem, do you have to copy-and-paste an error message into a window to get an explanation, or does the platform build a pull request for your review? If your tool is increasing developer toil, it’s a sign that the AI checks the box but won’t help your velocity.
- Context should be correct by default. The DevOps tool should know about your code and your system. Do you have to suggest the files to update in your prompt, or does the AI update the correct files for you? Will the tool connect a deployment failure back to a build error and ultimately to a change?
- Your vendor should be confident in its AI governance. Indemnity statements today mostly protect the vendor, not you. Will your vendor indemnify you if you make changes to what the AI generates, or do they require that AI-generated source code stay forever untouched by development and operations? Can you determine which code was AI-generated and which was not? What about at audit time? Can you show the prompts that were used to generate the build?
- AI models must be interchangeable. At the outset of integration of AI into the SDLC, some vendors picked the AI models for you. But businesses need the ability to choose the AI models (including on-premises models or private models in the cloud) to suit their risk and cost preferences. A DevOps tool should be as flexible with AI models as it is with source control or security scanning.
AI Is Not Just Hype
There’s a lot of puffery around AI, and DevOps vendors are not helping. A lot of their marketing emphasizes fear: “Your competitors are using AI, and if you’re not, you’re going to lose” is their message. Yet DevOps vendors themselves are only one or two steps ahead of you in their AI adoption journey. Don’t adopt AI pell-mell due to FOMO, and don’t expect to replace everyone under the CTO with a large language model. Do, however, explore AI thoughtfully to see if there are places where it can help improve the software development process at your organization. I guarantee that some of your developers have personal accounts and are doing that already.
