In the face of growing IT complexity, Cisco IT unified observability across its global environment. The results: 25% fewer major incidents, 45% faster resolution, and scalable automation. Here’s how we did it—and recommendations for your digital resilience journey.
Our challenge: fragmented visibility
Many organizations struggle with fragmented monitoring and prolonged incident resolution. We faced the same challenges and needed to break down data silos to protect our fast-changing environment.
In Cisco IT, managing our global, dispersed IT landscape is increasingly complex. Fragmented data and visibility gaps were making it increasingly difficult to maintain digital resilience amid our fast-paced innovation and frequent environmental changes. We needed a way to turn our data into actionable insights but lacked a unified observability platform to centralize and make sense of it all.
When a major database outage in 2024 revealed our fragmented data — uncorrelated alerts across related devices delaying cause identification — we knew we had to rethink our observability approach.
“After the outage, we knew we had to rethink everything. The transformation wasn’t just about technology, but empowering our engineers to act faster and smarter.”
– Chuck Churchill, Snr. Director, IT Observability, Cisco IT
This inflection point sparked a broader shift in how Cisco IT approached observability. In the video below, Cisco IT leaders share what changed — and how it led to a 25% reduction in major incidents.
Getting started
Like any IT issue, understanding the root cause was essential before remediation planning could begin. Building stronger digital resilience is no different.
In working with our teams, we pinpointed the following as top challenge areas that were hindering our efforts:
- Fragmented data and visibility gaps: With over 100,000 endpoints, we generated massive volumes of telemetry data. Siloed monitoring tools led to alert fatigue and visibility gaps, slowing response times and limiting our ability to predict and prevent issues.
- Risks from frequent changes: Our culture of innovation and early adoption means constant changes to our IT environments. We saw a direct link between rapid change and increased incidents, so minimizing disruption without slowing innovation became essential.
- Resource optimization: As our environment and data complexity grew, it became critical to improve how we leverage AI and data more effectively. We needed to turn our data into actionable insights that empower engineers, not overwhelm them, to ensure productivity kept pace with growth.
“Bringing all our data together was the first step—turning it into real, actionable insights is what truly enables us to stay resilient as our environment evolves.”
– Jon Heaton, Director, Network Engineering and Operations, Cisco IT
Deciding on our three-pronged observability approach
Digital resilience requires more than simply deploying new tools. We needed a holistic approach that could span our entire IT landscape. We achieved this by structuring our IT observability practice across three interconnected pillars:
- The network: Secure, reliable network performance is essential for keeping the business up and running. This pillar focuses on comprehensive network visibility, including third–party provider networks, to ensure the network is running securely and optimally for a smooth user experience.
- Platforms and data: Here, we focus on observability across our data centers, cloud, and underlying infrastructure, centralizing our observability data to be accessible to the entire organization, including our DevOps and SRE teams, through our platform consolidation and data strategy.
- Service Operations: Our Service Operations and Enterprise Operations Center engineers monitor, analyze, and resolve issues using rich data and insights delivered from our network, infrastructure, applications and services — which feed AI automation to improve efficiency.
Adding in the critical technology
Each of these observability pillars is powered by a combination of data, technology, and processes that enable us to use the full potential of our data and drive greater efficiency through AI automation. These core elements are essential to our observability approach and success:
- Splunk: Splunk acts as the backbone of our observability strategy, centralizing data from across our network, infrastructure, and applications to deliver a single source of truth for our IT teams.
- ThousandEyes: ThousandEyes delivers end-to-end network visibility and user experience monitoring across internal and external environments, enabling rapid identification and resolution of connectivity issues.
- Configuration Management Database (CMDB): Our CMDB provides a single source of truth for all IT assets, enriching alerts and incidents with essential context and powering efficient, proactive operations.
- AI Operations: Our AI systems leverage the centralized observability data in Splunk to automate event analysis, reduce alert fatigue, and accelerate incident response — empowering engineers to focus on higher-value work. For example, our Service Operations uses various AI assistants for AI-driven incident management to predict incident assignments and suggest resolution steps.
By integrating Splunk with ThousandEyes, our CMDB, and other tools, we can ensure a seamless, scalable approach to observability that grows with our business.
Tangible outcomes
This unified observability approach has helped us tackle our most pressing challenges and improve outcomes that collectively strengthen our digital resilience. In the past 18 months, we’ve seen:
- Significant reduction in major incidents: We reduced major incidents by 25% year over year and had zero major network incidents, down from 3-4 per quarter previously.
- Faster, more effective restoration: We lowered our Mean Time to Detect and Resolve by 45% year over year, enabling faster recovery and minimized disruption.
- Improved change management: We decreased incidents caused by change by 20%, enabled by unified data insights and end-to-end visibility that supports smarter change management processes.
- Strengthened visibility and data usage: We now monitor 10x more network telemetry data that yields deeper insights and 4x greater visibility — enabling earlier detection and proactive resolution of potential issues before they escalate.
- Scalable automation and efficiency: Centralizing data in Splunk has established a foundation for continued advancements in AIOps, enabling us to develop AI-automations that now handle 99.998% of ~4 million daily alert — significantly improving operational efficiency.
“This journey is as much about changing mindsets as deploying technology. We’re empowering every engineer to act on insights, not just alerts.”
– Mark Hutchins, Director, IT Service Management, Cisco IT
Practical takeaways
Our digital resilience journey is ongoing, but we learn more each day that we share with customers enhancing their own journey. We recommend:
- Collect telemetry from everywhere: Centralize telemetry to optimize the most relevant metrics, events, logs and traces from across your network, infrastructure, cloud, applications, and services.
- Prioritize data – the bedrock of success: focus on data quality and hygiene. Accurate, clean data is essential for generating trustworthy insights and enabling effective automation.
- Use what’s already smart: First, leverage the systems that you have: product capabilities that have AI and observability built in. Second, empower teams to experiment with custom AI solutions to maximize value where your existing systems have gaps.
Stay tuned for updates and learnings as we continue to innovate and optimize.
“Digital resilience is a moving target. We’re always learning, adapting, and refining our approach as our environment evolves.”
– Jon Heaton, Director, Network Engineering and Operations, Cisco IT
More resources:
Explore more case studies to see how Cisco strengthens digital resilience
