video1.0<iframe src="https://www.loom.com/embed/39d2491795df45bd992f606e226628f9" frameborder="0" width="1920" height="1440" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>14401920Loomhttps://www.loom.com14401920https://cdn.loom.com/sessions/thumbnails/39d2491795df45bd992f606e226628f9-60b730bff2ba8be6.gif2242.127Satish (Atlassian) explains how AI can assist incident management and root cause analysis by combining code intelligence, observability data, and a teamwork/service graph to ground investigations. The proposed AI SRE agent architecture uses context engineering, multi-agent LLM orchestration, and skills (change analysis, metric/log analyzers) to produce top-K hypotheses and actionable runbooks. Challenges include scale of observability data, incomplete tracing, causal linking between changes and incidents, and the need to limit investigation scope (depth/search-space reduction). ### Talk scope and outline 0:00 - Introduced topic: using AI to help engineers perform detective work for incident management and root cause analysis (RCA). - Presentation outline: define software incidents and RCA, role of AI, building blocks of an AI SRE agent in Jira Service Management, and challenges/lessons learned. - Emphasis on practical walkthrough of an AI SRE agent built within Atlassian's products. ### Definition and phases of software incidents 1:46 - Defined software incident as unplanned interruption, degradation, or functional/non-functional breakage resulting in availability, reliability, or performance loss. - Described three incident phases: detection (monitoring alerts or customer reports/social media), user impact, and resulting business impact (availability, reputation, possible data/revenue loss). - Described common company response flow: engage correct incident response and subject-matter experts, investigate potential root causes, apply short-term mitigation to restore service, then perform deeper debugging for long-term fixes. ### Root cause analysis process and five whys 4:18 - Defined RCA as a structured process to identify underlying causes; aim is to go beyond symptoms to what/why/how an incident occurred. - Introduced the 5-Whys technique: iterative questioning (peeling layers) to reach fundamental causes; can be hierarchical rather than strictly sequential. - Example RCA chain: broken checkout API → DB connection issue → increased load/inefficient DB calls → new feature introduced inefficient DB calls → cache key change causing query surge. ### Mttr and its components 6:48 - Stated RCA goal: reduce MTTR (mean time to recover/restore service). - Broke MTTR into subcomponents: MTTD (detect), meantime to acknowledge (on-call/responders join), meantime to engage (getting right experts in room), time to mitigate (short-term fixes), and meantime to resolve (incident fully subsided). - Noted that attacking MTTR requires improvements across these subcomponents. ### Role of ai in root cause analysis 8:25 - AI helps handle frequent and numerous changes (PRs, commits, deployments, feature flags) by semantically understanding changes and pointing to suspicious commits or PRs. - AI assists with observability triangulation: pattern matching across metrics, logs, traces, and alerts at petabyte/terabyte scales to detect anomalies and correlate signals. - AI can help determine correlation vs causation across services and reduce manual hunting when many microservices and dependencies exist (fault propagation/service call graphs). ### Complexities: progressive rollouts, feature flags and historical changes 13:59 - Highlighted complexity from progressive deployments and feature flags: changes may be rolled out to subsets of users, regions, or cohorts causing delayed or partial outages. - Noted need to examine not only very recent changes but also older code changes combined with recent flag/config rollouts. - Emphasized that AI must account for varied change timelines and rollout strategies when attributing causal links. ### Building blocks of an ai sre agent 14:12 - Framing: emulate human debugging using first-principles reasoning and runbooks but support novel incidents beyond static runbooks. - Agent requirements: generic runbooks for traversing changes, dashboards and observability queries, and the ability to deduce and run specific workflows based on incident context. - Agent must learn subject-matter expertise from provided context (documentation, teamwork graphs) because it lacks innate domain experience. ### Data foundations: teamwork graph and service catalog 16:51 - Teamwork graph: connected data ecosystem of nouns/relationships (first-party items like JSM incidents, third-party items like GitHub PRs) available for fast query; grounds investigations. - Service catalog (assets/backstage): tracks services and upstream/downstream dependencies and sources information from network proxies, distributed tracing, APM (New Relic/Dynatrace) to keep dependency data current. - Combined graphs allow building incident subgraphs linking incident → affected service → dependent services → repositories → PRs/commits/deployments for targeted investigations. ### Agent interfaces and context engineering 20:17 - Agent interfaces: product UI (Jira Service Management incident command center), chat interface (Rovo chat), and developer CLI (dev agent) for interaction during incidents. - Context engineering sources: service catalog, teamwork graph (first- and third-party data), and MCP/CLI connections used to fetch metrics/logs/third-party data on demand. - Emphasis on providing the right data and context at runtime to limit investigation scope and enable the agent to act effectively. ### Llm orchestration and multi-agent skills 24:12 - Multi-agent orchestrator hands off work to sub-agents and skills: RCA planner (creates on-demand investigation runbook), change analysis, metric analyzer, log analyzer, and service fault localization. - Agents use tools and ML models via an AI gateway connected to frontier models and on-demand MCP servers for third-party tool access. - System applies alert intelligence, entity classification, query/anomaly intelligence and outputs top-K hypotheses (commonly top 3) to evaluate accuracy. ### Physical workflow, tools and code intelligence 27:44 - Physical workflow: engineer interacts via Slack/chat/CLI; agent runs skills producing markdown and code snippets; tools execute queries against teamwork graph, metrics DBs, logs, etc. - Tools are exposed as functions, MCP servers, or CLI scripts; many teamwork graph tools are accessed through a CLI server. - Code intelligence: indexing repositories to build service profiles (deep wiki of service capabilities), enabling lexical/semantic search across repos to find suspicious code, link incidents to PRs/commits, and identify responsible classes/functions. ### Summary of agent elements and core challenges 31:19 - Recap of five major elements: code intelligence, LLM/agent orchestration, context engineering, data provisioning/runbooks, and investigation strategy. - Identified core challenges: causal linking between changes and incidents (descriptions may not state breaking intent), need for granular API-level dependency graphs (distributed tracing coverage is often incomplete), and scale/efficiency of querying observability data. - Noted that many observability connectors provide low-level query interfaces requiring query construction; some APMs offer higher-order APIs for agentic investigations. ### Audience q&a: non-change incidents and search reduction 33:45 - Question: incidents can be non-change-driven (organic traffic increases, resource exhaustion) and the agent must handle non-deterministic problems beyond change analysis. - Response: agents should deduce the appropriate workflow for the scenario (infrastructure, scaling, or feature-flag related) and run the most specific investigation runbook from tens of possible workflows. - Question: how to limit search space when service dependencies are large; answer: use service graph depth limits (e.g., restrict to depth of K) and employ techniques for search-space reduction both for services and PRs to optimize initial investigation set.