{"type":"video","version":"1.0","html":"<iframe src=\"https://www.loom.com/embed/39d2491795df45bd992f606e226628f9\" frameborder=\"0\" width=\"1920\" height=\"1440\" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>","height":1440,"width":1920,"provider_name":"Loom","provider_url":"https://www.loom.com","thumbnail_height":1440,"thumbnail_width":1920,"thumbnail_url":"https://cdn.loom.com/sessions/thumbnails/39d2491795df45bd992f606e226628f9-60b730bff2ba8be6.gif","duration":2242.127,"title":"AIOps: Leveraging AI for Software Incidents","description":"Satish (Atlassian) explains how AI can assist incident management and root cause analysis by combining code intelligence, observability data, and a teamwork/service graph to ground investigations. The proposed AI SRE agent architecture uses context engineering, multi-agent LLM orchestration, and skills (change analysis, metric/log analyzers) to produce top-K hypotheses and actionable runbooks. Challenges include scale of observability data, incomplete tracing, causal linking between changes and incidents, and the need to limit investigation scope (depth/search-space reduction).\n\n### Talk scope and outline 0:00\n\n- Introduced topic: using AI to help engineers perform detective work for incident management and root cause analysis (RCA).\n- Presentation outline: define software incidents and RCA, role of AI, building blocks of an AI SRE agent in Jira Service Management, and challenges/lessons learned.\n- Emphasis on practical walkthrough of an AI SRE agent built within Atlassian's products.\n\n### Definition and phases of software incidents 1:46\n\n- Defined software incident as unplanned interruption, degradation, or functional/non-functional breakage resulting in availability, reliability, or performance loss.\n- Described three incident phases: detection (monitoring alerts or customer reports/social media), user impact, and resulting business impact (availability, reputation, possible data/revenue loss).\n- Described common company response flow: engage correct incident response and subject-matter experts, investigate potential root causes, apply short-term mitigation to restore service, then perform deeper debugging for long-term fixes.\n\n### Root cause analysis process and five whys 4:18\n\n- Defined RCA as a structured process to identify underlying causes; aim is to go beyond symptoms to what/why/how an incident occurred.\n- Introduced the 5-Whys technique: iterative questioning (peeling layers) to reach fundamental causes; can be hierarchical rather than strictly sequential.\n- Example RCA chain: broken checkout API → DB connection issue → increased load/inefficient DB calls → new feature introduced inefficient DB calls → cache key change causing query surge.\n\n### Mttr and its components 6:48\n\n- Stated RCA goal: reduce MTTR (mean time to recover/restore service).\n- Broke MTTR into subcomponents: MTTD (detect), meantime to acknowledge (on-call/responders join), meantime to engage (getting right experts in room), time to mitigate (short-term fixes), and meantime to resolve (incident fully subsided).\n- Noted that attacking MTTR requires improvements across these subcomponents.\n\n### Role of ai in root cause analysis 8:25\n\n- AI helps handle frequent and numerous changes (PRs, commits, deployments, feature flags) by semantically understanding changes and pointing to suspicious commits or PRs.\n- AI assists with observability triangulation: pattern matching across metrics, logs, traces, and alerts at petabyte/terabyte scales to detect anomalies and correlate signals.\n- AI can help determine correlation vs causation across services and reduce manual hunting when many microservices and dependencies exist (fault propagation/service call graphs).\n\n### Complexities: progressive rollouts, feature flags and historical changes 13:59\n\n- Highlighted complexity from progressive deployments and feature flags: changes may be rolled out to subsets of users, regions, or cohorts causing delayed or partial outages.\n- Noted need to examine not only very recent changes but also older code changes combined with recent flag/config rollouts.\n- Emphasized that AI must account for varied change timelines and rollout strategies when attributing causal links.\n\n### Building blocks of an ai sre agent 14:12\n\n- Framing: emulate human debugging using first-principles reasoning and runbooks but support novel incidents beyond static runbooks.\n- Agent requirements: generic runbooks for traversing changes, dashboards and observability queries, and the ability to deduce and run specific workflows based on incident context.\n- Agent must learn subject-matter expertise from provided context (documentation, teamwork graphs) because it lacks innate domain experience.\n\n### Data foundations: teamwork graph and service catalog 16:51\n\n- Teamwork graph: connected data ecosystem of nouns/relationships (first-party items like JSM incidents, third-party items like GitHub PRs) available for fast query; grounds investigations.\n- Service catalog (assets/backstage): tracks services and upstream/downstream dependencies and sources information from network proxies, distributed tracing, APM (New Relic/Dynatrace) to keep dependency data current.\n- Combined graphs allow building incident subgraphs linking incident → affected service → dependent services → repositories → PRs/commits/deployments for targeted investigations.\n\n### Agent interfaces and context engineering 20:17\n\n- Agent interfaces: product UI (Jira Service Management incident command center), chat interface (Rovo chat), and developer CLI (dev agent) for interaction during incidents.\n- Context engineering sources: service catalog, teamwork graph (first- and third-party data), and MCP/CLI connections used to fetch metrics/logs/third-party data on demand.\n- Emphasis on providing the right data and context at runtime to limit investigation scope and enable the agent to act effectively.\n\n### Llm orchestration and multi-agent skills 24:12\n\n- Multi-agent orchestrator hands off work to sub-agents and skills: RCA planner (creates on-demand investigation runbook), change analysis, metric analyzer, log analyzer, and service fault localization.\n- Agents use tools and ML models via an AI gateway connected to frontier models and on-demand MCP servers for third-party tool access.\n- System applies alert intelligence, entity classification, query/anomaly intelligence and outputs top-K hypotheses (commonly top 3) to evaluate accuracy.\n\n### Physical workflow, tools and code intelligence 27:44\n\n- Physical workflow: engineer interacts via Slack/chat/CLI; agent runs skills producing markdown and code snippets; tools execute queries against teamwork graph, metrics DBs, logs, etc.\n- Tools are exposed as functions, MCP servers, or CLI scripts; many teamwork graph tools are accessed through a CLI server.\n- Code intelligence: indexing repositories to build service profiles (deep wiki of service capabilities), enabling lexical/semantic search across repos to find suspicious code, link incidents to PRs/commits, and identify responsible classes/functions.\n\n### Summary of agent elements and core challenges 31:19\n\n- Recap of five major elements: code intelligence, LLM/agent orchestration, context engineering, data provisioning/runbooks, and investigation strategy.\n- Identified core challenges: causal linking between changes and incidents (descriptions may not state breaking intent), need for granular API-level dependency graphs (distributed tracing coverage is often incomplete), and scale/efficiency of querying observability data.\n- Noted that many observability connectors provide low-level query interfaces requiring query construction; some APMs offer higher-order APIs for agentic investigations.\n\n### Audience q&a: non-change incidents and search reduction 33:45\n\n- Question: incidents can be non-change-driven (organic traffic increases, resource exhaustion) and the agent must handle non-deterministic problems beyond change analysis.\n- Response: agents should deduce the appropriate workflow for the scenario (infrastructure, scaling, or feature-flag related) and run the most specific investigation runbook from tens of possible workflows.\n- Question: how to limit search space when service dependencies are large; answer: use service graph depth limits (e.g., restrict to depth of K) and employ techniques for search-space reduction both for services and PRs to optimize initial investigation set."}