标签：#sre

On ne gère pas ce qu'on ne mesure pas. C'est l'une des premières leçons de l'exploitation, et pourtant je l'ai apprise à l'envers, en pilotant à l'aveugle bien trop longtemps. Sans mesure, tu ne sais pas si un système va bien. Tu le supposes. Il tourne, personne ne se plaint, donc tout va bien — jusqu'au jour où quelque chose se dégrade lentement, sous le radar, et où tu ne l'apprends que lorsque c'est déjà une panne. La lente fuite de mémoire, le disque qui se remplit, la latence qui grimpe d'une milliseconde par semaine : rien de tout cela ne crie. Ça glisse. La mesure transforme les suppositions en faits. Un tableau de bord, quelques alertes bien choisies, et soudain tu vois le problème arriver au lieu de le subir. Tu n'attends plus que l'utilisateur t'apprenne que ton système est cassé ; tu le sais avant lui. Mais il y a un piège que j'ai appris à éviter : mesurer trop. Cent métriques que personne ne regarde ne valent pas mieux que zéro. Le bruit noie le signal, et les alertes qui se déclenchent sans raison finissent par être ignorées — jusqu'à celle qui comptait vraiment. Bien mesurer, ce n'est pas tout mesurer. C'est choisir les quelques signaux qui prédisent réellement un problème. Alors, avant de bâtir la prochaine chose, demande-toi comment tu sauras si elle va mal. Si la réponse est « quelqu'un finira par le remarquer », tu ne la gères pas encore. Tu espères. Et l'espoir n'est pas une stratégie d'exploitation. – Serguey Shinder

A sensor on my machine returned nothing at all — empty stdout, empty stderr, exit code 2 — on every invocation for 36 days. It was not crashed. It was not misconfigured. It was doing exactly what one line of well-intentioned code told it to do: announce a condition once . The line looked like this, and I suspect you have written it: if [ ! -f " $OFFLINEFILE " ] ; then echo "body context n/a — phone unreachable" > &2 touch " $OFFLINEFILE " fi exit 2 Read it as a rate limiter and it is obviously fine: don't spam the log with the same message every five minutes. Read it as what it actually is and it is a bug, because the guard does not limit a rate. It changes the tense of the sentence. Every number, code listing, and command output below was re-measured on the machine while writing this, not quoted from the commit that fixed it. Two of the things I expected to find turned out to be false; both are in section 6, and one of them is the most interesting part. 1. Present tense, past tense phone unreachable is a claim in the present tense . It is a statement about the world right now, and it is what a reader of this tool wants: is the body sensor readable at this moment? Wrapping it in [ ! -f "$SENTINEL" ] silently rewrites it into the past tense : the phone became unreachable, at some earlier point, at least once. That is a different proposition. It is true exactly once per transition and false forever after, which is why the guard can never fire twice, and why the sentinel's own mtime is the only surviving record of when the sentence was last true. The two propositions coincide on the first run. That is the whole trap. A first-time-only notice is indistinguishable from a live one for the length of one invocation, which is exactly the length of the test you will write for it. 2. What the reader got instead Here is the tool, before the fix, run twice in a row against a phone that is genuinely away. I pulled the pre-fix version straight out of git into a scratch path and ra

The page says that a property manager never received a welcome email. The useful signal should have arrived earlier, when that tenant's sending domain or delivery-event polling stopped matching the expected state. Short answer: keep welcome-email templates in the application when review history and portability matter most; use provider-owned templates when authorized non-engineers need to edit and preview copy, then select a transactional email provider that supports your chosen ownership model, per-domain management, and occasional batch sends. For a multi-tenant property SaaS, don't let the provider choose the template owner by accident. The reliable design is small: one authoritative template, one tenant-to-domain mapping, and one delivery ledger keyed by an application-generated message ID. Provider selection comes after those decisions. This ordering matters because a successful API request cannot prove that the correct branded message reached the correct property manager. Ownership comes first. How should multi-tenant SaaS welcome email templates be owned? Start with the people allowed to change the welcome message. Application-owned templates put markup, variables, tests, and review history beside the workflow that creates a manager account. They fit when a copy change must ship with a schema change, security-sensitive wording requires code review, or provider portability is a firm requirement. The catch is that a typo correction joins the engineering release path, and the team must build or adopt its own preview step. Provider-owned templates invert that arrangement. A lifecycle or support team can edit copy inside a controlled delivery workflow, and template preview lets a junior developer inspect the branded result before activation. Template identifiers and variable contracts then become deployed configuration. Rollback means selecting a known template revision, not merely reverting application code. I'm not sure which ownership model fits your organizati

Most Kubernetes advice assumes you have a platform team: specialists who own upgrades, ingress, security policies, and the 2 a.m. pages. The teams I am writing for usually have three to ten engineers, one of whom “knows Kubernetes,” and no dedicated platform team. They depend on a cluster that nobody fully owns. I work in enterprise environments where platform teams are large and everything is process. This article is the opposite exercise: what is the minimum discipline a small team needs to run Kubernetes in production—and what enterprise baggage should it refuse to copy? The question that matters more than any tool Before any checklist: who owns the platform after the migration is finished? Not “who set it up.” Who owns upgrades next year, certificate renewals, the CNI version, and deprecated APIs? If the answer is one person's name, you do not have a platform. You have key-person risk with YAML on top. If the answer is “nobody, really,” Kubernetes is invisible operational debt accumulating interest. The rest of this checklist exists to make that ownership small enough for a small team to carry. For each item, score 0 if it does not exist, 1 if it exists but is informal or untested, and 2 if it is documented and tested. The purpose is not to produce a flattering number. It is to expose the next few conversations the team needs to have. 1. Deployments: Git is the source of truth Treat Git as the source of truth for workloads and cluster configuration, including temporary fixes. Use one reconciliation path—for example, Argo CD or Flux—so production changes are reviewed and reproducible. Keep emergency access, but reconcile every emergency change back into Git. Define and test a rollback path for every service. A Git revert is useful only if your delivery process can deploy it safely. This converts your cluster from a mystery into a diff. Every other practice gets easier once “what is running?” has an answer. 2. The rollout basics that prevent late-night incidents R

If you love the simplicity of Docker Swarm (native Compose files, lightweight single binary) but miss the advanced capabilities of Kubernetes (targeted label placement, SRE-grade observability, built-in DNS service discovery, and zero-trust ingress), meet Gubernator (gbnt) . We are excited to release Gubernator v2.13.0 , introducing three massive feature suites natively integrated into a single binary and a modern Material Design 3 Flutter Web Dashboard: Google SRE Multi-Burn-Rate SLO Engine & Interactive Suite CoreDNS 4-Tab Management Suite & Interactive Dig Playground Caddy Ingress & Zero-Trust Reverse Proxy Suite Fun Fact: The entirety of Gubernator's codebase, multi-node deployment pipelines, and SRE features were designed, built, and pair-programmed using **Google Antigravity (AGY) , Google DeepMind's agentic AI coding assistant! Let's dive into what's new and how you can level up your self-hosted or production container clusters! 1. Google SRE Multi-Burn-Rate SLO Engine & Web Suite Defining Service Level Objectives (SLOs) and tracking Error Budgets is the gold standard of Site Reliability Engineering. Until now, implementing SLOs meant running heavy Kubernetes CRDs (via tools like Sloth or Pyrra) or using costly SaaS platforms. Gubernator v2.13.0 brings Google SRE Workbook (Chapter 5) compliant multi-burn-rate alerting straight to simple docker-compose.yml services: version : " 3.8" services : payment-api : image : hashicorp/http-echo:latest labels : gbnt.slo.enable : " true" gbnt.slo.target : " 99.9" gbnt.slo.window : " 30d" gbnt.slo.template : " caddy-http" gbnt.slo.journey : " Checkout Flow" What makes Gubernator's SLO Suite unique? Google Multi-Burn-Rate Alerting : Automatically generates standard 4-window Prometheus recording and alert rules ( Critical Page 1h/6h & Warning Ticket 3d/14d ). Dynamic "No-Code" Management : Click "+ Configure / Add SLO" in the Web UI or call POST /v1/slo/edit to create, edit, or disable SLOs on the fly without editing Compose

Use a kill switch inside the checker when your uptime probes start amplifying an incident — one feature flag, read on every tick, that can disable the noisy checks and stop the retries at the source. Reach for tuned backoff and jitter instead when the retry storm stays inside a single process and never fans out onto a dependency somebody else is paging for. Both are cheap to build. Only one of them lets you quiet a polling client while its target is already on fire. I run cron and queue infrastructure, so most of my pages arrive as either "the job didn't run" or "the job ran four times." Health checking sits in the same family of problems: a small, frequent, automated request that multiplies badly when something upstream changes shape. What follows is the runbook I settled on after a fleet of pollers turned a non-incident into a real one — the failure mode, where the switch belongs, the implementation, and how to verify the flip before you walk away from the terminal. What actually turns a polling client's uptime checks into a retry storm? Amplification. A single check is one request every 15 or 30 seconds, which nobody notices; a fleet of checks with retries layered on top is a synchronized load generator pointed at whatever you decided was important enough to monitor. The math is unkind. Take 40 instances, a 5s interval, and 3 retries per failed attempt, and a dependency that normally handles a trickle of health traffic suddenly sees a couple thousand requests a minute — all of them arriving at the exact moment it's least able to absorb them. Retries stack on top of the polling interval rather than replacing it, and because every poller sees the same failure at the same time, they all back off together and return together. The Google SRE book calls out this shape under cascading failures, and the load pattern that comes out of it looks nothing like organic traffic: sawtooth spikes, perfectly aligned, growing until something sheds load. The worst one I've dealt wit

TL;DR: Grafana Agent reached End-of-Life on November 1, 2025 and has been replaced by Grafana Alloy. Alloy consolidates Agent's Static mode, Flow mode, and Kubernetes Operator into a single collector built on the OpenTelemetry Collector while maintaining native support for Prometheus and Loki. If you're using Flow mode, migration is relatively straightforward. If you're using Static mode, the migration process will involve reviewing and testing the converted configuration. Before switching over, verify relabeling rules, recheck resource usage, and confirm that Prometheus and Loki are receiving the same data and labels as before. If you're still running Promtail, it's worth migrating both to Alloy at the same time since Promtail is also End-of-Life. If you deployed Grafana Agent a couple of years ago, there's a good chance you haven't thought about it since. It quietly collects metrics, ships logs, and generally stays out of the way. What you may not realize is that Grafana Agent reached End-of-Life on November 1, 2025. That includes Static mode, Flow mode, and the Kubernetes Operator. Grafana Labs has stopped creating bug fixes, security patches, and official support. If you're still running it, your collection layer is probably still performing normally, but is now unsupported. That doesn't necessarily mean it will stop working tomorrow, plenty of unsupported software continues running for years. It does mean you're taking on the risk yourself, especially as the rest of your monitoring stack continues to evolve. This article covers why Grafana Labs replaced Agent with Alloy, what actually changes during the migration, and where people tend to run into problems. Why Grafana Agent was deprecated One of the biggest issues with Grafana Agent is that it was essentially three agents, not one product: Static mode, which used YAML and looked similar to Prometheus. Flow mode, which introduced a component-based configuration using River. The Kubernetes Operator, which manage

Series: Platform engineering with Go | Topics: Go, Kubernetes, Cobra, client-go, metrics-server, Platform Engineering This is part of the Platform Engineering with Go series. This post builds on the Cobra CLI patterns from post 4 and client-go from post 3. Read post 4 first if you haven't yet. kubectl top tells you what's happening. It doesn't tell you how close to the edge you are. In post 3 and post 4 , we built a health reporter and learned how to structure a Go CLI with Cobra. Now we put both together into something with real operational value. kubectl top pods -n production NAME CPU ( cores ) MEMORY ( bytes ) go-api-7d6b9f8c4-xk2pq 240m 490Mi go-api-7d6b9f8c4-mn9rt 180m 210Mi go-api-7d6b9f8c4-p8wvz 200m 198Mi That first pod is using 490Mi of memory. Is that fine or is that a problem? Without knowing the limit, you can't tell. You'd have to run kubectl describe pod go-api-7d6b9f8c4-xk2pq , find the resources section, do the mental arithmetic, and repeat for every pod you care about. ferctl top does all of that in one command: ferctl top -n production NAMESPACE NAME CPU USE CPU REQ CPU LIM CPU% MEM USE MEM REQ MEM LIM MEM% STATUS production go-api-7d6b9f8c4-xk2pq 240m 250m 500m 48% 490Mi 256Mi 512Mi 95% !! CRITICAL production go-api-7d6b9f8c4-mn9rt 180m 250m 500m 36% 210Mi 256Mi 512Mi 41% OK production go-api-7d6b9f8c4-p8wvz 200m 250m 500m 40% 198Mi 256Mi 512Mi 38% OK One pod is at 95% of its memory limit. In production, that's a page waiting to happen. ferctl top catches it before it becomes an incident. What you'll learn How to extend the Cobra CLI structure from post 4 with a real subcommand How to query the metrics-server API using k8s.io/metrics How to correlate live metrics with pod specs to show usage vs limits How to implement configurable near-limit warnings How to format clean aligned output with tabwriter How to verify the tool against your real minikube cluster Prerequisites Posts 1–4 read; client-go patterns from post 3 , Cobra CLI structure from pos

Originally published at wostal.eu . TL;DR : My Hetzner k3s lab quietly became a platform. Dozens of operators with leader-election leases hammered the default datastore — SQLite via kine — until compaction entered a death-spiral: 1.36M rows, a 13.8 GB WAL that wouldn't checkpoint, CPU pinned at 99%, load average 79 on 8 cores. I stopped the bleeding by truncating the WAL, then migrated the control plane to embedded etcd (7.5 GB SQLite → 313 MB etcd, load 79 → 5). This is the full postmortem — and the lessons. This is a war story, not a tutorial. It's about the moment a homelab stops being a homelab and starts behaving like production — without ever announcing it. The cluster in question, homelab , is the Hetzner k3s setup I wrote about previously . It started small. It did not stay small. In this post I'll cover: How an overgrown lab broke the default datastore — the kine/SQLite compaction death-spiral The firefight — measuring instead of guessing, and the fix that actually worked The permanent fix — migrating the control plane to embedded etcd, and the honest caveats The meta-lesson — how to recognize when your lab has become a platform A diagnostic runbook — so next time it's minutes, not hours There's a companion piece to this incident. The CI pipeline that ran this etcd migration was itself freshly — and badly — migrated, and debugging it cost me hours over a single missing newline. I split that into its own post: I Let an AI Re-Platform My CI Pipeline. Here's What Broke. Context: it's "just a homelab" — except it isn't homelab began like any homelab: one k3s node on Hetzner, a few things to play with. The problem is that over months it quietly became a platform . A single master node ( cx43 , 8 vCPU / 16 GB, untainted, and also carrying Longhorn and workloads) now runs: ArgoCD, Kargo, Crossplane/Upbound, CloudNativePG, EMQX, Longhorn, trivy-operator, kubescape, Gatekeeper, Goldilocks/VPA, VictoriaMetrics, Loki, OpenTelemetry, Argo Workflows/Events/Rollouts, kga

Autonomous agents don't just fail loudly—they fail expensively. A single misconfigured retry loop between an agent and an LLM can generate thousands of redundant tool calls and API requests before anyone notices, turning a minor logic bug into a five-figure cloud bill. PolicyAware is built to be the operational safety net that catches this class of failure before it reaches your finance team's dashboard. 1. The Recursive Agent Crisis Every SRE and platform engineer who has run agentic workloads in production has a version of this story. An agent is wired to call an LLM, interpret the response, and take an action—often invoking another tool, which produces output that gets fed straight back into the same LLM. Under normal conditions this loop terminates in a few steps. Under a bad prompt, a malformed tool response, or a subtle logic error, it doesn't. The agent gets stuck reasoning in circles: it calls a tool, receives an ambiguous or malformed result, decides the task is incomplete, and calls the LLM again to "retry." Each retry consumes tokens, each tool call hits a downstream API, and there is no natural circuit breaker unless one has been explicitly engineered. Within minutes, a single stuck session can produce: Thousands of duplicate or contradictory API calls to internal and third-party services. Sustained LLM token consumption that dwarfs normal daily usage. Cascading load on downstream systems that were never designed for machine-speed request volume. By the time monitoring dashboards catch the anomaly—if they catch it at all—the damage is already done: a runaway bill, a rate-limited API partner, or a compromised production database from thousands of unchecked write attempts. Traditional APM tools tell you a service is under load; they don't tell you an autonomous agent is the one generating that load, or why. This is why the recursive agent crisis is fundamentally a governance problem, not just a monitoring problem. Rate limits and cost alerts fire after the

Your monitoring catches problems on everything except itself. Here is how an always-firing Watchdog alert plus an external heartbeat check turns silence into a signal, so you find out when your own alerting dies. TL;DR A monitoring system can't reliably monitor its own failure, so use a dead man's switch. Create an always-firing Prometheus Watchdog alert and route it to an independent external heartbeat service. As long as the monitoring pipeline is working, the Watchdog continuously refreshes the heartbeat. If Prometheus, Alertmanager, or the delivery path fails, the heartbeat stops and the external service alerts you through a separate channel. The key is independence: the system responsible for detecting that your monitoring is down must not depend on the monitoring stack itself. One of the traps of creating alerts on a monitoring stack is the hidden assumption that the mechanism evaluating the alert is running properly and has the ability to evaluate it. Prometheus watches your hosts and Alertmanager delivers the warnings. But what watches Prometheus? If something goes wrong and the monitoring stack fails in the middle of the night, no alerts are going out but there is definitely a problem. That is the failure mode that you should be most concerned about, because it is the one your monitoring cannot report on. The fix is an old idea with a grim name: a dead man's switch. A train's dead man's switch stops the train when the operator stops holding it down. The safe state requires continuous positive action, while the absence of that action is what triggers the response. Applied to monitoring, it means building one alert whose silence is itself the alarm. Step one: an alert that always fires This feels backwards the first time you see it, and it took me a little time to get it right. Basically, you create an alert with a condition that is always true, so it fires constantly, forever, on purpose. In the Prometheus world this is conventionally called Watchdog. - aler

As a site reliability engineer in a global company, I'm running a modern (well, relatively modern, to be honest and modest) cloud-native stack: HashiCorp Vault as the secret manager, workloads on Kubernetes clusters in AWS (EKS), and development workflows automated through Jenkins (legacy) and GitLab CI. This setup is, quite likely, familiar to you — it's the normal playbook in the cloud-native era. In theory, we have the right tools for both security and efficiency: After all, we have a state-of-the-art secret manager integrated with everything. But in reality, it's far from the truth. See if you resonate with the following scenarios: Scenario A: A new colleague just joined the team. Manager: "Your initial password to log in to your corporate account came to me via email, but since you can't log in to your mail account just yet, here, take a picture of my screen." (In some companies, taking a picture of a computer monitor would get you fired, I'm not kidding.) Scenario B: A developer needs a temp password to access a database. Dev: "Where is the newly created temporary password? Need it for debugging." Ops: "In the Vault." Dev: "I can't access Vault." Ops: "No, you can't. It's not safe to open UI access to Vault. Corporate policy." Dev: "Then how can I get the password?" Ops: "Well... Technically, the password isn't in the Vault. There is a Jenkins pipeline that calls the Vault API to generate a temp password, then stores it in Jenkins secrets. You need to request access to the corresponding Jenkins pipeline, trigger it, then get the secrets from Jenkins." Dev: "Why on earth do we store secrets in Jenkins when we have Vault, which we aren't allowed to use?" Ops: "Corporate policy, just told you." Scenario C: A new ops team member needs to update a certificate for a service running in production for the first time. Ops: "Where is the old cert?" Mentor: "In K8s as a secret." Ops: "Where is the cluster?" Mentor: "In AWS." Ops: "How do I access that?" Mentor: "You need

I've built dozens of dashboards. Most have been ignored. A few have been used constantly. The difference isn't the graphs. It's the design. The 3-second test A useful dashboard answers 'is everything OK?' in 3 seconds. Not 'let me scroll through 40 graphs to find out.' Big colored header at the top: green = healthy, yellow = watching, red = broken. That's the 3-second answer. Everything else is drill-down. The hierarchy rule Three layers, no more: Overview — one line per service, status color, key SLI Service detail — one dashboard per service, 6-12 graphs max Deep dive — triggered from service detail, domain-specific Anything beyond 3 layers is 'please get lost in my dashboard tree.' The on-call test Imagine you're on-call at 3 AM. You get paged for 'service X is slow.' Can you, in 30 seconds, use this dashboard to tell if the problem is the service itself, its database, its upstream dependency, or its downstream consumers? If yes, the dashboard works. If no, redesign. What to cut Graphs with no baseline (flat line or spiky forever — how do you know if it's bad?) Metrics you've never used in an actual incident Vanity metrics (total requests ever) Graphs where the y-axis is in units nobody understands The hidden metric The real measure of a dashboard's value: does the on-call engineer open it before or after the paging tool? If they open it first — it's their compass. If they open it only after being paged — it's a reference, not a dashboard. Aim for the first. Written by Dr. Samson Tanimawo BSc · MSc · MBA · PhD Founder & CEO, Nova AI Ops. https://novaaiops.com

Liquid syntax error: Variable '{{service=\"{service}' was not properly terminated with regexp: /\}\}/

I’ve seen enough postmortems to know that a large percentage of them are essentially polite fiction. We sit in a meeting, everyone is exhausted from the outage, and we agree on a narrative. We write down that 'the database connection pool was exhausted' or 'a bad deploy caused an error spike.' Then we add an action item like 'add more monitoring' or 'improve testing,' and we move on to the next feature request. Three months later, the exact same thing happens. The same service, the same error, the same fatigue. We didn’t solve anything; we just documented our failure with slightly better prose. The problem is that most postmortems fail at the fundamental level of investigation. They stop at the symptoms. They treat human error as a root cause—which is lazy engineer shorthand for 'we don't want to fix the system.' And they treat vague timelines as acceptable data, which makes reconstruction impossible when you’re trying to correlate logs from different subsystems. This is why I became interested in using MCP (Model Context Protocol) not just to give agents access to my tools, but to act as an auditor for these processes. Most people use LLMs to summarize what happened. That's useless. You don't need a summary; you need someone to tell you where your investigation is weak. I’ve been working with the Incident Postmortem Prover ( https://vinkius.com/mcp/incident-postmortem-prover ) because it doesn't try to be a scribe. It acts as an adversarial auditor for SREs and engineers. It uses semantic trap lists designed to catch exactly the kind of hand-wavy logic that ruins investigations. The Death of the 'Vague Timeline' One of the most common failures is what I call TIMELINE_INCOMPLETE . You'll see things like: 'Around 3 PM, we noticed a spike in errors. By 4 PM, everything was back to normal.' That isn't a timeline; it's an anecdote. An actual investigation needs minute-by-minute reconstruction in UTC. What happened at 15:02? Who acknowledged the PagerDuty alert? When did

TL;DR: Passing a historical blockNumber to eth_call , eth_getBalance or eth_getLogs silently routes your request to the archive tier of hosted RPC providers. In our production metrics, archive calls cost on average 26.7x more compute units than the same call at latest . This post explains why, shows the exact client code pattern that triggers it, and gives you three Prometheus queries to measure your own archive exposure in under a minute. Full cross provider measurements are published in the OpenChainBench RPC benchmarks . Last week our RPC cost dashboard flagged an overage projection above four thousand dollars for a single billing cycle on a single provider. On paper, our services were doing normal eth_call operations. In practice, one small pattern buried in three separate indexers had multiplied our compute unit consumption by more than an order of magnitude, and nothing in the code review process had surfaced it. This post breaks down what an archive multiplier is, why it silently inflates blockchain RPC bills across every major hosted provider, and how to detect it in your own Prometheus stack before the next overage alert lands in Slack. What does "archive" mean at the Ethereum node level? Every request that reads the state of a smart contract, whether through eth_call , eth_getBalance , eth_getStorageAt , eth_getCode , or a batch of these, requires the RPC node to reconstruct the world state at a specific block height. Ethereum clients handle this in two modes. Full node mode. The state trie is kept in memory or on fast SSD for the tip of the chain plus a rolling window of recent blocks. On Geth default settings that window is 128 blocks deep. Any query targeting latest , pending , or a block within that window resolves in a few milliseconds against the current state. Archive node mode. The client preserves every intermediate state trie since genesis. Answering a query at a block from months or years ago requires reading historical trie data off disk and re

Over the last few years, I've been exploring AI agents, and one thing became obvious. There are hundreds of AI agents available today, but almost all of them are general-purpose. They can answer questions, write code, or browse the web, but very few truly understand the day-to-day challenges of running production infrastructure. As someone who has spent years working in DevOps, I wanted something different. That's why I built DevOps Open Agent, an open-source, self-hosted AI platform designed specifically for DevOps engineers, SREs, and Platform teams. Today, the project includes: ✅ Kubernetes Debugging Agent for AI-assisted cluster troubleshooting ✅ AWS DevOps Agent for investigating infrastructure issues ✅ Cloud Cost Detector to identify optimization opportunities ✅ GitHub PR Reviewer with DevOps-focused code reviews ✅ Slack, Microsoft Teams, and PagerDuty integrations ✅ MCP support for connecting external tools and services ✅ Support for multiple LLM providers including OpenAI, Anthropic, Gemini, OpenRouter, and Ollama But this is just the beginning. There is so much more we can build together: ✔️ Better Kubernetes diagnostics ✔️ Smarter AWS investigations ✔️ Terraform and Infrastructure-as-Code analysis ✔️ Observability integrations ✔️ Performance debugging ✔️ Security analysis ✔️ Historical investigation memory And many more AI-powered workflows for production engineering If you're passionate about DevOps, SRE, Platform Engineering, or Generative AI, I'd love to have you involved. Whether you contribute code, improve documentation, report bugs, review pull requests, or suggest new ideas, every contribution helps move the project forward. ⭐ Give the repository a star 🍴 Fork the project 🚀 Pick an issue and submit a pull request If you've been looking for an opportunity to work at the intersection of DevOps and AI, this is it. Let's build the open-source AI platform that every DevOps engineer wishes existed. 🔗 Repository: https://github.com/ideaweaver-ai/devops-op

Build software that heals itself in the agentic era Bucabay Bucabay Bucabay Follow Jul 1 Build software that heals itself in the agentic era # ai # agents # architecture # security 13 reactions 3 comments 10 min read

Circuit breakers and bulkheads are design patterns — their numbers are operational weapons. Failure ratio 50% or 30%? Max concurrent calls 25 or 100? During an outage the right answer changes hourly . Code the pattern once; tune thresholds live . Kiponos.io separates resilience structure (in Java) from resilience parameters (in live config tree). Pattern in code, numbers in Kiponos public boolean allowCall ( String downstream ) { var cfg = kiponos . path ( "resilience" , downstream ); return breaker ( downstream ) . failureRateThreshold ( cfg . getFloat ( "failure_rate_threshold" )) . waitDurationInOpenState ( cfg . getInt ( "open_seconds" )) . permittedInHalfOpen ( cfg . getInt ( "half_open_calls" )) . tryAcquire (); } Ops opens circuit sensitivity during brownout — dashboard edit, not redeploy. Resilience tree resilience/ payments-api/ failure_rate_threshold : 0.5 open_seconds : 30 half_open_calls : 5 bulkhead_max_concurrent : 40 inventory-api/ failure_rate_threshold : 0.35 open_seconds : 60 bulkhead_max_concurrent : 25 global/ force_open_all : false Extreme: coordinated degradation Platform SRE sets force_open_all: false normally. During regional disaster, flip selective open_seconds sky-high on non-critical downstreams — bulkhead by configuration , Java still executes pattern logic. Performance Breaker checks are per-call — getFloat() must be local. See rate limits article . Getting started Externalize Resilience4j YAML values to resilience/* Incident drill: tighten failure_rate_threshold live Resources: github.com/kiponos-io/kiponos-io Kiponos.io — resilience patterns with live numbers. Breakers that bend during the outage.

The Post-Mortem Nobody Learns From I've sat through hundreds of post-mortems. Most follow the same pattern: something breaks, someone writes a Google Doc, we have a meeting, we list action items, nobody follows up, the same thing happens again in 3 months. Here's how to break the cycle. The Blameless Culture Trap "Blameless" doesn't mean "actionless." The biggest failure mode I see is teams that use blameless culture as an excuse to avoid accountability. Blameless means: we don't punish the person who pushed the bad deploy. Blameless does NOT mean: nobody is responsible for fixing the systemic issue. My Post-Mortem Template # Incident: [SERVICE] [SYMPTOM] on [DATE] ## Impact - Duration: X minutes - Users affected: N - Revenue impact: $X - SLO budget consumed: X% ## Timeline (UTC) - HH:MM - First alert fired - HH:MM - On-call acknowledged - HH:MM - Root cause identified - HH:MM - Fix deployed - HH:MM - Service recovered - HH:MM - All-clear declared ## Root Cause [2-3 sentences. Technical but readable.] ## Contributing Factors 1. [Factor that made the incident possible] 2. [Factor that made detection slow] 3. [Factor that made resolution slow] ## What Went Well - [Something that worked] - [Something that helped] ## What Went Wrong - [Process failure] - [Technical gap] ## Action Items | Action | Owner | Priority | Due Date | Status | |--------|-------|----------|----------|--------| | ... | ... | P1/P2/P3 | ... | Open | ## Lessons Learned [1-2 paragraphs of genuine insight] The Action Item Problem Action items from post-mortems have a 30% completion rate industry-wide. That's terrible. Here's why: Too many items (I've seen post-mortems with 15 action items) No clear ownership No deadline No follow-up mechanism Competing with feature work The Fix: Three Rules Rule 1: Maximum 3 action items per post-mortem. If you can't narrow it to 3, you haven't identified the real problems. Rule 2: Every action item gets a JIRA ticket linked to the next sprint. Not "someday." Not "bac