标签：#cicd

1. Retomando: de imagens a containers em execução Na primeira parte desta série vimos o que é o Docker, o problema que ele resolve e os três conceitos fundamentais — imagens, containers e registries. Agora que a base teórica está posta, o foco deste artigo é prático: os comandos que efetivamente viram hábito no uso diário — run , exec , logs , ps , build — aplicados a containers reais, não só ao hello-world . 2. docker run além do básico O artigo anterior já usou docker run para subir um Nginx. Vale conhecer as flags que aparecem o tempo todo: # Modo interativo, útil para explorar uma imagem manualmente docker run -it ubuntu bash # Variáveis de ambiente docker run -e POSTGRES_PASSWORD = segredo -d postgres # Montar um diretório do host dentro do container (volume bind mount) docker run -v $( pwd ) /dados:/dados -d minha-imagem # Remover o container automaticamente quando ele parar docker run --rm -it python:3.12 python3 # Limitar recursos docker run --memory = 512m --cpus = 1 minha-imagem -it combina -i (interativo, mantém STDIN aberto) com -t (aloca um pseudo-terminal) — é o par de flags para "entrar" em um container e usar um shell como se fosse uma máquina normal. --rm evita acumular containers parados no disco depois de testes rápidos e descartáveis — sem ela, cada docker run deixa um container parado para trás até ser removido manualmente. -e define variáveis de ambiente; imagens oficiais como a do Postgres costumam documentar quais variáveis elas esperam (usuário, senha, nome do banco inicial). 3. Inspecionando o que está rodando O comando mais usado para ter uma visão geral do que o Docker está gerenciando na máquina: docker ps # containers em execução docker ps -a # todos, incluindo parados docker ps -q # só os ids (útil em scripts) Para investigar um container específico mais a fundo: docker inspect meu-container # todos os metadados em JSON: rede, volumes, config docker top meu-container # processos rodando dentro do container docker stats # uso de CPU/mem

You installed GitLab Community Edition, pushed a .gitlab-ci.yml , and watched the pipeline sit at pending until it went grey. No error, no failed job, nothing in the logs worth reading. Nothing is broken. Your instance has no runners, and it never had any. Why a fresh instance has none GitLab is two things that people assume are one thing. There is the application: repositories, issues, merge requests, the CI/CD system that reads your .gitlab-ci.yml and builds a pipeline out of it. And there is GitLab Runner: a separate program, on a separate machine, that actually executes jobs. The Omnibus package installs the first. It does not install the second, and it does not come with any machines to run it on. The confusion comes from GitLab.com, where shared runners are switched on by default and most people's first experience of CI is that it simply works. That shared fleet is hardware GitLab owns and operates as part of their hosted service. It is not part of the software you downloaded, so it does not come across when you run your own instance. So on a self-managed install, GitLab will happily accept your pipeline definition, parse it, create the jobs, and queue them. Then it waits for a runner to ask for work. If no runner ever asks, the jobs wait indefinitely. Confirming it in thirty seconds Go to Admin Area → CI/CD → Runners on your instance. If the list is empty, that is your answer. For a single project, Settings → CI/CD → Runners shows the same thing scoped narrower. A fresh instance shows nothing in either place. From the command line on the GitLab server: sudo gitlab-rails runner "puts Ci::Runner.count" If that prints 0 , no runner has ever been registered against this instance. One thing worth ruling out at the same time: a job can also sit pending when runners do exist but none of them match the job's tags. If your runner list is not empty, check whether your jobs specify tags: that no runner carries. That is a different problem with a different fix, and it is

1. O Problema que o Docker Resolve "Na minha máquina funciona." Poucas frases resumem tão bem um problema que atormentou (e ainda atormenta) times de desenvolvimento: um código que roda perfeitamente no notebook do desenvolvedor, mas quebra no servidor de produção — porque a versão do Python é outra, uma biblioteca do sistema está faltando, uma variável de ambiente não foi configurada, ou o sistema operacional simplesmente se comporta de forma diferente. O Docker resolve exatamente isso: ele empacota uma aplicação junto com tudo que ela precisa para rodar — código, dependências, bibliotecas do sistema, variáveis de ambiente, configuração — em uma unidade isolada e portátil chamada container . Essa unidade roda da mesma forma em qualquer lugar que tenha o Docker instalado: no notebook do desenvolvedor, no servidor de CI, ou em produção. Esta é a primeira parte de uma série que vai do zero ao avançado em Docker: hoje o foco é entender o problema que ele resolve, os conceitos fundamentais e como eles se encaixam. 2. Containers vs Máquinas Virtuais A comparação mais comum ao explicar Docker é com máquinas virtuais (VMs), porque ambos resolvem um problema parecido — isolar e empacotar aplicações — mas de formas muito diferentes. Uma máquina virtual virtualiza o hardware inteiro: cada VM roda seu próprio sistema operacional completo (kernel incluso), gerenciado por um hypervisor. Isso garante isolamento forte, mas tem um custo alto: cada VM consome centenas de MBs a alguns GBs de disco e memória só para o SO, e leva de dezenas de segundos a minutos para inicializar. Um container , por outro lado, virtualiza no nível do sistema operacional: todos os containers em uma máquina compartilham o mesmo kernel do host, mas cada um enxerga seu próprio sistema de arquivos, processos e rede isolados — usando recursos do kernel Linux como namespaces (isolamento de visão) e cgroups (limites de CPU/memória). O resultado é que containers são muito mais leves: alguns MBs a poucas centenas

The Slack thread started with a screenshot of a green GitHub Actions run. By the third reply someone had pasted a Lighthouse JSON artefact, a link to a Chrome release note, and a question nobody wanted to own: "Which client repository still pins Lighthouse 10?" That is the week Lighthouse CI stopped being a merge gate and became a second job. The pipeline still passed and the portfolio still needed evidence, but the difference was who paid in hours: the developer shipping a feature, or the one person who inherited every lighthouserc file across client repositories. When does Lighthouse CI maintenance outgrow a CI/CD pipeline? Lighthouse CI earns its place early. You wire assertions on a preview URL, block a CLS regression, and the team trusts the red build. The cost is front-loaded configuration, not ongoing calendar time. The shift happens when success creates obligations a CI/CD pipeline was never designed to carry: Every new client repository needs a copied workflow, pinned Chrome, and preview URL rules that match their host. Assertions need tuning after flaky LCP on cold runners, so thresholds loosen until they barely catch real regressions. Account managers ask for client-ready reports, and the only export is a CI/CD artefact someone must turn into slides. Production URLs outside the two preview paths regress while the job stays green. At that point you are not "running Lighthouse CI in a pipeline." You are operating a small internal product: version pins, runner hygiene, assertion policy, and reporting glue. For a single product team that can be fine. For an agency portfolio it competes with billable delivery. How do you know Lighthouse CI in CI/CD became an unpaid side role? We treat these as signals to shrink CI/CD scope or add a managed monitoring layer, not as moral failure. Teams hit them around five to fifteen client sites, sometimes sooner when preview hosts differ wildly. Flaky Lighthouse CI runs on GitHub Actions Engineers merge after the third "Re-ru

The owner had already asked for the alert emails to stop. A fix shipped. Then another email landed. Then another. "ong it just ssent me abother email," he said, voice-dictated, unedited. Fifteen minutes later: "go another one." The system was reporting an outage that did not exist. The Transport That Only Ever Failed A 14-PR merge train had just moved every cron producer's alerting off shared email and onto Buzz, a Nostr-relay team chat. One producer per PR, each with its own liveness contract and a bead receipt. It shipped cleanly. But the library backing those producers carried a default that had only one job: fail. AF_BUZZ_CMD = " ${ AF_BUZZ_CMD :- af_default_buzz_post } " af_default_buzz_post returned 1 with "no Buzz transport injected". Every caller that sourced the library (which is every cron producer) exhausted its Buzz retries and fell through to the email floor. The system reported a false Buzz outage while the relay was healthy. It did this 2 to 5 times per hour. Evidence arrived in the logs: 581 dedup markers, a steady stream of "[INTENT ALERT FLOOR: Buzz unreachable]" emails, and sweep.log showing buzz=ok only for the handful of callers invoked through the CLI entrypoint rather than by sourcing the library. That asymmetry was the bug. The CLI had a one-line fixup swapping in the real transport, annotated in a comment as "the library path is unchanged". The library path did not, and the cron producers all take the library path. The fix promoted the real transport to the default for both seams. af_buzz_transport already discovers the installed buzz-notify.sh and already fails closed when it is genuinely missing. The dead CLI fixup was deleted. Fail-closed behavior survives, but now it is conditional on genuine absence rather than on every caller remembering to opt in. Why not migrate callers one at a time? Because the per-caller route leaves the next new producer to rediscover this the same way. Flipping the default fixes the class, not the instance. The

Week 3. "The deploy is done. Everything's green. Now what am I actually supposed to be looking at?" Previously Runner ↓ Cache ↓ Artifact ↓ Deployment Today ↓ Monitoring Junior Engineer: The canary rolled out fine yesterday. 100% traffic, all healthy. I closed my laptop. Was that wrong? Senior Engineer: Not wrong, exactly. But let me ask you something first. Your service is running on a server somewhere. Right now, this second — is it healthy? Junior Engineer: I mean... I assume so? Nobody's messaged me. Senior Engineer: "Nobody's messaged me" isn't an answer. It's the absence of one. That's the entire problem monitoring exists to solve. The Thing Nobody Says Out Loud Senior Engineer: Here's an uncomfortable fact about production systems: you cannot see them. Not directly. You're not standing next to the server, watching electricity move through it. Everything you know about whether it's healthy is a claim — something a piece of software told you, that you're choosing to trust. Junior Engineer: That sounds obvious when you say it, but I don't think I've ever actually thought about it that way. Senior Engineer: Most engineers don't, until the gap between "the system told me it's fine" and "the system is actually fine" bites them. Monitoring is the discipline of shrinking that gap — of making sure what you're told is close to what's actually true, and told to you fast enough to matter. 📒 Senior Engineer's Notebook You don't monitor a system because you don't trust it. You monitor it because you can't see it. Trust isn't the issue — visibility is. The Car Dashboard Analogy Junior Engineer: Can you make this concrete? Senior Engineer: Think about driving a car. You can't see the engine. You can't see the oil level, the coolant temperature, how much fuel is actually left in the tank, mid-drive. All of that is invisible to you, sealed inside metal, while you're doing 100 km/h. So the car gives you a dashboard. Speed, fuel, engine temperature, warning lights. You're not wat

Originally published at wostal.eu . TL;DR : I handed a CI re-platform — GitHub Actions → Argo Workflows, GitHub → Codeberg — to an AI agent. The rewrite looked correct and passed lint, but it didn't account for one thing: the runtime environment had changed completely. The old runner was an external VM; the new one is a pod inside the cluster, where Tailscale IPs aren't routable. Nobody ran the pipeline end-to-end, so every difference stayed as a landmine — six of them. The worst part wasn't the AI's mistake. It was that I spent hours blaming the network when the real culprit was a single missing newline in an SSH key. This is the companion to a different incident. While I was migrating my homelab k3s control plane from SQLite to etcd , the pipeline that ran that migration turned out to be broken in ways that taught me more than the migration itself. In this post I'll cover: Where the mess came from — an AI-assisted CI re-platform that nobody validated end-to-end The phantom CNI bug — hours of chasing MTU, Tailscale, firewall, and conntrack The real culprit — a single missing newline in an SSH key Why early failures hide later ones — six latent bugs, surfaced one at a time The lesson about delegating migrations — to a human or an AI Where the mess came from The entire CI had just been migrated from GitHub Actions to Argo Workflows , and from GitHub to Codeberg — and that migration was handed to an AI agent 🙂. The re-platforming rewrote the logic roughly 1:1, but it didn't catch that the runtime environment had changed fundamentally . The old runner (GitHub Actions / ARC) behaved one way; the new one is a pod in the cluster — from which Tailscale IPs aren't routable, but private IPs are. Nobody (human or AI) ran the pipeline end-to-end, so all of those differences stayed as landmines. That's why there were six latent bugs and the hang I'm about to describe — not random faults, but traces of a GHA→Argo translation that was never validated in the new environment. The p

After setting up Jenkins and creating my first Declarative Pipeline, the next step was preparing my machine to build Docker images. Since my pipeline will eventually clone code from GitHub, build a Docker image, and push it to a container registry, Jenkins needs access to Docker. Without Docker installed, the docker build stage would fail because Jenkins wouldn't be able to execute Docker commands. Installing Docker On my Ubuntu machine, I installed Docker using: sudo apt update sudo apt install docker.io -y Once the installation completed, I verified it using: docker --version This confirmed that Docker was successfully installed and ready to use. Verifying Docker Installation Installing Docker is only half the job. The next step is to check whether your current user has permission to use Docker. Run: docker ps Expected Output If everything is configured correctly, you should see something similar to: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Even if no containers are running, getting an empty table like the one above means Docker is working correctly. What If You Get a Permission Error? If you see an error like: permission denied while trying to connect to the Docker daemon socket it means your current user doesn't have permission to access the Docker daemon. First, check which user you're currently logged in as: whoami Example output: nishant Now add your user to the docker group: sudo usermod -aG docker $USER What does this command do? usermod modifies a user account. -aG means append the user to a supplementary group without removing existing groups. docker is the group that has permission to communicate with the Docker daemon. $USER automatically refers to your currently logged-in username. Apply the Changes The group membership won't take effect immediately. You have two options: Option 1 (Recommended): Log out of your Ubuntu session and log back in. This refreshes your user groups and is the most reliable method. Option 2: Simply restart your ter

The Situation Our team's CI/CD pipeline on Azure DevOps was taking 15 minutes to complete on every push to develop. You'd merge a PR, grab a coffee, come back — and it was still running. A 15-minute feedback loop breaks flow state — by the time the pipeline finishes, you've already switched context twice and forgotten what you were checking. I spent an afternoon digging into the Azure DevOps logs. Here's what I found. The Numbers (Before) Artifact content (uncompressed): 1,218 MB (1.2 GB) Artifact downloaded (compressed): 614 MB Download time: 3-4 min Pipeline breakdown: Build stage: ~5 min (Docker build + artifact) Download artifact: ~3 min (614 MB over the wire) Configure App Service: 2m54s (5 Azure API calls) Deploy (AzureWebApp@1): ~1 min Validate: 2m07s (sleep 30 + 3×30s probes) ───────────────────────────────── Total: ~15 min Root Cause #1: Ignoring output: 'standalone' next.config.js had this: const nextConfig = { output : ' standalone ' , // ← was there the whole time ... }; output: 'standalone' tells Next.js to produce .next/standalone/ — a self-contained directory with only what's needed at runtime. Trimmed node_modules . Auto-generated server.js . No source files. No dev dependencies. But the pipeline was ignoring it: # Old pipeline — copies everything from Docker docker cp deployImage:/app/node_modules . # 600 MB 😱 docker cp deployImage:/app/src . docker cp deployImage:/app/.next . docker cp deployImage:/app/server.js . # ... more files /bin/zip -r deploy.zip .env .next public node_modules package.json \ next.config.js jsconfig.json postcss.config.mjs decs.d.ts src server.js # Then published the ENTIRE working directory as the artifact - task : PublishPipelineArtifact@0 inputs : targetPath : ' $(System.DefaultWorkingDirectory)' # 1.2 GB of loose files + zip Azure DevOps compressed this to 614 MB for transfer. The deploy stage downloaded 614 MB to use a 24 MB zip buried inside it. The fix: # New pipeline — standalone only docker cp deployImage:/app/.next/

Introduction Transitioning from GitLab’s centralized CI/CD pipeline structure to GitHub Actions presents a unique challenge for developers accustomed to GitLab’s modular approach. In GitLab, a central 'pipelines' repository acts as a single source of truth, referenced by individual projects via the include keyword. This mechanism eliminates duplication of CI/CD configurations, ensuring consistency and reducing maintenance overhead. However, GitHub Actions operates under a different paradigm, where workflows are typically defined within the .github/workflows directory of each repository. This disparity forces users to rethink how to achieve centralization without GitLab’s native include functionality. The core issue lies in GitHub’s scoping rules for reusable workflows. While GitHub supports uses to reference workflows from a central repository, these workflows must reside in a publicly accessible repository or the same repository. This constraint introduces versioning challenges , as changes to the central workflow can inadvertently break dependent projects if not managed carefully. For instance, updating a reusable workflow without tagging a stable version can lead to inconsistent behavior across projects, as GitHub defaults to using the latest commit. Another friction point is the lack of direct equivalence between GitLab’s include and GitHub’s uses . GitLab’s include allows for seamless integration of CI configurations, treating the included file as part of the local context. In contrast, GitHub’s uses references an external workflow, which operates in its own scope . This means inputs and outputs must be explicitly defined, increasing the complexity of migration. For example, a GitLab CI job that references a shared script might fail in GitHub Actions if the script relies on environment variables not passed through the uses interface. To address these challenges, developers must adopt a hybrid approach . Composite actions , which bundle multiple steps into a sin

A test failure that takes 20 minutes to surface, buries the error in 3000 lines of log output, and gives no context about what changed is nearly useless. Good test reporting transforms raw pass/fail data into actionable signals. Failing fast — stopping the pipeline the moment you have enough information to make a decision — keeps feedback loops tight and respects developer time. These two concerns are deeply connected: you can only fail fast confidently when your reporting is good enough that a fast failure still gives you everything you need to fix the problem. What Good Test Reporting Looks Like Before discussing implementation, it's worth being precise about what "good" means here: Immediate visibility — failures are surfaced at the PR/commit level, not buried in logs Failure context — what failed, with what input, producing what output, and in which file/line Historical comparison — is this a new failure or a pre-existing one? Trend data — is this test getting flakier? Is the suite getting slower? Actionability — the report points to a fix, not just a symptom Most teams get #1 and stop. The teams that nail all five have fundamentally different debugging velocity. JUnit XML: The Universal Format JUnit XML is the lingua franca of CI test reporting. Almost every test framework can emit it, and almost every CI platform can ingest it. Understanding the format helps you produce better reports. <?xml version="1.0" encoding="UTF-8"?> <testsuites name= "My Test Suite" tests= "42" failures= "2" errors= "0" time= "8.432" > <testsuite name= "UserService" tests= "15" failures= "1" time= "2.1" > <testcase name= "should create user with valid email" classname= "UserService" time= "0.234" > <!-- Empty = passed --> </testcase> <testcase name= "should reject duplicate email" classname= "UserService" time= "0.089" > <failure message= "Expected 409, got 200" type= "AssertionError" > Expected status code 409 but received 200 Request: POST /api/users Body: {"email": "existing@example

Six failed CI runs. Thirty-six hours of GitHub Actions time. Every run timing out at exactly the 6-hour limit. The culprit was one line in tests/setup.js . The Setup We were building a multi-tenant platform with a PostgreSQL backend — around 76 database models handling everything from user accounts and billing to visitor logs and real-time notifications. The test suite had grown to roughly 1,140 test cases across 36 files. Standard stuff. CI ran on every PR. Tests passed locally. And then one day, CI just... never finished. The Anti-Pattern Here's what the test setup looked like: // tests/setup.js beforeEach ( async () => { const tableNames = await getTableNames (); // 76 tables await sequelize . query ( `TRUNCATE TABLE ${ tableNames . join ( ' , ' )} CASCADE;` ); }); The intent was clean isolation — every test starts with a blank slate. Reasonable in theory. Catastrophic in practice. The Math Do the multiplication: 76 tables × 1,140 tests = 86,640 TRUNCATE operations Each TRUNCATE TABLE ... CASCADE is not a cheap operation. PostgreSQL has to: Acquire exclusive locks on all referenced tables Walk the foreign key graph to find dependent tables Truncate each in dependency order Release locks With a moderately complex schema where most tables reference others (users → societies → members → invoices → payments → ...), a single TRUNCATE ... CASCADE on a central table can fan out into dozens of implicit truncations. Multiply that by 86,640 and you have a test suite that will never complete within any reasonable timeout. Why It Wasn't Caught Sooner Two reasons: 1. It used to be fast. When the suite had 50 tests and 20 tables, this pattern worked fine. 50 × 20 = 1,000 truncations — uncomfortable but survivable. Nobody noticed when the suite crossed a tipping point. 2. Local runs used a different database state. Locally, developers often ran a subset of tests with --grep or file-specific runs. The full suite was only ever run on CI, and CI was slow enough that most assumed i

Most teams treat "monorepo vs multi-repo" as a binary choice. Pick one, commit, move on. We ended up with a hybrid, and it turned out to be the right call — not out of indecision, but because our apps have genuinely different deployment and ownership characteristics. Here's what we built, why, and what it costs. The System The platform consists of six applications: App Type Primary Users REST API backend Node.js + TypeScript — (consumed by all apps) Society dashboard React web app Society managers, admins, accountants Company admin panel React web app Internal operations Marketing website Next.js Public Resident mobile app React Native (Expo) Residents Guard mobile app React Native (Expo) Security personnel All six apps talk to the same API. But they have very different deployment cycles, team ownership, and testing requirements. The Structure: 4 Repositories repo: main-platform (monorepo) ├── api/ — Express + Prisma backend ├── web-society/ — Society dashboard ├── web-admin/ — Company admin panel └── web-marketing/ — Marketing site repo: mobile-resident — Resident app (React Native) repo: mobile-guard — Guard app (React Native) repo: mobile-staff — Society staff mobile app (React Native) The web apps and the API live together in one monorepo. The three mobile apps each have their own repository. Why Split Mobile From Web? The driving factor was deployment cadence and review process . Web apps deploy on push — merge to main, CI builds, CDN updated within minutes. The feedback loop is fast, rollbacks are instant, and there's no approval gate between code and production. Mobile apps go through app store review. A release cycle includes building a release APK, submitting to Google Play (and Apple App Store), waiting for review, and then a staged rollout. The cadence is measured in days, not minutes. Mistakes are expensive to reverse — a bad release means submitting a patch, waiting again, and potentially having a broken version live for days. Given that difference, mob

One of the biggest mistakes beginners make when learning Terraform is treating their local machine as the deployment server. A typical workflow looks like this: terraform init terraform plan terraform apply While this approach is perfectly fine for learning, it quickly becomes problematic when working on real-world projects with multiple engineers. Consider these questions: Who deployed the infrastructure? Was the infrastructure reviewed before deployment? Can someone else reproduce the deployment? What happens if the engineer's laptop is lost or misconfigured? How do we know exactly what changed? These are some of the reasons Infrastructure as Code (IaC) is almost always integrated with Continuous Integration and Continuous Deployment (CI/CD) pipelines in professional environments. In this article, we'll build a simple Terraform CI/CD pipeline using GitHub Actions. Instead of focusing only on the YAML syntax, we'll first understand why each stage exists and how they work together to produce safe, repeatable infrastructure deployments. What is Terraform CI/CD? Terraform CI/CD is the process of automating the validation, planning, and deployment of infrastructure whenever changes are made to Terraform code. Instead of running Terraform commands manually from a developer's laptop, a CI/CD platform executes those commands automatically in a controlled environment. The workflow typically looks like this: Developer │ ▼ Git Push │ ▼ GitHub Repository │ ▼ GitHub Actions │ ▼ Terraform Init │ ▼ Terraform Validate │ ▼ Terraform Plan │ ▼ Manual Approval │ ▼ Terraform Apply │ ▼ AWS Infrastructure This approach provides consistency, visibility, and security while reducing the chances of human error. Why Not Run Terraform Manually? Running Terraform from your laptop works well for personal projects, but it introduces several risks in a team environment. Manual Deployment CI/CD Deployment Requires someone to remember every command Runs automatically Easy to skip validation Validat

Over the past few months, I've been exploring how AI agents can fit into a modern CI/CD pipeline—not to replace engineers, but to eliminate repetitive work that slows teams down. Here's what worked well: ✅ Automatically categorized incoming pull requests ✅ Flagged potential security and dependency issues ✅ Suggested fixes for linting and test failures ✅ Generated review summaries for faster code reviews ✅ Reduced context switching for reviewers The biggest lesson? AI is most valuable before the human review begins. The Problem Every engineering team eventually runs into the same issue. Developers submit pull requests faster than reviewers can process them. A typical PR often goes through several repetitive steps: CI builds Unit tests Linting Dependency checks Security scanning Style comments Reviewer assignment Documentation validation None of these tasks require deep architectural thinking, yet they consume valuable engineering time. I started wondering: What if an AI agent handled the first round of triage automatically? The Workflow Instead of waiting for a human reviewer, the pipeline lets an AI agent inspect every pull request immediately after CI starts. Developer │ ▼ Pull Request Created │ ▼ CI Pipeline Starts │ ▼ AI Agent ├── Analyze changed files ├── Review commit summary ├── Detect risky changes ├── Check coding standards ├── Explain failing tests ├── Suggest fixes └── Generate PR summary │ ▼ Human Review By the time a reviewer opens the PR, much of the routine analysis is already complete. Example GitHub Actions Workflow A simplified workflow might look like this: name: AI Pull Request Review on: pull_request: types: [opened, synchronize] jobs: ai-review: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Run Tests run: npm test - name: Run Linter run: npm run lint - name: AI PR Analysis run: ./scripts/ai-review.sh The AI step can analyze: Test failures Lint violations Changed files Security findings Dependency updates before publishing a r

A practical framework for test planning, hardware scheduling, artifact traceability, failure classification, and evidence-based quality gates Disclaimer: The views expressed in this article are my own. The architecture, examples, terminology, and code snippets are generalized for educational purposes and do not describe or disclose any employer’s proprietary systems, confidential information, or internal implementation details. A software change can compile successfully, pass unit tests, and still introduce a serious GPU regression. The failure may appear only on one GPU generation. It may depend on a particular driver, firmware revision, operating system, graphics API, or workload. A change may preserve functional correctness while quietly reducing performance. It may also cause an intermittent failure that disappears when the test is rerun. This is why GPU validation cannot be treated as conventional CI/CD with a GPU runner attached to the end of the pipeline. A dependable GPU validation platform must coordinate: Software and firmware artifacts Hardware configurations Test coverage GPU resource scheduling Failure classification Performance baselines Engineering evidence It must do all of this while operating under an important constraint: compatible GPU capacity is limited and expensive. The objective is not simply to run more tests. It is to produce reliable evidence quickly enough to support engineering decisions. Why conventional CI/CD is not enough A conventional application pipeline often resembles: Commit ↓ Build ↓ Unit tests ↓ Integration tests ↓ Deployment A GPU validation pipeline is more multidimensional: Code or configuration change ↓ Build software and firmware artifacts ↓ Determine affected GPU configurations ↓ Reserve compatible hardware ↓ Prepare the driver and runtime environment ↓ Run functional, stability, and performance tests ↓ Collect logs, traces, metrics, and crash artifacts ↓ Classify failures and compare results with baselines ↓ Make a mer

Originally published on hexisteme notes . I have a small pipeline that crossposts my notes to dev.to. It parses a Markdown file's front matter, builds a payload, and calls the dev.to API to publish. It has 19 gate tests, and every one of them was green the whole time it was shipping. It published nine articles. All nine went live with their titles broken — the front-matter quotes were sitting right there in the title, visible to anyone who looked, for nine days, and nothing in the pipeline noticed. I didn't notice either. A human had to open the dev.to profile page by accident before anyone found out. This is the postmortem, and the reason I'm writing it up as a general essay rather than just a fixed-bug log is that the root cause isn't specific to dev.to, or to Markdown front matter, or to Python. It's a category of mistake that any pipeline with an external endpoint on the other end can make: testing the payload you build, and never testing what the other system does with it. The pipeline that had "passed everything" The shape of it is ordinary. A draft file has YAML-style front matter — title: "Some Title" — because that's the convention. A parser reads the front matter and pulls out the title. A payload builder takes that title and a few other fields and assembles the JSON body for the dev.to API. The API gets called, dev.to accepts it, the article is live. Nineteen gate tests cover this path — the front-matter parsing and the payload/API contract of the pipeline's own code. All green, every publish. The gap is in what "parses the front matter" actually means. The parser isn't a real YAML parser. It's closer to line.partition(":") — split each line on the first colon, take the right-hand side as the value. That works fine for tags: testing, devops where there's nothing to unwrap. It does not work for title: "Some Title" , because the quote characters are part of the string on the right-hand side of the colon, and a partition-based parser has no concept of "this

The moment we realized our staging environment was broken It was 3 PM on a Thursday, and our team was scrambling. A critical API change had just been merged to main, but the staging environment—our supposed safety net—was showing false positives. The integration tests passed, but the mobile app was completely broken in production. That's when we knew: our shared staging environment was failing us. The Problem: Shared Staging Is Broken by Design Like many engineering teams, we operated with a single, shared staging environment. Every developer deployed their changes to the same place, leading to: Deployment conflicts: "Who deployed that breaking change?" Cascading failures: One broken PR would block the entire team Test contamination: Data from one test would leak into another Delayed feedback: You'd only discover issues after merging your PR and deploying to staging The "works on my machine" syndrome, now at scale The worst part? Our API contracts were changing constantly, but we only discovered breaking changes during integration testing—often too late. The Solution: Ephemeral Environments per PR We made a radical change: every PR gets its own isolated, short-lived environment. Here's our architecture: Our Implementation Stack Infrastructure: Kubernetes (EKS) with namespace-per-PR Orchestration: Custom GitHub Action workflow Database: Isolated RDS instance per environment Contract Testing: Pact flow + OpenAPI validation Cleanup: AWS Lambda that runs every hour, destroying environments older than 2 hours The Game Changer: Automated Contract Testing The magic wasn't just in isolated environments—it was in what we did with them. Every time a PR deployed to its ephemeral environment, we ran: Consumer-Driven Contract Testing (Pact) Our mobile and web clients would verify their expectations against the actual deployed API. If a change broke what the client expected, the PR would fail. Provider Contract Validation We'd automatically verify that the deployed API matched ou

* When a project first starts, CI/CD pipelines are usually simple. * Build the application. Run a few tests. Deploy somewhere. Done. Then six months pass. Another test gets added. Then another deployment step. A security scan. Performance tests. Notifications. More environments. Before long, a pipeline that once took five minutes now takes forty-five. I've seen this happen more than once, and it's rarely because Azure DevOps is the problem. It's usually because nobody ever stops to ask one simple question: Does this step still belong here? Everything Ends Up in the Same Pipeline One of the most common mistakes I see is trying to make a single pipeline do everything. Every Pull Request ends up running: Every unit test Every API test Hundreds of UI tests Security scans Deployment steps Report generation The result? Developers wait longer for feedback, releases become slower, and people eventually start ignoring failed pipelines because they happen too often. Fast Feedback Wins Not every test needs to run on every commit. A better approach is to think about the purpose of each pipeline. For a Pull Request, I want answers quickly. That usually means: Build the application Run unit tests Run a small smoke test suite Stop if something important fails Everything else can happen later. Long-running regression tests, cross-browser testing and other expensive checks are often better suited to scheduled or nightly pipelines. Pipelines Should Evolve A pipeline isn't something you build once and forget about. Every few months it's worth reviewing it. Ask yourself: Which step takes the longest? Which tests fail most often? Are there any tasks nobody remembers adding? Are we getting useful feedback, or just more output? Removing unnecessary work is just as valuable as adding new automation. Final Thoughts Azure DevOps is an incredibly powerful platform, but even the best tools become frustrating if they're overloaded with unnecessary work. The goal isn't to build the biggest pipel