标签：#Engineering

When working on a web project, one of the easiest ways to create unnecessary development work is to start coding before the feature requirements and integration approach are clear. I’ve found that creating an issue, proposal, or short technical plan before development can make a big difference. It gives everyone an opportunity to discuss the idea, identify potential problems, and agree on an implementation approach before code changes begin. This is particularly useful for projects that evolve over time. New features can affect existing components, user flows, APIs, databases, and the overall interface. Thinking about these dependencies early can reduce redesigns and duplicated work. For example, while working on projects such as Simulator Drag Race , planning new simulation features before implementation helps keep the existing functionality organized while making room for future improvements. A simple pre-development process can be: Describe the feature and the problem it solves. Create an issue or proposal for discussion. Identify which existing components will be affected. Discuss possible implementation approaches. Agree on the approach before development starts. Break the approved approach into smaller development tasks. This process doesn't need to be complicated. Even a short issue with clear requirements and a few implementation notes can prevent misunderstandings later. Another benefit is that early communication gives maintainers and contributors visibility into upcoming changes. Someone may already be working on a related feature, or a maintainer may know about an architectural limitation that isn't immediately obvious. For open-source and collaborative projects, I think this approach is especially valuable. Good communication before development can be just as important as the code itself. How does your team handle feature proposals before development? Do you prefer detailed technical proposals, simple GitHub issues, or discussing the implementation dire

Implement multi-layer circuit breakers, payload hashing, and financial cutoffs before an autonomous agent drains your backend. The Bottleneck in Production Autonomous AI agents running in tool-use loops fail unpredictably. When an LLM encounters an unexpected schema, a transient network error, or an ambiguous prompt, it often enters a hallucinated retry storm. In standard web apps, a runaway loop hits a rate limit or returns a 500 Internal Server Error . In agentic architectures, an unconstrained ReAct loop executes external API calls continuously, burning tokens, exhausting upstream quotas, and running up massive cloud bills in minutes. Here is the anti-pattern running in far too many codebases: # Anti-pattern: Unbounded autonomous agent loop while not task_complete : action = llm . decide_action ( state ) result = external_api . call ( action . endpoint , action . params ) state = update_state ( result ) If the LLM fails to transition state due to an unparseable response, this loop runs indefinitely. Cloud providers do not issue refunds for self-inflicted API usage. The System Architecture & Fix To make AI agent tool execution production-safe, never allow direct API calls from agent code. Route every external request through an isolated API Safety Wrapper implementing three distinct layers of defense: Deterministic Request Firewall: A hard cap on execution count per task session (Time-To-Live counter). Sliding-Window Loop Detector: Hashing outgoing request payloads to catch repetitive or oscillating tool invocations. Financial Kill Switch: A pre-flight budget validator that cuts credentials immediately if projected cost exceeds session limits. [ AI Agent Engine ] │ ▼ [ API Safety Wrapper ] ├── 1. Call Counter Check (Limit < N) ├── 2. Hash Duplicate Detector (Window: last 3 calls) └── 3. Pre-flight Cost Estimator (Budget < Limit) │ ┌────┴──────────────────────────┐ [ Passed ] [ Tripped ] │ │ ▼ ▼ [ External Upstream API ] [ Emergency Kill Switch ] (Revoke Token & Ab

A router, an IP camera, an industrial controller: somewhere in that device's firmware there's a Linux kernel with modules, a handful of statically linked binaries, and a userspace built from a dozen open source components. You don't have the vendor's source tree. What you have is a .bin file, or after unpacking it, a pile of .ko , .o and stripped ELF binaries. The question you actually need answered is boring but important: is any of this running something with a known CVE? This comes up constantly in embedded and IoT work, and it's a different problem from auditing your own codebase. You're not hunting for a new bug, you're checking for old ones the vendor never patched. In practice that's the more common finding: not a novel zero-day, but a five-year-old OpenSSL or BusyBox build nobody was tracking. Unpack first, guess later binwalk is still the first move. Point it at the firmware image and let it scan for known magic bytes: SquashFS, CramFS, JFFS2, gzip streams, kernel headers. Most consumer and SOHO firmware is a bootloader plus a compressed filesystem, and binwalk's extraction mode gets you the actual filesystem tree instead of one opaque blob. Once you have that, you're auditing files, not guessing at a blob. Fingerprint by version string, not by hash Hash-matching binaries against known-vulnerable databases sounds appealing and mostly doesn't work here, because vendors relink, strip and sometimes patch without touching anything else. What works more often: grep the extracted binaries for version banners. strings on busybox , openssl , dropbear , lighttpd , zlib and similar userspace binaries usually still leaks a version string even when the binary is stripped of debug symbols, because those strings are compiled-in constants the program itself prints or logs, not debug metadata. strings <binary> | grep -iE "openssl|busybox|dropbear|zlib" is unglamorous and it's the single highest-signal step in this whole process. Cross-reference what you find Once you have

Over the past year, I've watched a lot of engineering teams go through the same adoption pattern with AI tools. They start using GitHub Copilot or Claude. Productivity goes up. And then someone in a meeting asks the question: "Do we still need as many junior developers?" I think that question reveals exactly the wrong mental model. The teams getting the most value from AI tools aren't the ones who figured out what AI can automate. They're the ones who figured out what AI should automate, and then designed their workflows around that distinction. That sounds like a small difference. It isn't. Most of the debate around AI and junior developers focuses on the wrong question: can AI do what juniors do? In a previous article, I explored why that question leads teams in the wrong direction. In another, I looked at what happens when organizations quietly remove the work juniors need to grow. This article is about what the best teams actually do instead. They don't pick AI over junior developers. They redesign how work flows. The AI and Junior Developers Debate Is Asking the Wrong Question The argument goes like this: AI can generate code, write tests, and produce documentation. Junior developers also generate code, write tests, and produce documentation. Therefore, AI can replace junior developers. This looks logical at the task level. But it misses something important. Junior developers aren't primarily valuable for their output. They're valuable for what they become while producing that output. Every bug they debug, every test they write, every pull request they review is quietly building something that doesn't appear in any sprint metric. You can automate a task. You can't automate the learning that comes from doing it. That's where the replacement narrative breaks down. What AI Is Actually Good At After using AI coding tools seriously for a while, certain patterns become clear. AI is fast and reliable for repetitive, well-defined work: boilerplate, standard implementat

Cloudflare has recently detailed how it is using AI to transform internal engineering standards from passive documentation into an actively enforced control system across the software development lifecycle. By Craig Risi

Wie mein Agent aus 211 Fehlern ein besseres System geworden ist als ich es je programmieren könnte Heute Morgen hat mein Agent etwas getan, das er vor drei Monaten nicht konnte. Er hat einen eingehenden Webhook-Payload selbstständig klassifiziert, die richtige Skill-Route gewählt und dabei einen Edge Case abgefangen, den ich nie explizit beschrieben hatte. Ich habe das erst bemerkt, als ich die Logs durchgesehen habe. Der Agent hatte eine Regel angewendet, die ich nie geschrieben habe. Entstanden aus einem Fehler vom 14. März, bei dem er den falschen Dispatcher aufgerufen hat. Damals habe ich ihn korrigiert. Heute hat er die Korrektur automatisch angewendet, ohne dass ich auch nur daran gedacht hätte. Das ist der Crystallization-Loop. Und er verändert grundlegend, wie ich über KI-Systeme denke. Was der Crystallization-Loop eigentlich ist Die meisten KI-Workflows funktionieren so: Man gibt dem Modell einen Prompt, bekommt eine Ausgabe, korrigiert manuell, wiederholt. Jede Session beginnt von vorne. Das Modell lernt nichts. Du lernst vielleicht etwas, aber das nächste Mal ist die Chance hoch, dass der gleiche Fehler wieder passiert. Der Crystallization-Loop bricht diesen Kreislauf auf. Jede Korrektur, jedes Feedback, jeder Fehler wird automatisch in eine persistente Regel umgewandelt. Diese Regel landet in einer strukturierten Wissensbasis, die der Agent bei jeder neuen Session lädt. Das Prinzip ist einfach. Die Konsequenz ist dramatisch. Nach drei Monaten habe ich: 211 kristallisierte Regeln in strukturierten Markdown-Dateien 73 Learnings aus echten Fehlern und Korrekturen 61 Skills, die automatisch aus wiederkehrenden Aufgaben entstanden sind 308 Memory-Dateien, die den Kontext meines Projekts dauerhaft speichern Kein einziges dieser Dokumente habe ich manuell geschrieben. Sie sind alle aus echten Interaktionen entstanden. Die technische Implementierung Das System besteht aus drei Komponenten, die zusammenspielen. 1. Der Feedback-Collector Jedes Mal, wenn ich den Ag

A parent class must be able to be substituted by its child classes without breaking the application. In practice, this helps to organize the idea of inheritance, as it prevents us from extending a parent class only to later remove an already implemented method or do a “throw new Error(‘Not implemented’)”. Making us much more careful during planning. THE BIGGEST SYMPTOM OF ERROR Unfortunately, it is a symptom that appears late, but it is exactly when we are going to make a new implementation. You realize you violated Liskov when you are going to build a class or subclass and need to purposely throw an error in the implementation of a method. Exactly because that method shouldn't be there, but it is. A BAD EXAMPLE For example, in a delivery system. In this case, the “Delivery” class should be the parent/base for the other implementations. But the ‘MotoboyDelivery’ class breaks this. Code Example: // BAD: The subclass breaks the parent class contract. class Delivery { public calculateShipping (): number { return 15.0 ; } public getTrackingCode (): string { return " TRK123456789 " ; } } class MotoboyDelivery extends Delivery { public calculateShipping (): number { return 8.0 ; } // ERROR! There is no tracking code. public getTrackingCode (): string { throw new Error ( " Motoboys do not have a tracking code. " ); } } THE SOLUTION For those who do not yet know the 'Liskov Substitution Principle', it might seem that fitting in a sequence of 'if's is the solution. But in reality, the ideal path is to rethink how this abstraction is built. A good guiding principle is to think that a child class must always be able to take the place of the parent, without breaking the application. A GOOD EXAMPLE Still in the delivery system. ‘Delivery’ now has ‘TrackableDelivery’ in the middle of the way. With this, each “leaf”/edge of the application inherits what makes the most sense and nothing is broken. Code Example: interface Delivery { calculateShipping (): number ; } interface Trackab

1. The question that started everything Three weeks into my internship, my supervisor sat down across from me and asked, very casually: "OK your NLP pipeline extracts intentions and rules from legacy Java. Nice. And then what? " I looked at him. I looked at my laptop. I looked back at him. The whole project — Pulsar Modernizer — was supposed to eventually turn legacy Java into modern Spring Boot code. My part was the "understand the old code" part. F1 = 0.857 on the annotated corpus, a shiny React UI, everything humming in Docker. But the "and then?" was doing a lot of work in that sentence. That evening I wrote in my notes: "Nobody has actually tried the generation part. Everyone assumes it'll be easy because LLMs. That is very obviously wrong." So I decided to try. 2. Why "just prompt an LLM to rewrite it" doesn't work The naive move — feed the old code and the extracted rules to an LLM and say "please modernize this" — has three problems and I hit all of them in the first hour: The model hallucinates. It happily invents helper classes that don't exist and calls methods with the wrong signature. You have no criterion for stopping. The model tells you "it's done ". OK. Is it? By what test? You have no criterion for equivalence. Even if it compiles, how do you know the new code actually does what the old one did? I needed something more constrained than "prompt it and pray". 3. The setup — a chain, not a monolith I ended up building three specialized agents in sequence: IntentCard + RuleCards │ ▼ [APIDesigner] ──► JSON contract (class, methods, DTOs, throws) │ ├───────────────┐ ▼ ▼ [CodeGenerator] [TestGenerator] │ │ ▼ ▼ .java *Test.java │ │ └────► verifier (mvn test) The key insight: each rule extracted from the legacy code should become a test that the generated code has to pass. This flips the whole thing. I don't trust the LLM. I trust javac and JUnit. I did all of this on a local model — Qwen 2.5 Coder 3B via Ollama. No cloud APIs, no data leaving my Mac. On a

Uma classe mãe deve ser capaz de ser substituída pelas suas classes filhas sem que a aplicação quebre. Isso na prática ajuda a organizar a ideia de herança, já que nos faz evitar estender uma classe mãe, apenas para depois remover um método já implementado ou fazer um “throw new Error(‘Not implemented’)”. Fazendo com que tenhamos mais cuidado no planejamento. O MAIOR SINTOMA DE ERRO Infelizmente é um sintoma que aparece de forma tardia, mas é justamente quando vamos fazer uma nova implementação. Você percebe que feriu o Liskov quando você vai construir uma classe ou subclasse e precisa lançar um erro proposital na implementação de um método. Justamente porque aquele método não deveria estar ali, mas está. UM EXEMPLO RUIM Por exemplo em um sistema de entregas. Nesse caso a classe “Delivery” deveria ser a mãe/base para as demais implementações. Mas a classe ‘MotoboyDelivery’ quebra isso. Exemplo de Código: // RUIM: A subclasse quebra o contrato da classe mãe. class Delivery { public calculateShipping (): number { return 15.0 ; } public getTrackingCode (): string { return " TRK123456789 " ; } } class MotoboyDelivery extends Delivery { public calculateShipping (): number { return 8.0 ; } // ERRO! Não tem código de rastreio. public getTrackingCode (): string { throw new Error ( " Motoboys não possuem código. " ); } } A SOLUÇÃO Para quem ainda não conhece o 'Liskov Substitution Principle', pode parecer que encaixar uma sequência de ifs é a solução. Mas na verdade o caminho ideal é repensar como essa abstração é construída. Um bom norte é pensar que uma classe filha sempre deve ser capaz de substituir o lugar da mãe, sem quebrar a aplicação. UM EXEMPLO BOM Ainda no sistema de entregas. ‘Delivery’ agora tem no meio do caminho ‘TrackableDelivery’. Com isso, cada “folha”/ponta da aplicação herda quem faz mais sentido e nada é quebrado. Exemplo de Código: interface Delivery { calculateShipping (): number ; } interface TrackableDelivery extends Delivery { getTrackingCode (): st

InfoQ has opened enrollment for the InfoQ Certified AI-Assisted Engineering Program, a five-week online certification program for senior engineers and architects who already run a coding agent against production code daily, where the open questions have moved past prompting into what the agent is allowed to touch and what catches its mistakes before a human does. By Artenisa Chatziou

📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram Originally published on software-engineer-blog.com . You fixed the CSS. You deployed. You opened the site and checked it yourself — perfect. Then a customer sends a screenshot of last week's layout. Nothing is broken. No deploy failed, no CDN is lying to you, no file is corrupt. The browser is doing exactly what you told it to do, several days ago, in a header you probably never wrote by hand. This is the part of web performance that gets skipped, because caching looks like a setting rather than a contract. It is a contract. And like any contract, the interesting part is not what it gives you — it is what you can no longer do once you have signed it. Throughout this post I will use one running example: PlantPal , a small plant shop. One stylesheet ( app.css ), one logo ( logo.png ), one API endpoint ( /api/products ). The floor: a page load is not one thing Before caching means anything, you have to see what it is acting on. Loading PlantPal's homepage is not a request. It is roughly 40 separate requests — the HTML, the stylesheet, a few fonts, the logo, a dozen product images, the JavaScript bundle, the product API. Each one is a full round trip: DNS is probably warm, but you still pay connection setup, the request, the server's think time, and the bytes coming back. The numbers for a first visit: ~40 requests 1.2 MB transferred 2.1 s to a usable page Which gives us the only sentence in this post that you actually need to remember: The fastest request is the one the browser never sends. Not a faster server. Not a closer edge node. Not a smaller file. No request at all. Everything below is a way of getting closer to that. max-age: buying silence The blunt instrument is Cache-Control : HTTP / 1.1 200 OK Content-Type : text/css Cache-Control : max-age=31536000 31536000 is one year in seconds. You are telling every browser that receives this response: keep this copy and use it for a year without asking me again. O

Many beginner developers believe software engineering is all about mastering programming languages, framework syntaxes, and clearing error logs. In reality, writing code is only a fraction of the actual job. The true core of software engineering lies in analyzing complex domain problems, evaluating deep trade-offs, and designing robust systems that stand the test of time. Let's explore what it genuinely takes to transition from a coder to a modern software engineer with the right engineering mindset. 1. Writing Code vs. Solving Problems Anyone with a healthy brain can learn syntax and write functional scripts after a few tutorials. However, the real engineering challenge begins long before you touch your IDE. Understanding the Domain: Breaking down business logic and user requirements. Evaluating Alternatives: Assessing whether a feature needs a complex custom hook or a simple native state. Long-term Value: Building solutions that won't break when requirements shift tomorrow. 2. The Importance of Maintainability Code is read much more often than it is written. When you are working on large-scale applications, you are never coding alone—even if you are solo for now, your future self is essentially a stranger six months down the line. Crafting clean, self-documenting code with meaningful, intention-revealing names. Enforcing single-responsibility functions to keep modules decoupled. Using predictable patterns so teammates can navigate and scale the application without getting buried in technical debt. 3. Pragmatic System Design and Trade-offs There is no silver bullet in software engineering. Every architectural decision—whether choosing a database, state management library, or caching strategy—comes with heavy trade-offs. Performance vs. Development Speed: Knowing when to optimize early and when to ship MVP code. Scalability vs. Complexity: Avoiding over-engineering simple features just because a shiny new tool exists. Balancing Constraints: A great engineer evaluate

En el post anterior exploramos qué es un Data Lake y por qué son tan importantes en el ecosistema de datos actual. Ahora es momento de ensuciarnos las manos y ver exactamente qué servicios de AWS necesitamos para construir un Data Lake completamente serverless y cómo orquestarlos. Los Servicios Fundamentales Un Data Lake serverless en AWS se construye sobre cinco pilares fundamentales que trabajan en conjunto para crear una solución escalable y costo-eficiente: Storage Procesamiento Catalogo Seguridad Explotación Amazon S3 - El Corazón del Storage S3 no es solo nuestro sistema de archivos, es la piedra angular del Data Lake. Aquí almacenamos tanto los datos crudos como los procesados, y su organización es crucial para el rendimiento y los costos. Estructura de carpetas de un data lake estandar: data-lake-bucket/ ├── raw/ # Datos sin procesar │ ├── year=2024/ │ ├── month=12/ │ └── day=15/ ├── processed/ # Datos transformados │ ├── bronze/ # Limpieza básica │ ├── year=2024/ │ ├── month=12/ │ └── day=15/ │ ├── silver/ # Transformaciones de negocio │ ├── year=2024/ │ ├── month=12/ │ └── day=15/ │ └── gold/ # Datos listos para consumo │ ├── year=2024/ │ ├── month=12/ │ └── day=15/ └── athena-results/ # Resultados de queries Notarás que todo el data lake se encuentra en un mismo bucket, esto es lo más recomendable ya que S3 tiene un límite de 100 bucket que podemos crear por cuenta (no importa la región, ya que S3 es un servicio global) Configuraciones clave en S3: Versionado habilitado para auditoría y rollback Lifecycle policies para optimizar costos (Standard → IA → Glacier) Server-side encryption con KMS para seguridad si es necesario. Cross-region replication para disaster recovery AWS Glue - El Motor de Transformación Glue es suite de servicios de data serverless que maneja tanto el descubrimiento de esquemas como las transformaciones de datos. Componentes principales: Glue Jobs : Herramienta predilecta para ejecutar ETLs, nos permite procesar y transformar los dato

I Deliberately Destroyed My Kubernetes Cluster at 2 AM. Here's What Died First. Chaos engineering is not about breaking things. It's about discovering that your "production-grade" homelab is held together by hope and a single etcd snapshot before someone else finds out for you. The Setup I was lying in bed at 1:47 AM, staring at the ceiling, unable to sleep. Not because of caffeine. Because of a thought that had been gnawing at me for weeks: If one of my nodes died right now, would my cluster actually survive? I run a 4-node bare-metal Kubernetes cluster on Talos Linux. Dell OptiPlex control plane. Three Raspberry Pi workers. Cilium eBPF. ArgoCD. Longhorn distributed storage. Prometheus. Grafana. The whole cloud-native stack, shoehorned into $220 of scrap hardware and stubbornness. From the outside, it looks solid. ArgoCD syncs green. Cilium status shows healthy. Longhorn volumes are replicated across three nodes. I have etcd snapshots every 6 hours to S3. On paper, I'm resilient. But I had never actually tested it. Not a controlled test. Not a graceful node drain. I mean chaos . Sudden death. The kind of failure that happens at 3 AM when a power supply dies, or a kernel panics, or a neighbor's construction crew hits the wrong breaker. So I got out of bed, walked to my desk, and installed Chaos Mesh. Why Chaos Engineering on a Homelab? Professionally, I design AWS infrastructure with multi-AZ failover, auto-scaling groups, and managed services that abstract failure away. At Siemens, if an EKS node dies, the managed node group replaces it before I finish reading the alert. But my homelab has no managed control plane. No AWS SLA. No auto-repair. If a Pi's USB boot drive corrupts, that node is gone until I physically fix it. I needed to know: What dies first when a worker vanishes? Not "what should die" — what actually dies. Does Longhorn really failover? Three replicas sound great until you realize two of them were on the same node. Does Cilium handle network partitio

Designing Reliable APIs for Production Applications: Lessons From Building Real-World Digital Products APIs are often described as the “bridge” between different parts of an application, but building a production-ready API involves much more than sending data from a frontend to a backend. Through my experience building full-stack applications, I've learned that a good API needs to be designed around reliability, security, maintainability and the actual needs of its users. Here are some of the principles I now consider when designing APIs: Design around resources, not screens An API shouldn't simply mirror the frontend interface. It should expose meaningful resources and operations that can evolve independently from the UI. Validate everything at the API boundary Data coming from a client should never be trusted automatically. Request validation, type checking and clear error responses help prevent invalid data from propagating through the system. Authentication is only the beginning An authenticated user should not automatically have access to every resource. APIs need appropriate authorisation and access-control rules for sensitive operations. Design predictable errors A useful API doesn't just return “something went wrong.” Clients need consistent status codes and structured error responses so that applications can respond appropriately. Think about idempotency This becomes particularly important when an API handles operations such as payments, orders or other actions that shouldn't accidentally happen twice because of a network retry. Don't expose unnecessary data APIs should return what the client needs rather than exposing entire database records. This reduces unnecessary data transfer and can also reduce the risk of accidentally exposing sensitive information. Logging and observability matter An API can appear perfect during development and still fail in production. Good logging and monitoring make it possible to understand what happened when requests fail, la

Cloudflare is introducing WriteGuard, now in private beta, to provide fine-grained security controls for MCP (Model Context Protocol) servers. It aims to make AI agents safer by controlling their access to tools that can modify data or perform actions, rather than simply read information. By Sergio De Simone

Moving from AI-Assisted Engineering to AI-Agentic Software Engineering The rise of AI coding assistants has transformed how developers write software. Tools like GitHub Copilot, ChatGPT, Claude, and Gemini have significantly improved developer productivity by helping generate code, explain concepts, and automate repetitive tasks. However, the industry is now entering the next evolution: AI-Agentic Software Engineering . Instead of AI simply assisting developers, AI agents can now take ownership of entire software engineering tasks—from requirement analysis and architecture design to implementation, testing, documentation, and code reviews. The challenge is no longer whether to use AI, but how to integrate AI agents into a structured Software Development Lifecycle (SDLC). This requires moving away from vibe coding toward specification-driven development , where AI agents operate using well-defined requirements, standards, and engineering principles. Today, I'd like to discuss two of the most popular frameworks enabling this transition. 1. Spec Kit Spec Kit is a specification-driven framework designed for Human + AI collaborative software development . The philosophy is simple: define the specification before generating the code . Rather than asking an AI to build an application from a vague prompt, Spec Kit encourages teams to create structured specifications, architectural decisions, and engineering principles that guide AI throughout the development lifecycle. Some key benefits include: Structured and repeatable software development Better requirement traceability Consistent architecture decisions Reduced AI hallucinations Lower development costs through predictable AI interactions Support for selecting the most appropriate LLM based on project requirements Integration of quality engineering practices from the beginning of the SDLC Spec Kit is particularly valuable for engineering teams that want to adopt AI without sacrificing software quality or maintainability.

Netflix open-sourced an agentic workflow for Observational Causal Inference (OCI) that reduces toil in causal analysis. Given observational data and the human user's analysis plan, the agent uses an actor-critic loop to estimate causality, write a report, and suggest next steps. By Anthony Alford

How to Replicate MySQL to BigQuery with Sling Last updated: July 2026 Getting MySQL data into BigQuery usually means picking a tradeoff. Hand-rolled scripts are cheap to start and expensive to keep alive once schemas drift. Managed connectors are quick to set up but bill per row and put your pipeline behind someone else's control plane. Sling sits in between: a single binary, a few lines of YAML, and a load path that uses BigQuery's own bulk ingest underneath. This guide walks through a real replication, end to end. Everything below — the row counts, the timings, the type mapping — comes from an actual run against a MySQL 8.4 source and a live BigQuery dataset. You can reproduce it. Installation Sling is a single binary with no runtime dependencies. Install it however suits your setup: # macOS / Linux curl -fsSL https://slingdata.io/install.sh | bash # Windows irm https://slingdata.io/install.ps1 | iex # Python pip install sling Confirm it's on your path: sling --version Connection setup Sling needs two connections: the MySQL source and the BigQuery target. Both can be set with sling conns set , which writes them to ~/.sling/env.yaml . MySQL source sling conns set mysql_source type = mysql host = 127.0.0.1 port = 3306 \ user = root password = mypass database = demo Or with a connection string: sling conns set mysql_source url = "mysql://root:mypass@127.0.0.1:3306/demo" BigQuery target BigQuery authenticates with a service-account key. The account needs BigQuery Data Editor and BigQuery Job User on the target project. sling conns set bigquery_target type = bigquery \ project = my-project dataset = demo \ key_file = /path/to/service-account.json If you have a Google Cloud Storage bucket handy, add gc_bucket=my-bucket . Sling will stage batches there and trigger a BigQuery load job from GCS, which is the fastest bulk path. Without a bucket, Sling stages locally and still loads in bulk — that's the setup used for every number in this guide. Test both connections sling c

Well, first real job and I still haven't gotten used to waking up for a 6:30 AM shift... I've been...